This series of articles is an irreverent, tongue-in-cheek look at the serious business of risk management and compliance and the lack of scientific rigor dressed up in charts and graphs, which have an appearance of legitimacy, but tell us little about risks.
First of all, let me say that risk management and compliance are important functions and deserve to be taken as seriously as any other discipline in business and government to ensure efficient operational outcomes. My point in these articles is to point out where many firms diverge from serious risk management into the realm of mystery cloaked as rigor.
My First Victim: Risk & Compliance Self-Assessments!
Risk & Compliance Self-Assessments (RCSA) have become a handy tool to communicate to management, regulators and others that an organization has conducted an analysis of their risks to understand both the severity and likelihood of the occurrence of an event. Each risk category is highlighted with its own color-coordinated assessment based on a “Table Top” exercise wherein subject matter experts participate in a facilitated session to list these risks and assign severity and probability based on nothing more than memory!
I can’t remember what I ate for dinner three weeks ago. Should I trust my memory to document the threat level of risks to an organization based on recall? Yes, experience matters and yes, experts in their field do have important contributions to make regarding the risks they experience doing their jobs. However, what does this chart really tell us about risk? The answer is very little!
Of course, we all understand that RCSAs are subjective but the “risk” in risk self-assessments is the false sense of security we place in believing these exercises are really a representation of risk exposures in an organization. They are not, and here is why!
Statistically speaking, risks tend to have a shape. In some cases, the shape of risk is a normal curve. In other cases, the shape may be skewed to the right or left. But in an RCSA, the shape of risk is uniform. Each risk, with slight variation, looks exactly like this chart above. Intuitively, we understand that risks are not uniform, but we never question charts and graphs that look like some effort went into producing the results.
Additionally, these charts lack the benefit of the law of large numbers. You might be surprised to learn that risk management is based on scientific laws of statistical analysis. The RCSA is flawed because it’s based on a small sampling of data (your memory) that is inherently biased by recent events that are easy to recall, not representative of frequencies found in a large stochastic database of risk events. What does stochastic mean? Stochastic is a process involving a randomly determined sequence of observations, each of which is considered as a sample of one element from a probability distribution.[i] In other words, if you are not using a stochastic process for measuring risk, you are guessing!
While sitting in a conference with professional risk managers from a range of industries, I asked my fellow participant how he managed risk and if he used a system to facilitate the process. His answer did not surprise me. He jokingly said, “yes I use a system. It’s called Excel.” Each year he conducts a table-top exercise with senior management in which they list their Top 20 risks and fill in their assessment of each risk. He laughed and said he is the Wizard behind the curtain who controls the process. Once the exercise is completed, an entire year goes by before the Wizard unlocks his Excel file for another year’s list to be documented.
If your risk management program looks like this, you are practicing Outrageous Compliance! Unfortunately, many risk professionals are taught to perform this exercise, because it is easy to do and senior management feels a false sense of security in the process. By the way, show this exercise to your Board of Directors, internal or external auditors or regulators and no one will challenge you or the process to understand what it says about your risk profile. The process appears to be rigorous much like the Wizard of Oz who fears that Toto may someday pull back the curtain to unveil the truth.
RCSAs have some value as a tool for understanding the risks subject matter experts deal with on a daily basis. These tools are a great starting point, from which you should begin to develop a stochastic database of risk events — they should not be the conclusion. This brings us to the last point about Outrageous Compliance: the risk repository.
A risk repository represents a third flaw in thinking about risks. Capturing risks in a risk repository is called a deterministic model. A deterministic model is one in which every set of variable states is uniquely determined by parameters in the model and by sets of previous states of these variables; therefore, a deterministic model always performs the same way for a given set of initial conditions. Conversely, in a stochastic model—usually called a “statistical model”—randomness is present, and variable states are not described by unique values, but rather by probability distributions.[ii]
Why is this wrong? When developing deterministic models (risk repository), you predetermine the outcome. Lots of organizations make this mistake, including insurance actuary models, financial analysts on Wall Street, medical researchers and risk professionals in many organizations. The reality is that all models are wrong, but some models are useful! Understanding how to develop useful risk assessment models takes time and patience, but knowing the difference means being able to avoid Outrageous Compliance.
