Risk, Uncertainty and Everything Else

In a recent article published in the journal Risk Analysis: An International Journal, me and my coauthors’ suggested that all risk analyses, particularly those supporting homeland security decision making, should provide decision makers with more than statements such as “your risk to so-and-so is high.” In much the same spirit of actionable intelligence, we suggested that all risk assessments should produce actionable risk information, that is, information that provides insight into how to go about reducing risk. Unfortunately, the article containing this discussion was ambitious in its scope, thus leaving behind little space (i.e., a fraction of one column) to fully elaborate on this idea (for the abstract or full-text of the original article, see Ayyub, et al., “Critical Asset and Portfolio Risk Analysis: An All-Hazards Framework.” Risk Analysis, 27(4), 789-801, doi:10.1111/j.1539-6924.2007.00911.x). I intend to use this blog-opportunity to expand on my views of “actionable risk analysis,” to include why it is important and what comprises an actionable risk analysis.

The Back Story

On numerous occasions over the past five years I spent supporting risk assessment activities for DHS and DOD, I witnessed first-hand the frustration decision makers express in response to results presented in standard risk assessment products. Too often have I heard retorts such as “so what?” or “tell me something I don’t already know.” You hear this in the intelligence community too – for example, when you deliver an assessment to a defense policy maker that seems to be lacking in explicit purpose or relevance to the current situational environment (an extreme response I only heard about involved a decision maker who physically tore up an intelligence assessment in front of its author because of its irrelevance). At the recent Second National Conference on Security Analysis and Risk Management hosted by SARMA and the GMU School of Law, Mr. Chris Geldart, Director of the Office of National Capital Region Coordination at FEMA (and Monday’s keynote speaker), echoed this general sentiment by emphasizing that risk assessments aren’t enough in themselves to protect the nation. What are needed are assessments that produce information on where to focus decision maker attention and resources to buy down risk.

Standard risk assessments inform decision makers of what scenarios pose the greatest risk to their interests, and in some cases, to what degree. But typically, a knowledgeable and well-qualified decision maker already knows what his biggest threats are, and more likely than not he can rank them in order of severity based on his own opinion and achieve results that are close, if not the same as, what can be produced by deliberate risk analysis. So what incentive is there to invest in doing risk analysis? After all, a risk assessment is expensive, and who is to say that it is worth it to spend the money doing analysis when I already know that I need a fence? To make risk assessments a value-added proposition to decision makers, they need to provide more than just a ranking of threats, risk curves, and so on. They need to provide fresh insight on what can be done to make threats less threatening.

What Does Actionable Risk Analysis Do?

Let’s start by asking the question of where the risk analyst sits on the decision-making continuum? Are risk analysts mere advisors in much the same way intelligence analysts advise policy makers, or do they play a more active role in the decision making process, such as by directly suggesting alternative risk mitigation strategies? Of course the answer to this question is “it depends.” But as I have often argued (though I have yet to do so in this blog), risk analysis and intelligence analysis, for all practical purposes, are similar if not the same in purpose. Risk analysis informs decision making by communicating to decision makers plausible futures in light of the current strategic and situational environment, assessing their likeliness of occurrence, and, if not otherwise implied, assessing the potential for undesirable or desirable outcomes in light of these alternative futures. A complete set of scenarios and the likeliness and consequences associated with each combine to form the risk picture. In the process of doing a risk assessment, risk analysts will identify a set of factors or variables thought to bear on the assessment of likeliness and consequences for each alternative future, and leverage all available information to make statements about risk with some degree of analytic confidence shaped by the quality of the model/reasoning and available information. Having done the intelligence analysis thing for a bit, I can say that this is exactly the kind of activity most intelligence professionals contribute to on a daily basis.

Now, when I speak about actionable risk analysis, I do not mean that the risk analyst is going to go so far as to prescribe solutions to a decision maker’s problems. Risk analysts, like intelligence analysts, must be careful to not cross the dividing line between [policy] analysis and policy-making. But in the process of making statements about risk, the risk analyst had to identify and make statements about the current state of all risk factors relevant to their particular problem. An actionable risk assessment goes the next step by “tweaking” each of these factors in a favorable direction to determine its effect on the risk picture. For example, does improving the economic condition in a country, ever so slightly, decrease the potential for state failure? Or does decreasing the security response time at a high-value facility decrease the probability of adversary success, which in turn decreases the contribution to overall risk due to criminal or terrorist acts? An actionable risk assessment provides insight into which variables, if influenced in a favorable direction, stands to increase the potential for desirable futures and decrease the potential for undesirable futures. It is then up to the decision maker to construct strategies aimed at favorably influencing these variables.

What an actionable risk assessment amounts to is a sensitivity analysis with respect to the variables that influence risk. Engineers, particularly reliability engineers, are very familiar with this idea of sensitivity analysis. For example, in the course of doing a reliability analysis of, say, a nuclear power plant safety system, a reliability engineer would start by constructing a logic diagram for the system, assess the reliability of the system as a whole from knowledge of the reliability of the constituent system components, then tweak the reliability of each to see which component contributes most to the unreliability of the system. This information then informs maintenance engineers on which components are most critical to unreliability in order to help them design suitable maintenance and replacement schedules, and also informs engineering designers on which components to consider redesigning. An actionable risk assessment does exactly same sort of thing; it begins by constructing a model of the situation, proceeds to make statements about risk based on information on the underlying model parameters, then tweaks each variable by some uniform amount to identify which ones contribute most to overall risk.

There is one more element that distinguishes an actionable risk assessment from a standard risk assessment. An actionable risk assessment communicates actionable risk information – that is risk results and sensitivities of model variables – in a manner that makes intuitive sense to the informed decision maker. Well, in general all risk assessments should do this, but in the spirit of redefining the thought process behind risk assessment for homeland security, I want the idea of an actionable risk assessment to place significant weight on the manner in which risk information is communicated. One approach to doing this might be to explain the model at a high level, describe the risks resulting from this model, and then follow up with insight into how variables comprising this model influence the risk.

So How Do We Produce Actionable Risk Information?

Simply put, an actionable risk assessment is simply a standard risk assessment plus sensitivity analysis and communicated in a manner that makes sense to the decision maker. In order to perform sensitivity analysis well requires the risk analyst to be explicit about all contributing factors leading to the baseline statement of risk and how they interact to create risk. In other words, to perform a sensitivity analysis requires that the underlying risk model and its variables be clear and explicit. This is all too often not the case, particularly for qualitative security risk analysis methodologies that boil the very complex problem of security down to the two vaguely-defined variables, probability and consequence.

So, in attempt to provide structure for future developers of risk assessment methodologies, I propose the following eight step process:

1. Start by defining the problem. This step provides a frame for answering risk question #1 (what can happen?)
2. Identify as complete a set of plausible scenarios as possible. This step answers risk question #1.
3. Identify all relevant factors and variables that contribute to the likeliness of each alternative scenario, and describe mathematically how these variables interact to make one scenario more likely than another.
4. Leveraging all available data and information, assess the relative likeliness of each alternative risk scenario. This step answers risk question #2 (how likely is it to happen?)
5. Again, leveraging all available data and information, assess the potential for desirable or undesirable outcomes given the occurrence of each scenario. This step answers risk question #3 (what are the consequences if it does happen?)
6. Combine the information gathered in steps 2-5 to make statements about risk.
7. Determine how risk changes when each variable, in turn, is adjusted in a favorable direction (or rather, a direction that decreases risk). This step addresses (not answers) risk question #4 (what are my options?)
8. Communicate statements of risk and the impact of model variables on reducing risk in a manner that makes intuitive sense to the decision maker. This is what actionable risk assessment is all about.

There is one more step as follows:

9. Communicate the maximum justified degree of analytic confidence that can be afforded to the assessment as a whole

Given the results just obtained from the first eight steps of an actionable risk assessment process, this ninth step answers the question of how confident the decision maker can be in the results. But as fun a topic this is to discuss, I will defer speaking about it until a later time…

I am interested in your thoughts on this issue. Feel free to post comments.