Can You Put a Number on a Risk That Never Happened?

Start by separating two claims that get said in the same breath. We have never experienced this is a fact about the organization. Therefore we cannot say anything quantitative about it is a much stronger claim, and it is false.

Insurers price events that have not happened to a particular policyholder. Engineers specify against loads a structure has never seen. Both do so with numbers, and neither waits for the event.

Three routes to a number

Decompose until you reach something you know

Most unmeasurable risks are unmeasurable only at the level they are stated. The statement a major cyber incident has no natural frequency. Its components do.

This is the logic of the FAIR taxonomy, published through The Open Group, which decomposes risk into loss event frequency and loss magnitude, and decomposes each of those in turn: contact frequency, probability of action, threat capability against control strength, and on the magnitude side the separable categories of loss the organization would actually incur. At the bottom of that tree sit quantities the organization does have information about, including how many records it holds, what its notification obligations cost per record, how long recovery took the last time a system was down for any reason, and what its own logs show about attempt volume.

The same move works outside cyber. The systems integration fails resists estimation. How many interfaces there are, how many have been tested, what proportion of interfaces historically require rework, and what rework has cost per interface do not.

Decomposition does not manufacture information. It relocates the estimate to a level where the organization already has some, which is usually two or three levels below where the risk was written.

Widen the class until it has members

You have no history of your own facility experiencing a total loss. There is extensive history of facilities like yours experiencing total losses.

This is reference class reasoning applied to a discrete event rather than to a cost forecast, and it carries the same discipline: the class must be defined before the estimate is wanted, on characteristics that plausibly drive the outcome, and it must be wide enough to contain enough members to say something. The temptation, always, is to narrow the class until only favorable cases remain, which is how an outside view gets quietly converted back into an inside view.

Elicit under a protocol, not in a workshop

The third route is expert judgment, and it deserves care because it is the one most often done badly.

The core finding in the calibration literature, consolidated by Lichtenstein, Fischhoff and Phillips and replicated extensively since, is that people asked for a range give one that is too narrow. Ask for a ninety percent interval and the true value falls outside it far more than ten percent of the time. This is not a failure of intelligence or of subject knowledge, and experts are not exempt.

Cooke's response, the classical model, is to stop treating expert opinion as a single undifferentiated input. Experts are asked seed questions to which the answers are known but which they have not memorized, their responses are scored for calibration and informativeness, and their judgments on the questions of interest are weighted accordingly. The TU Delft database that Cooke and Goossens assembled records a substantial body of applications of this method across domains, which is what makes it something other than a proposal.

Structured elicitation does not assume experts are well calibrated. It measures whether they are, and weights them by the answer. That is the entire difference between elicitation and a show of hands.

Calibration also improves with training. In a large geopolitical forecasting tournament, Mellers and colleagues found that a short training intervention in probabilistic reasoning produced measurably better forecasts. An organization that runs a calibration exercise with its estimators once a year is doing something with published support behind it, and it takes an afternoon.

What this does not solve

Three limits, stated plainly, because the case for quantification is weakened rather than strengthened by overselling it.

Tail extrapolation assumes stability. Statistical methods for extremes work by fitting a tail shape to the observed portion of a distribution and extending it. That extension is only valid if the mechanism producing extreme outcomes is the same one that produced the observed range. For a genuinely novel event, or one arising from a system that has changed structurally, the assumption is unsupported and the resulting tail estimate is a modelling artifact.

Elicitation inherits its own quality. A distribution obtained from three people in a hurry, anchored on the first number spoken, is not evidence merely because it emerged from a process labelled elicitation. The protocol is what does the work, and a protocol that is skipped produces a number with more authority than the workshop it came from.

Precision is not accuracy. A P80 stated to five significant figures is a presentational choice, and it invites more confidence than the underlying inputs support. Where an estimate is built primarily on elicited judgment, that fact belongs next to the number rather than in an appendix, and the range should be presented at a resolution that reflects what is actually known.

When the number genuinely will not come

There is a reframe that resolves more of these arguments than any estimation technique, and it is underused.

Stop asking what the probability is and ask whether the answer would change the decision.

Run the analysis across the whole plausible range of the disputed value rather than at a point. If the organization would fund the same contingency, choose the same procurement route and approve the same schedule whether the annual likelihood is one percent or ten, the number was never needed and the argument about it was a way of avoiding a decision that was already determined.

If the decision does flip somewhere in that range, the analysis has located the threshold. Now the question is bounded and answerable: what would it cost to find out which side of that threshold we are on, and is that less than the difference between the two decisions. That is a tractable question about the value of information, and it is a far better use of a meeting than another round of arguing about whether the risk is a four or a five.

Evidence matrix

What to do with this

Find the largest risk currently recorded qualitatively because there is no data, and do two things to it.

Decompose it into components until you reach a level where the organization can say something from its own records. Then run the decision at both ends of the plausible range for whatever remains unknown. One of two outcomes follows, and both are useful. Either the decision holds across the range, in which case the risk can be closed as quantified enough and the debate ends. Or it does not, in which case you now know exactly what is worth finding out and roughly what it is worth paying to find it.