Are Risk Matrices Valid? What Cox Actually Proved

Ask a risk professional whether the five by five matrix is sound and you will usually get one of two answers, both unhelpful. Either it is the standard and therefore fine, or it is mathematically discredited and anyone still using one is not serious. The first ignores a body of published work. The second overstates what that work concluded.

What is a risk matrix?

A risk matrix is a grid that places each risk in a cell according to two ordered categories, usually a likelihood band and a consequence band, and assigns a rating or color to that cell. Three by three, four by four and five by five are the common sizes. The output is a tier, typically red, amber or green, or a score formed by multiplying the two category numbers.

That last step is where the trouble starts, and it is worth being precise about why. The rows and columns are ordinal: they tell you that band four is worse than band three, but not by how much. Multiplying two ordinal positions produces a number that looks like a measurement and is not one.

What did Cox actually prove?

Cox's 2008 paper in Risk Analysis is a formal treatment rather than an opinion piece, and it establishes three distinct limitations.

The first is resolution. Because a matrix compresses a continuous range into a handful of bands, two risks whose quantitative expected loss differs by orders of magnitude can land in the same cell and receive an identical rating. The matrix is not wrong about them. It simply cannot see the difference.

The second, and the one that surprises people, is ranking reversal. Cox shows that under conditions he specifies formally, a matrix can assign a higher qualitative rating to a risk that is quantitatively smaller. The condition that most often produces this in practice is negative correlation between frequency and severity, which is extremely common: the events that would hurt most are usually the ones least likely to happen. When that relationship holds, the banding can invert the true ordering, and the organization dutifully works on the wrong risk first.

The third is non-additivity. Ordinal ratings cannot be summed or averaged in any way that preserves meaning, which means a matrix cannot legitimately aggregate to a portfolio view or support an allocation of budget. This is the limitation with the largest practical consequence, because aggregating and allocating is precisely what most organizations do with their matrix output.

Hubbard and Evans made a closely related argument about ordinal scales in risk assessment two years later in the IBM Journal of Research and Development, and Thomas, Bratvold and Bickel carried the argument into petroleum industry practice in 2014, working through how matrix-driven decisions diverge from decision-analytic ones on real cases.

When does a risk matrix actively mislead?

Not always, and knowing the difference is more useful than a blanket position. A matrix is most likely to mislead in four situations.

When the risks being compared have negatively correlated likelihood and consequence, which is the ranking reversal case. When the ratings are multiplied into a score and the scores are then summed, ranked across a portfolio, or used to size a reserve. When the bands are wide enough that the items inside a single cell differ materially, which is a resolution failure and gets worse as the program gets larger. And when different people populated the matrix without a shared definition of what the bands mean, which is a problem the matrix inherits rather than causes, and which is treated at length in Why 'Likely' Is Not a Probability.

Conversely, a matrix used to sort forty candidate risks into rough tiers so that a team knows where to spend its next two weeks of attention is doing something it is entirely adequate for. That is triage, and triage does not require arithmetic.

Does making the matrix bigger help?

Very little, and it can hurt. Moving from three by three to five by five multiplies the number of cells without changing the nature of the inputs, so it produces finer-looking output from the same coarse judgments. The visible effect is more confidence rather than more information.

The published work on improvement points somewhere else entirely. Levine argued in the Journal of Risk Research for logarithmically scaled axes, so that each step up an axis represents a consistent multiple rather than an arbitrary widening, which addresses part of the resolution problem directly. Duijm's review in Safety Science is the most practical single source: it works through axis definition, the rules by which cells are colored, and the case for a continuous probability and consequence diagram in place of a discrete grid.

If you are debating grid dimensions, the more productive question is what happens to the output. If anything downstream of the matrix needs a number, the size of the grid will not save you.

Why do organizations keep using them?

Because they solve a real problem cheaply, and because the alternative has a cost that the critique tends to skip over.

A matrix lets a group of people with different backgrounds reach a shared rough view of a long list in an afternoon. It requires no software, no distributions, and no specialist. Ball and Watt made essentially this case in their response to Cox, arguing that the matrix should be judged against the practical alternatives available to a working practitioner rather than against an ideal. That is a fair point and it deserves to be represented honestly.

There is also a less flattering reason, and it is worth naming. A red, amber and green grid produces a defensible-looking artifact quickly. It survives a gate review. Organizations that are optimizing for the appearance of risk management rather than for changed decisions will always prefer an instrument that produces a clean deliverable, which is the same dynamic that turns a register into paperwork.

What should you do instead, given real constraints?

Divide the work according to what each instrument can support.

Keep the matrix for screening, and be explicit on the page that its output is a triage tier and not a measurement. Define each band numerically, so that a likelihood band names a probability range and a consequence band names a cost or duration range, which removes the worst of the shared-meaning problem and makes the resolution limits visible. Stop multiplying the bands to make a score, or if the score is politically unavoidable, never sum or rank across it.

Then take the small number of risks that plausibly drive the outcome, usually far fewer than the register contains, and express them as ranges rather than categories. Combine those ranges and you have a distribution, and from a distribution you can read a defensible contingency figure. That is the move described in What P80 Means, and it is what the matrix was never able to give you.

Why is this an Alignment problem?

Because the matrix is a shared instrument in appearance and frequently not one in fact. Two people can place risks in the same grid, using the same words, and mean different things by every band. The format hides that disagreement rather than surfacing it, which is worse than an obvious inconsistency, because the output looks like consensus.

The deeper alignment failure is between what the instrument produces and what the organization asks of it. A matrix answers the question of rough priority order. It is routinely asked to answer questions about money. Nothing in the mathematics of the grid can close that gap, and no amount of additional cells will either. ISO 31000:2018 is clear that risk management should be based on the best available information and should be dynamic; a matrix built once at inception, multiplied into a score, and never revisited fails both tests before the question of validity even arises.

Evidence matrix

The summary a board can act on is short. The matrix is a reasonable way to sort a long list and an unreasonable way to produce a number, and most organizations are using it for both.