Five Myths About Quantitative Risk Analysis

I recently saw yet another article about quantitative risk analysis that made me cringe because of its mistaken claims. It was, in the words of the famous physicist Wolfgang Pauli, “not even wrong.” As a long-time practitioner and advocate of quantitative risk analysis, that article inspired me to think about common myths surrounding quantitative risk analysis (QRA). Here are five I commonly encounter:

1. QRA is often not done well, so it should not be done at all.

I heard this one directly from a senior executive at the Department of Defense to explain why his analysts did not incorporate uncertainty in their cost estimates. This is encapsulated to some extent by the maxim that the perfect is the enemy of the good. There are plenty of bad risk analyses out there, but just because doing QRA correctly is hard, does not mean that it should not be done. Leadership should instead work to promote good risk analysis practices to enable better QRAs.

2. QRA is concerned with reducing uncertainty.

No, QRA is about measuring and understanding uncertainty to the fullest extent possible. Risk management is a process that utilizes QRA to decide which risks to mitigate and which risks to accept. QRA can help provide information on which risks should be mitigated and which should be accepted.

3. Big data allows us to improve our estimates to such an extent that uncertainty is minimized.

More applicable data is always better, but just because we have more of it does not mean that uncertainty goes away. More data can help reduce one component of uncertainty, but not all or even most of it. The generalization error, or the amount of uncertainty in the estimate when used in practice is the sum of the bias, the variance, and the irreducible error. The irreducible error includes things that a model cannot predict, such as the impact of labor strikes, supply chain issues, and funding cuts for public/government projects. Big data can help improve model accuracy, but it cannot predict unknown unknowns.

4. Most uncertainty is not measurable, so QRA is a waste of time.

This seems to be a favorite among some economists. While there are many events we cannot measure, that does not mean QRA is a waste of time. A good QRA provides meaningful information to decision makers about potential uncertainty. While black swan events occur occasionally, that does not mean QRA is fruitless. A good QRA provides information that decision makers can act upon. This is another example of the perfect being the enemy of the good.

5. Using averages is a good way to incorporate the effects of uncertainty in projections.

The use of averages is a proven way to underestimate the resources required to develop and produce a system. Sam Savage coined the phrase The Flaw of Averages to describe this phenomenon. For example, the batting average in Major League Baseball has hovered around .275 for the last 100 years, but no one has hit .400 since Ted Williams accomplished the feat in 1941? The reason why is that even though the average has remained the same over time, the variation around that average has decreased significantly. In the face of uncertainty, we need to consider more than just averages – we need to consider uncertainty distributions, which can be represented by shapes called probability distributions.

For more on the subject of QRA, some of the issues with the current practice with QRA, and ways to do it better, check out my book Solving for Project Risk Management: Understanding the Critical Role of Uncertainty in Project Management. You can read Chapter 1 for free, or purchase the hardcover, ebook, or audiobook from Amazon, Barnes and Noble, and other booksellers.