When risk analysis is used as the basis for funding a project, the level of funding is typically at or below the 80th percentile, depending on the industry, and often at significantly lower levels. For major projects, NASA funds to the 70th percentile of a joint cost and schedule risk analysis. In 2009, Congress passed the Weapon Systems Acquisition Reform Act, which put pressure on major Department of Defense projects to fund to the 80th percentile. There was immediate and strident pushback to funding to the 80th percentile so this language was removed in a 2010 update to the law. Since then, when risk analysis is used as the basis of funding Department of Defense projects tend to fund to either the mean or the median.
Why do projects fund to such a low level? Wouldn’t it make sense to fund to a high percentile of a risk analysis, such as the 90th or even the 99th percentile? One reason is because of inflection points. Mathematically, an inflection point is where the concavity of a probability density function changes from concave down to concave up or vice versa. The impact of for cost risk is that when the probability density function is concave down, you can achieve higher percentile levels with relatively small increases in funding. However when it is concave up it takes increasingly larger amounts of funding to achieve smaller and smaller increases in percentile.
For example consider a normal, or Gaussian, distribution. For a continuous probability density function like the Gaussian, concavity can be measured by the second derivative. In the case of the Gaussian, the probability density function is always concave down within one standard deviation of the mean, and concave up outside that range. See the graph below.
The upper inflection point for a Gaussian distribution is always at one standard deviation greater than the mean, which is approximately the 84th percentile . As the percentile goes above that value, the amount needed to achieve that higher level of confidence begins to increase at an increasing rate, as is evident from the graph below.
If the upper inflection point is not reached until a project a funded above the 84th percentile, then why not fund to at least the 80th? The reason that many organizations such as the Department of Defense recoiled at the direction to fund to this level is that cost risk does not follow a Gaussian distribution. Rather, cost risk tends to follow a lognormal distribution, and the inflection points for a lognormal behave differently than for a Gaussian distribution. Rather than being fixed at a single percentile as with the Gaussian, the upper inflection point for a lognormal is a function of the coefficient of variation (CV). The CV is the ratio of the standard deviation to the mean and measures the relative amount of risk. It’s useful for comparing risks between projects of different sizes as it measures the amount of variation relative to the size of a project. See below for a graph displaying the percentile for the mean and the upper inflection point of a lognormal as a function of the CV.
As can been seen from the graph, the inflection point drops steadily as the CV increases. What is a reasonable level for the CV in a cost risk analysis? As often measured in quantitative risk analyses, the CV is often fairly low, typically below 30%, but what is implied by cost growth is higher. As I discuss in my book (available from Amazon here), the CV as implied by historical cost growth for a wide range of industries is closer to 90% on average. This implies that the upper inflection point is at or below the 50th percentile for many projects. In light of this, the tendency for the Department of Defense and other organizations to fund to low percentiles makes sense.