IT projects are frequently among the most ambitious and highest risk projects many organizations embrace, and so it would seem reasonable to try and capture that risk in a meaningful way. Of course risk isn’t just a measure of doing something, it is also a measure of not doing something. As part of the measurement of risk we should be very concerned with the return we expect from a project, although all endeavor has a spectrum of possible outcomes.
Most IT projects are justified on the basis of a return on investment (ROI) calculation. This is simply projected benefits minus costs at its most basic level, and although the calculation can become fairly elaborate it gives no meaningful measure of risk. In practice we all know that any project may have outcomes that range from the disastrous to the sublime and many shades of gray in-between. A useful addition to the standard ROI calculation is modeling of expected return (ER). This technique is widely used in many areas of human endeavor where risk is present, although not so widely used in analyzing IT projects. There are a number of reasons for this, including politics and the nature of the way many projects are sponsored – but we won’t go there. These are internal problems, and if they cannot be addressed then things will remain as they are.
The need for a risk based approach to project evaluation is made more than clear if the success rates of IT projects are considered. From the surveys conducted by Tom de Marco in the 80s through to the current annual survey by Standish Group, the likelihood of a large IT project delivering what was expected is fairly slim – around twenty per cent. The IT press is littered with stories of public entities whose large IT projects have failed or run into massive over-spend. Corporations can hide their disasters from view and so they are typically not reported on. A risk evaluation will highlight many issues that might lead to greater project success rates.
Expected return is a particular use of a concept that sits right at the heart of statistics and risk calculation. Its more general form is known as mathematical expectation, or expected value. The concept is very simple. We simply consider a number of scenarios and apply a likelihood, or probability to each. Obviously the probabilities must add up to one, meaning they embrace all possible outcomes. We then simply multiply each outcome by its associated probability and sum them. This is simply an act of weighting each outcome and summing to find an expected value.
To make all of this real let’s consider a large organization wishing to deploy a new suite of applications for its core transactional processing – usually encapsulated within an Enterprise Resource Planning (ERP) application. The range of scenarios we might consider range from ruin (the organization goes bust) through to a very significant increase in productivity. Both of these scenarios are the least likely ones, but large IT projects have ruined organizations and delivered benefits way beyond expectations. A range of other scenarios sit in the middle. So lets list the scenarios with associated probabilities.
Ruin
This does occasionally happen and I have witnessed it twice. Ruin means the organization fails and is wound up. We’ll give this a probability of 0.1% or one in a thousand. We shall assume that this organization has a market capitalization of US$1 billion, and failure would mean a loss of this amount.
Severe Adversity
The organization survives but experiences losses which are around 30% of market capitalization – so US$300 million. We’ll give this a probability of 1% or one in a hundred.
Moderate adversity.
The applications create considerable disruption with result in losses of around 10% of market capitalization – US$100 million. This will have a probability of 28.9% (just to round things up) – or about one in three.
ROI Expectation
This is the outcome that is the basis of the ROI calculation and results in gains of 5% of market capitalization – US$50 million. This will have a probability of 50%.
ROI Moderately Exceeded
We’ll assume here a gain of 10% of market capitalization – US$100 million. This will have probability 18% – around one in five.
ROI Exceeded by Large Amount
Unlikely (as with ruin) but we’ll be generous and say that gains are 20% of market capitalization – US$200 million. This will have probability 2% – one in fifty.
These are obviously numbers plucked out the air to provide an example, although the actual scenarios are fairly realistic. I’ve been very generous with the probabilities. Nowhere near 50% of projects deliver their ROI projection.
The expected return (ER) is easily calculated. In its most general from it is simply:
ER = V1.P1 + V2.P2 + … + Vn.Pn
The V values (V1 through to Vn) are the values associated with each probability for the n scenarios. The P values represent the probabilities associated with each scenario. For our particular example this becomes:
ER = 0.001 x (-1000) + 0.01 x (-300) + 0.289 x (-100) + 0.5 x 50 + 0.18 x 100 + 0.02 x 200
Note that the percentages have been converted to probabilities (1% is the equivalent of 0.01 probability), and the amounts are in millions of US$.
ER = -1 – 3 – 28.9 + 25 + 18 + 4
ER = 14.1
And so the return we can expect from this investment is US$14.1 million instead of the US$50 million given in the ROI calculations. The big negative factor in this calculation is moderate adversity – change the numbers and the ER will change accordingly. But I have to say that this is quite a generous treatment based on my experience.
The expected value is actually a mean value – the average you might expect. It in no way says that this will be the value, just that if you exceed it you have done well. But would the investment proposal be indulged if the ER was nearly a quarter the calculated ROI?
There are many, many reasons why ER might not be used in most organizations. Most IT projects are very political and ROI is often no more than political tool – but that’s not my problem.
Expected return is just part of the story. The other part is more directly connected with risk and is called the variance, or for those familiar with statistics the standard deviation (square root of the variance). This too is fairly easily calculated, although a spreadsheet will make the job easier. I’ve calculated the risk (standard deviation) for the above project and it turns in at a whopping US$90 million – so there is lots of risk. This should not be surprising given the track record of large IT projects.
The most valuable part of this exercise in my opinion is the awareness of risk it creates. Once this awareness is imprinted upon our psyches then we can go about finding ways to reduce risk. Phased implementation rather than the ‘big-bang’ approach most often deployed, is a very good way of reducing risk – and there are many other ways too. The above simply serves the purpose of getting a feel for likely returns and the risk involved. Software is available for simulation and project management that will do the whole thing in a much more rigorous manner.
There are many many reasons why this sort of analysis will not be used in your organization. Very few managers would willingly admit that there is a possibility that things might go badly wrong. At the inception of a project over-confidence is rife and political correctness will dictate that only positive outcomes are considered. This type of behavior is well understood and has been analyzed to death. Some industries, and notably financial services are required to do a more thorough job of analyzing risk, but as the sacking of risk analysts during the run up to the crash of 2008 illustrates that even here risk is not the most favorite of words to use in the Board room.
Nonetheless I do know of large firms that use a more formal approach to risk analysis of IT projects and they benefit accordingly. As projects grow larger and more complex so the need for a more formal (and realistic) approach to risk will be needed. However don’t hold your breath. Expect instead to see more high profile IT blunders reported in the press – and for every one that is reported there is a hundred that are not.
