The only reason to indulge in analysis is to get a handle on the future. While business intelligence is positioned as a descriptive and diagnostic activity, we only do these things so we can modify future behaviors. Someone who is simply interested in looking through the rear view mirror for its own sake probably isn’t going to last all that long.
What is lurking in the back of most peoples’ minds when they conduct analysis is the desire to catch any changes in trend early in the game. During the early stages of any change in the status quo the signals are very weak, drowned out by random noise and the prevailing trends. In reality the vast majority of business managers would rather that things stay as they are, and they will unconsciously (or even consciously) dismiss any anomalies in the data. Weak signal analysis is the deliberate attempt to identify meaningful changes in an ocean of noisy data.
There are problems associated with weak signal analysis, which is why most organizations are hopeless at detecting early signs of change. Company culture is often based on a “steady as she goes” mentality, where what has worked in the past will continue to work in the future. Anyone suggesting that green widgets with pink spots might be less popular in the future and that there are early signs that yellow widgets with blue spots are on the ascendant, will likely be greeted as an upstart. Business managers typically do not want change – it means uncertainty and risk.
If a business is going to indulge in weak signal analysis it needs to put the activity outside the normal run of the business. Delphi techniques are often employed, where a team of external experts are canvassed for their opinions on relevant matters. One might expect that an impartial group of experts should come up with some level of consensus – and sometimes they do. However, there is a gremlin hidden in this strategy. Who picks the experts? It cannot be someone from the management team – they will have their own agenda. This is a real stumbling block in many of these projects, and may be an intractable problem. It is also the case that a group of experts often cannot form a consensus.
We could also call upon technology. We can mine huge quantities of data searching for tends and the like. But there is a problem here too – it’s historical data, and because it’s historical analysis will throw up prevailing dominant trends. Weak signals are likely to be treated as noise. Most analysis treats outliers and aberrant data as though they are something to be avoided, when in reality they may hold the keys to the kingdom. If we are going to use data mining and machine learning technologies then we need to change the emphasis, taking interest in the unusual and blotting out the expected.
As I’ve already mentioned, the production machine that represents your business has no real interest in weak signals that might reveal an imminent change. And in any case who will tell your Chief Marketing Office that his brilliant innovation of coloring the widgets green and pink is just about to fall off a cliff. The group of people that perform weak signal analysis need the ear of the person at the top of the business and they need to be insulated from various political nonsense that infects all businesses.Is it likely to happen? In the main no, which is why most established businesses will succeed or fail based entirely on whether the winds of fortune are blowing in their direction.