Why BI Needs AI
Business Intelligence busies itself with descriptive and diagnostic analysis in the main. Descriptive analysis is essentially a look in the rear view mirror, and diagnostic analysis consists mainly of exploring data to see why things were as they were. The current enthusiasm for visual analysis tools claims to make these tasks easier, and they often do – although not without introducing new sources of error. Despite these advances in BI technologies, analysis is still labor intensive, prone to various biases, expensive and purely retrospective.
The most important of these problems is the various biases that creep into analysis. Stephen Few, the author of Signal, is a practicing BI consultant and states that these biases mean business people are making erroneous decisions on a regular basis, simply because of misinterpretation of data. So let’s have a look at some biases.
- Seek and you will find. It is very well known that we like to find what we are looking for. So someone using a visual BI tool will cycle through dozens, if not hundreds of charts to validate their suspicions – for various reasons. One particular paper was written on this topic some ten years ago, and Bayer, the giant agrochemicals and pharmaceutical company decided to see if the claims were correct. They found that 70% of analysis could not be replicated by changing the analyst. Human biases are almost impossible to eliminate, and so maybe we need a technology that is not human to carry out the analysis.
- Representation Bias. Most charts consist of two dimensions – vertical and horizontal. And they are usually linear. That business data should conform to these assumptions is a very heavy, and often incorrect bias. So sometimes we use a log scale, but it’s the same old cartesian system we use. How about polar coordinates (maybe you remember them from high school) ? Of course it all starts to get a bit technical at this point, and so maybe we should let a machine do the thinking.
- Ghosts in the data. A chart may show that a trend is in place. But trends can show up in data that are nothing other than flukes generated from random noise. This is true of all ‘patterns’ that might show up on a chart. So some statistical validation might be useful, and because statistics can get fairly complex, once again it might be better to let a machine determine what is significant and what is not – outliers, trends, cyclical activity, and so on.
Analysis is also labor intensive, and as such becomes a significant cost. Playing endlessly with chart types, color schemes, data combinations and other parameters is time consuming, and becoming more so as we have more data sources and charting options. Some suppliers of BI tools are all in favor of your business providing these tools to as many people as possible (obviously), but then the uncomfortable question raises its head as to who will actually be doing productive work! So again, if a machine can cycle through data sources, combinations of data sources, representations, and work out what is significant and what is not, then we might save ourselves an awful lot of money and time.
And so by now you can guess that the solution to these problems will come in the form of artificial intelligence (AI) – BI needs AI. A product called BeyondCore does some of these things, and has recently been snapped up by Salesforce. I’ve also seen a few presentations from more sophisticated suppliers, willing to talk about biases and how they might be overcome through machine learning – GoodData is an example of this. So these things are not just fringe, but are real issues that need a solution – and the industry is already making overtures in this direction.
When we consider that AI is starting to affect almost every aspect of business life – marketing, sales, business process improvement , and even AI managers (Hitachi now uses AI managers in some roles), then it seems fairly obvious that the technology will find its way into BI. I think we can expect this to happen fairly quickly – 3 to 5 years. Initially it will come in the form of AI aided BI, but eventually it will do most of the task without human intervention – I’m sorry to say.