Fat Free Guide to What is Business Intelligence?
Business intelligence is the eyes and ears of your business – or at least it should be. We use business intelligence technologies to tell us what is happening in our business and to diagnose any problems. In fact these are the two main roles for business intelligence – diagnosis and description. So for example, we might run a monthly sales report simply to tell us the state of play with sales, or it might be used to try and diagnose sales problems – drilling down into the data to see exactly where the problems are occurring.
The best way to describe the scope of business intelligence is to trace its history. This goes back a long way, to the scheduled reports that described transaction based activity for a preceding period – the previous month say. Even today most reports are based on transaction data – sales, purchasing, accounting, human resources, production etc. The thousands and millions of individual transactions mean nothing until they are summarised and sorted in some way – and so reporting was borne.
This was the state of play in many large organisations for a couple of decades – but it was inadequate in a number of ways. Firstly the format of the reports was fixed, and any changes required IT to write new programs, and typically this would take weeks, and usually months. So business managers became frustrated, and IT was seen as a barrier to the information they needed. Not only were fixed reports inflexible, but the tabular format was not conducive to digging down into detail. A manager might want to look at sales by region, by product, by sales person, and navigating dozens or hundreds of printed paper was time consuming and not very productive.
The Data Warehouse
The in the late1980s a new technology came to market – the data warehouse. Traditional reporting ran directly against the transaction databases, and since these were usually busy dealing with transactions, it wasn’t practical to run reports at the same time. The data warehouse was effectively a copy of the transaction database, but formatted for reporting. New exotic names came into being – the star schema – reflecting a new organisation of data aimed exclusively at supporting reporting and query needs. This enabled the business to generate reports when needed, rather than on a regular schedule, and provided much more flexibility in the format. And so if a manager wanted to see sales by region, by sales person, it was a relatively simple matter to do so. If this wasn’t adequate then maybe sales by region, by product would give more information. While data warehouses provided much more flexibility, they still had to be pre-formatted in a way that anticipated how they would be used. If an unusual request was made, the data warehouse might not be capable of satisfying it.
Things moved on further with the appearance of Online Analytical Processing (OLAP) technologies. They used the term ‘dimension’ to mean a particular feature that might be of interest – a product, region, period, sales person – and so on. Once again they allowed greater reporting and query flexibility, and supported multi-dimensional analysis. Three basic operations are supported – consolidation (i.e. summaries), drill-down (greater detail), and slicing and dicing (extracting specific sub-sets of data). This was another step forward, but again, unless the data were formatted in the OLAP database in the correct way it might not be possible to create reports in the format needed.
So it is easy to see that the primary driver in all of this is greater flexibility and less latency. Data warehouses and OLAP databases are still important resources in many businesses, but things are moving on. The history of business intelligence is littered with attempts to provide visual information in the form of charts, graphs, tables and any other visual constructs which might aid understanding. Executive Information Systems (EIS) was one such wave of enthusiasm, but ultimately these systems could only display the data they had access to, and in the formats supported by the databases.
Developments in hardware are now enabling a wholly different approach to business intelligence. Whereas a decade ago a desktop computer might be able to address just a few megabytes of memory, the widespread adoption of 64 bit architectures, and the falling price of memory means that tens of gigabytes are now common. Servers can now be configured with terabytes of memory. This changes the business intelligence scene completely.
Business intelligence today is becoming synonymous with in-memory databases. Developments in database technology also mean that memory is used much more effectively. The column oriented, in-memory, highly compressed databases are ideal for ad-hoc reports and queries, and the term self-service business intelligence has become commonplace. And so a humble laptop is now capable of processing queries that only a decade ago would have required a large data warehouse.
Hand-in-hand with the ability to process data in-memory, has been the emergence of highly visual environments such as those provided by Tableau, Spotfire, Qlik and many other suppliers. Today it is quite easy to create business dashboards, which consist of several charts and tables, and to modify them on the fly. It’s a game changer, putting power directly into the hands of the people who need the information.
So to go back to the original question – What is Business Intelligence? We’ve seen that its main use is for descriptive and diagnostic purposes – to understand what has happened, and why it happened in that way. We have also seen that over several decades we have gone from fixed format regular reporting using tabular output, to ad-hoc, anytime, any format, highly visual reporting and querying. The obvious question is – what next? Well there are a number of weaknesses associated with current business intelligence tools, and the main one is interpretation. Give a chart to ten people and you will probably get ten different views on what it means. And so it would be useful if business intelligence tools could help with interpretation – be smart instead of dumb. It would be extremely useful if a chart showing sales by period could also advise on the probability there is a real trend in place, instead of leaving it to the users to reach their own (possibly wrong) conclusions. The other, very real danger of the new breed of highly visual business intelligence tools, is that they allow us to make mistakes much faster. When there was some level of latency involved in getting a report, people had time to consider and reflect. Now however, it is just too easy to jump to conclusions, and in fact the suppliers of data visualisation platforms promote the notion that important decisions might only take a few minutes to be reached.
Business intelligence is always a look in the rear-view mirror and specifically serves the needs of business people who want to understand what has happened in the business. Other forms of analytics are appearing, and specifically predictive analytics, where patterns are found in data which allow prediction of future outcomes, prescriptive analytics which allow optimisation of resources, and event based analytics (complex event processing particularly) that looks out for patterns and events in real time, and initiate processeses based on them. Business intelligence will ultimately need to interface with these other modes of analysis.
Right now we are experiencing the start of the golden age of business intelligence. It is becoming democratised, more powerful, and easier to use, after decades of effort to make it the way we all want it to be.