Who shall be the Big Data King?


Perhaps the only benefit of being involved in the IT industry for a protracted period is that things repeat, and as a result it becomes fairly straightforward to estimate how trends might develop. When relational database was introduced in the early eighties, it became the province of various geeks (of which I was one), who thought they were really cool because they knew about data normalization (third normal form and all that). Leap forward 30 years and today we have data scientists strutting their stuff because they know what a support vector machine is (I thought it was a piece of construction equipment when I first came across it), and new database architectures capable of holding vast amounts of data, and processing it at speed.

Both eras have one thing in common. There was a revolution in the way we thought about data, and the initiative lay with the geeks. Such is the only way a new technology can be incorporated into business life. However there is a cycle this type of technology goes through, and we are still firmly at the first stage in a three step process.

  1. First of all we have the era of fascination. The technology is new, it promises much, and in true style often fails to deliver. Business management authorize budget, because everyone else seems to be authorizing budget, but generally speaking the geeks rule the roost.
  2. This is quickly followed by a period of fragmentation. Every department wants its own solutions, and at an enterprise level nothing is optimized, and systems often do not communicate with each other very well.
  3. The third stage is that of consolidation. This is where a few suppliers (typically no more than three) offer large businesses a true enterprise solution that exploits the technology which once was the exclusive domain of technicians. It creates a new generation of super-suppliers – the new giants that dominate a particular era in the evolution of information technology.

If you want a really good example of this we need look no further than Oracle. It came to market with a relational database management system (RDBMS) in the late 70s, and competed with the likes of Ingres, Informix and IBM’s own DB2. In my opinion Oracle was not the best of the breed, but it didn’t really matter. The company increasing sold a ‘strategic’ solution (whatever that means), but it was music to the ear of senior executives in large businesses, who desperately needed to consolidate proliferating departmental applications and databases. Oracle moved into the solutions business with its ERP platform, but still provides lots of database technology, somewhat improved from the early days, and still makes a ‘strategic’ sell. Many large corporations simply hand a similarly large check to Oracle every year, and as far as the senior management is concerned, that is one problem less to deal with.

So here we are with another data revolution – the big data revolution. Lots of people are playing, and some are even finding useful employment for the technology – but it’s all very volatile. Machine learning, predictive analytics, IoT data, and every other thing that has become vogue, is pitched at just one thing – making better decisions. And just as ERP systems replaced fragmented departmental systems, so businesses will start to look for a decisioning platform (for the want of a better name) to do all the things businesses need to do with smart applications – churn reduction, fraud detection, marketing spend optimization, predictive maintenance, and many other applications that are specific to particular industries. So in a word – business managers will be looking for solutions in the context of an integrated platform.

The candidates that might assume this role have to come from the data infrastructure players – MapR, Cloudera, and others. I have my own opinion where the next super-suppliers might come from, but I’m not in the investment advice business. The one thing I am certain of, is that in ten years from now, large businesses will be repeating what they did with ERP systems, and will simply pick a supplier for a ‘strategic’ solution. This will include standard decisioning apps, but also a framework for bespoke applications, and of course a data infrastructure.

The flip side of this is that the super-suppliers will grow exponentially large, while smaller players are acquired to form part of a product portfolio, and others will simply die a slow death. Such is life.