Oh no, not another article on big data, I hear you say. Breath out – it’s about something more fundamental. There are two ways of using data. The first, typified by big data, is to collect enormous amounts of the stuff, believing that a few gold nuggets might be found in the millions of tons of rubble. Some fundamental assumptions made here include the notion that there is some level of continuity in the behavior of markets, customers, competitors, suppliers, and other entities that make up a business environment. If there was no continuity, or very little, these petabytes of data would be wholly useless. So a sane person would only invest in big data once it was established that their business operated in an environment that displayed a high degree of continuity. No sudden jumps, no wild swings of customer sentiment, no sudden appearance of competitors with disruptive products – and so on. In your dreams. If there is one thing we know, business is becoming more volatile. Blame globalization and technology in the main.
There is also an underlying assumption that the analysis of big data is just a bigger version of traditional analysis. It isn’t. Problems such as the power law and the long tail effect (to name just two) make big data analytics a very high risk proposition, and particularly if these things are not understood. I’ve written about it in this article – Create a Big Data Analytics Benchmark. So big data analytics is founded on the assumption that what has happened in the past will be relevant to the future. If you believe this, then big data analytics is the way to go, provided the dangers lurking in big data are well understood.
The other approach a business might take to data, is to believe that the past has little bearing on the present, and to handle events as they happen. In other words deal with customers, suppliers, competitors, partners, and so on, in near real-time. This is why the Internet of Things (IoT) approach is so important. It’s a wholly different world view, and one that seems to be gaining some momentum. In a report titled ‘The Internet of Things Market’, O’Reilly found a much greater adoption rate for IoT technologies than that for big data. This report trawled the Internet for relevant data on hiring patterns, blog articles, forums, SEC filings, and much more. The results from Google Trends for example show that interest in IoT eclipsed big data in 2016 – big data remaining flat since 2014.
The implications of a move to near real-time business – something implied to some extent in IoT, are quite disturbing. It means we would need a real-time infrastructure, and methods of dealing with real-time events. Traditional analysis isn’t going to hack it. Artificial intelligence (AI) on the other hand, becomes very relevant. AI is primarily concerned with creating agents (things that do something), that take input from the environment, process it, and then initiate some action to try and maximize some performance measure (revenue or profit usually). It can be characterized as a real-time feedback and control environment. Human beings just cannot operate at the speed that will be required – intelligent agents can.
So here is a view of the future – and it may only be five years away. Businesses will increasingly operate in real-time, collecting streams of data, employing intelligent agents to respond to changing circumstances in near real-time, and using big data simply as a low cost archiving mechanism. If we listen to people like the Bank of England Governor, then AI will displace many people in the workplace. If we listen to suppliers, then it will just allow people to concentrate on higher value work. I don’t think so. Possibly in some cases, but we are facing quite a terrifying revolution in the workplace, and no one will be safe (not even technology bloggers). Hitachi already use AI managers, and a firm in Finland has an AI on the Board to advise.
IoT is the necessary complement to AI. Together they will transform work, leisure, home life, medical care – and a million other things. It never was about more data. It was always going to be about ‘now’ data.
The O’Reilly report can be downloaded for free from Talend.