A recent article in the Financial Times outlines some of the significant dangers associated with big data. I’ve written a number of articles on a similar theme, the latest of which outlines why predictive models on big data (or any data) might just lead us up the garden path. We are in the middle of a feeding frenzy – millions of managers, consultants and technicians eager to adorn their resumés with ‘big data’, and so it is unlikely that a measure of sobriety is going to be particularly welcome – something akin to a skunk at a party.
Despite the claims of everyone from your friendly neighborhood big data technology salesman, to documentaries on TV showing us how big data works its magic, the reality is that the analysis of data is a tricky business, and even trickier when the volumes and diversity increase – it’s so much easier to find fool’s gold – the insights that are not insights, but just mirages in the data with no representation in reality. There are lots of technical names for this phenomenon, but this doesn’t help us sort out reality from big data fiction.
As with all booms, in this case the big data boom, there will be a bust. This is not to say that big data and analytics have no value – quite the opposite. But as always with technology we overestimate short term impact and underestimate long term impact. We are probably not all that far away from the big data bubble bursting – there are already reports emerging that things are not so rosy. I’m thinking here of IBM’s Watson and it’s less than glowing reviews by some commentators.
Make no mistake however, the analysis of data will change most things in business and other areas of life. The big data boom will be seen as nothing more than a very successful marketing campaign by the IT industry, and eventually the term ‘big data’ will itself become obsolete – it’s just data. What we do with it is the real issue.