Most large organizations have spent the last half century or so accumulating copious amounts of data on every conceivable aspect of their operation. The cost of this acquisition has been enormous – much more than many business managers realize. Every detail of every customer has been input by someone. Every contract has been assembled and processed by someone. The creation of every piece of marketing collateral has probably involved several people. These are all information costs, and for companies in service industries these costs may be as high as eighty per cent of revenue. The largest component of information costs is labour. Most managers do not think of it in this way. Labour costs belong to sales, finance, design, production and so on. The common thread however for all these labour costs is that the main activity involves the creation and manipulation of information, and as such they are predominantly information costs. Unless we can get at this information it has no value at all.
Just consider any large corporation operating in a service industry – take insurance for example. Labour costs associated with information creation and manipulation will be around seventy per cent of revenues (and this is being conservative). If this company had average annual revenues of ten billion dollars over the last ten years it is easy to see that its information assets cost seventy billion dollars to acquire, manage and manipulate. Surely it would be worth investing in people, methods and technology to get at and derive some value from this information?
We have had no problem at all spending on systems and technology to acquire and manage information. The four billion dollars a year that a large bank might spend on IT primarily supports the systems that acquire and manage information – very little of it will be spent on information search and deriving value from this very expensive commodity. There is a strange kind of logic here. We accept that acquiring and managing information will be expensive, and that this is justified because it supports day-to-day operational activity. On the other hand we have something of a miserly attitude to the other half of the equation – getting the information out and deriving some value from it.
We only do three things with information. We create it, We manage it. We search it. Search is at least a third of this equation, and the most important third because it concerns risk management and value. The accumulation of information has accelerated in recent years, and looks set to accelerate even more. It is time for organizations of all sizes to get serious about search if they do not want risks to escalate and their most precious of all assets to simply decay in value with the passage of time. To drive the message home just consider how useful the Internet would be if we could not search it. Hundreds of millions of people and organizations creating information, but with no means of access. Many large organizations have created a private Internet without a search engine.
As if this was not bad enough the situation is made worse – much worse, by the fact that individuals, groups and organizations carry knowledge and information around with them that they do not, and often cannot share. If the terabytes of data sitting in corporate databases are underutilized, the knowledge of possibly thousands of individuals in your organization is a resource that is not only untapped, but is acquired and utilized with great inefficiency. Knowledge management has been touted as a means of reducing these inefficiencies, but coercion is not an effective mechanism for getting people to share what makes them valuable to your organization. This situation can be eased by creating a common language which everyone can share, and providing mechanisms to interrogate explicit expressions of knowledge in documents, emails and any other files created by knowledge workers.
The economics of information search is simple enough. We need to derive more value from search than it costs us to facilitate and execute. The problem in all of this is putting value on the results of search. Many managers complain that they cannot access the information they need for sound decision making. Effective search will help managers avoid bad decisions – but putting a value on these decisions is usually very difficult. Information workers spend anywhere up to thirty per cent of their time searching for information and many claim that a third of this time is totally unproductive. If effective search could reduce this we should see more productive workers and greater throughout – but who will measure it? Many organizations have been forced to do something about search because of the need to comply with various regulations. Document, email and database search is usually what is needed, and a search facility is installed to make sure that the organization does not fall foul of the various regulators – but it does little for productivity, decision making, intelligence gathering and opportunity identification within the organization.
So what does an organization need to do to get serious about search? A new or enhanced set of skills is needed. We need information professionals who understand how to categorize information and we need them to work with technicians implementing search engine technology. We also need structured dialogue with knowledge workers and management to establish a common understanding of terms so that search means the same thing to all people within an organization. Search really is not about just buying a black box and accepting whatever it throws up with gratitude – although this might be a good place to start just to experience the shortcomings and establish what else is needed.
A word of caution is needed here. It is just too easy to get side-tracked by long technology and methods detours. Someone needs to make sure that consulting, the setting up of various categories and structures, and the implementation of the search technology remains a pragmatic exercise with well-defined milestones. Attempts to create information categories that are all embracing or just too precise are a fruitless exercise that should be avoided.
Perhaps no other technology is so affected by smoke and mirrors as Enterprise Search. The jargon is formidable and the claims often outrageous. Not surprisingly many users of search technologies are dissatisfied and confused – sure signs of a technology that is nowhere near maturity. On the one hand we have the uber-high tech offerings from companies such as Autonomy who pour derision on the familiar Google type keyword search, and on the other hand practical, functional offerings from new startups that will do eighty per cent of what many organizations need with twenty per cent of the fuss. Clearly it all depends on what you want and what you believe the technology can deliver. Despite the aggressive claims of many of the vendors there are actually no free lunches. If you want more out of your search technology you will have to put more into it.
It is worth spending a few moments reflecting on how we got into this mess – because that is what it is. For decades we have merrily been accumulating data, and more recently this data has become richer and more complex. Structured databases, email, electronic documents, scanned documents, html, graphics, sound, video and whatever else the future delivers, all have different formats and programs to make them meaningful to human beings. The cost of producing and managing this data is formidable. Most organisations spend anywhere between two and fifteen per cent of their revenues managing data, and anywhere up to eighty per cent actually creating it. During the last ten years it has suddenly dawned upon us that maybe this expensive commodity can be used to create value and help manage risk. Unfortunately this was not thought of when we were busy creating data, and so the formats often have little or no information to help us make sense of the content. As a result we have to find ways to extract meaning from, and categorize data as an afterthought – so enterprise search is likely to be messy and less than satisfying if your expectations are too high.
Given the current state of the art organizations have three options. They can be summarised as:
- Do Nothing – and wait for the technology to mature and become cheaper.
- Buy a black box search engine and just accept that it will only do so much.
- Venture into the labyrinth of advanced search technologies with the attendant high cost and high risk, in an attempt to deliver something exceptional.
The first option will not be an option for many organizations, they have to comply with various requirements and a search technology makes such compliance feasible. There may also be a desire to reduce the cost of searching for information – a cost that is rising exponentially as we create more of the stuff, and as formats become more diverse. If doing nothing is not feasible then we should look at option two, and while it may seem like a lazy man’s approach to search it has many advantages. It is predicated on an understanding that the search technology will do the donkey work and people will have to do a certain amount of manual processing. Search technology will comfortably find all the documents and/or records with the words ‘returned by customer’, but finding out why might take some human intervention. This approach would largely avoid lengthy, costly technology detours into semantics, ontologies, taxonomies, tagging and other techniques to attach meaning, but it would also mean that more human effort might be required to process search results. It’s a trade-off – but it might be a good place to start. Venturing into the labyrinth should only be done by the prepared. Expect longer time scales, mistakes, greater complexity, more cost and a high on-going maintenance overhead. On the plus side you might just deliver a real advantage to your organization – but the risks are much higher