Your CEO asks for a status update, and so you present her with a dashboard containing fifteen charts, tables and graphs. If your CEO is polite (unlikely) she will take a look, pretend to understand, and quickly move on. What is more likely, is that she will become a little irritated, and simply ask whether there is anything worth noting. A sentence is worth a thousand charts!
Natural language conveys information much more efficiently than tables of numbers, charts and complex dashboards, and so it makes sense to use natural language for communication. This is not just the case for business reporting, but also for communicating with customers, regulators, suppliers – in fact anyone who needs information.
Over the last year natural language generation (NLG) has suddenly become flavor of the month – and quite rightly so. Advances in artificial intelligence (AI) are driving its adoption, making communication via natural language much more feasible. The simplest example of NLG is the mail merge – filling in the blanks for individual names and addresses. The next step up is to create templates where details can be filled in using various manual rules (if age > 50 offer extra 10% discount etc). This is clearly a useful form of automation, but there isn’t much intelligence added to the process. NLG in its current form can do much more than this, and specifically it can determine which facts are important and which convey almost no additional information – and in many cases this is exactly what we need.
NLG should not be confused with natural language processing (NLP). This is essentially concerned with extracting meaning from text – concepts, entities, sentiment etc. NLG creates text from data – and we have no shortage of data, but we do have a severe shortage of meaningful information. This is why NLG is becoming much more important than we could have imagined, even just a few years ago. To the credit of Gartner, they have been saying for several years that NLG will rapidly assume greater importance.
The applications of NLG are endless. It has seen rapid uptake in financial services, insurance and retail. NLG can help firms in financial services feed information to regulators, and it can also help them communicate more effectively with customers. Systems are also being created for medical professionals so that the most pressing patient data can be communicated in a speedy manner. However, one of the most significant developments is the uptake of NLG by business intelligence (BI) vendors, including Qlik, Sisense, Tableau, Spotfire and even Information Builders. While we have all been made to believe that staring at charts and graphs is a good way to extract information from data, the reality is often quite different. Advanced NLG takes the data used in various visual artifacts and creates a meaningful language based dialogue. It will identify significant trends, outliers, threshold breaches – and so on. And another very important idea that comes with NLG is that analysis should be top-down. We specify what it is we are interested in, rather than the bottom-up approach that has become dominant. In other words ’this is what I want to know’ rather than ‘tell me what you know’.
Of course we are just at the start of this journey. Adding AI (in the form of NLG) solves many issues that I outlined in ‘Why BI Needs AI’ – and not least the human biases that mean we see what we want to see, rather than what we need to see. We can expect that BI will become increasingly automated, and NLG will be a major component in this. In fact we could question which medium will become the primary output from data analysis – visuals or language. My money is on language, particularly when business people start to realize that a sentence really is worth a thousand charts. If you can pick up your phone and it says to you ‘Sales this month are 2% greater than same period last year, with x,y,z products doing particularly well and a, b, c products badly lagging’, then the case will be closed.
Vendors in the space include:
- Arria – cloud, hosted or embedded NLG
- Automated Insights – Wordsmith NLG used by several BI vendors and support for Excel data.
- LinguaStat – specifically targeted at automated landing pages for customers.
- OnlyBoth Benchmarking Engine – benchmarking customers, suppliers … and creation of tailored messaging.
- Narrative Science – sophisticated and used by many BI vendors.
- Yseop – two products – Savvy for use in BI and Compose for general purpose NLG.