Predictive models are used throughout the financial services industry to help make more accurate decisions in the face of uncertainty. From fraud detection through to credit scoring and quantitative trading systems, predictive models are employed in large numbers to aid decision making. Various factors such as big data, more powerful and cheaper hardware, the development of more sophisticated algorithms, and competitive need, means that predictive models will become more widely used and more complex.
On the face of it, this scenario seems rather harmless — but it’s also fraught with risk.. The development and use of predictive models is a process full of dangers and traps. It all starts with data and the selection of data samples. These are often biased in some way, and resulting models inaccurate. Or some of the data may be ‘dirty’, causing unstable model types such as decision trees to create flawed models. Most of the algorithms used to create predictive models are quite proficient at interpreting random noise as meaningful patterns. There are ways to minimise the risk of this happening but none of them is fool proof. Many other dangers exist, and BASEL and Fed/OCC regulation in this domain is targeted at addressing them.
The use of models in a live environment is also a target for regulation, since the prevailing conditions that existed when a model was built are unlikely to persist indefinitely into the future. And so there is a requirement to check issues such as population stability on an on-going basis
It should by now be apparent that the development and use of predictive models is a major undertaking, particularly when many firms in the financial services industry use hundreds of them. Of course they wouldn’t be used unless there were significant benefits to be realised, and lift of just a few per cent in a targeted marketing activity will usually translate to significant financial gain. The claims of some technology suppliers of improvements measured in the hundreds of per cent can be dismissed without second thought, and the reality of using predictive models is somewhat more modest; although ‘tens’ of per cent are not uncommon.
A particularly prominent requirement in financial services is that models be transparent, and that humans can understand the reasoning behind a decision. This deters use of especially complex models such as neural networks, and favours more easily understood methods like scorecards and decision trees.. Here again the regulators are making very specific demands for transparency and ease of understanding.
The ubiquity of predictive models and the requirements of the regulators mean that a model management and administration framework needs to be established. This needs to embrace many processes that deal with model origination, validation, production checking, data quality and sample design, transparency, and so on. It is an illusion to think that all the models currently being used in an organisation are accurate and useful. Left unchecked, inevitably there will be rogue models that cause financial harm and breach regulatory requirements. The primary aim of a model management framework is to address these issues, and through the information such a framework would generate reduce the risk associated with the use of predictive technologies. This is a pressing need as the volume and complexity of models increases, with regulators insisting on processes and information to reduce risk.
Finally we should not forget that despite all the glamour associated with the mathematics, science, technology and techniques behind predictive models, it is essential that human beings, and particularly domain experts, can sanitise their use. Just as models are used to reduce uncertainties associated with decision-making, so the models themselves have uncertainties associated with them which, through model management and regulatory adherence, can be minimised to acceptable levels.
Next article in this series is Financial Regulation and Predictive Models
Previous article in this series is The Regulatory Opportunity in Financial Services