Banks model risk, but how risky are their models? Chief risk officers are devoting more and more resources to model risk management (MRM) as regulators sharpen their focus on systems and governance. A global standard has emerged, but there are still significant differences between financial institutions based on size, regulatory environment and geography.
“Statisticians, like artists, have a bad habit of falling in love with their models.” So warned George E.P. Box, one of the great statistical minds of the 20th century.
By the time the 21st century dawned, you could say the same of bankers. For them, the dazzling appeal of one model in particular— the Gaussian copula function— was to prove disastrous. It enabled bond investors and mortgage lenders to price risk with a simple correlation number. They loved it, despite the alarm bells its inventor, David X. Li, was beginning to ring when he said in 2005: “The most dangerous part is when people believe everything coming out of it”. The financial crisis created in part by the overreliance on, and misapplication of, the Gaussian copula function prompted a massive overhaul of regulation and risk management that continues to evolve today.
Models underpin the modern financial system. They’re used for business strategies, risk measurement, the valuation of exposures and to assess the adequacy of capital. Model risk is the potential for adverse consequences created by the use of the models themselves. Model risk management enables banks to understand and handle that risk, but only recently has it become a board-level issue.
Since 2011 the global standard for MRM has been defined by the US Office of the Comptroller of the Currency’s SR 11-7 Guidance on Model Risk Management. It describes model risk as occurring “primarily for two reasons: (1) a model may have fundamental errors and produce inaccurate outputs when viewed against its design objective and intended uses; (2) a model may be used incorrectly or inappropriately, or there may be a misunderstanding about its limitations and assumptions”.
Modern MRM is hugely ambitious; it aspires to identify and handle the entire universe of risk around a model. It is an enterprise-wide activity that includes:
- a complete and structured inventory of models and their main assumptions,
- a view of model risk throughout the model lifecycle,
- a coordinating MRM function with model validation as a part thereof, and
- all four lines of defense (business units, risk management, internal audit, external audit/regulators).
In general, banks see MRM as a holistic practice best served by a distributed approach. An overarching model-risk or model-governance function may well track models throughout their lifecycles. But as the model passes through that lifecycle, a number of different teams are involved in the process, with precise responsibilities allocated differently from bank to bank. The developers of models, for example, are more likely to be involved in estimating uncertainty and benchmarking against alternative models, while independent validators check assumptions and implementation. So while governance is centralised, the practice of model risk management is considered too complex an activity to be allocated to a single function. It requires cooperation and coordination across the bank.
Banks need to measure risk for a whole host of reasons—for pricing, risk-appetite calculations and regulatory exercises, to name a few. To assess model risk effectively, they want to aggregate risk across the institution. Aggregation is typically viewed as a quantitative process. This is far from straightforward because models are interconnected and because the underlying metrics upon which uncertainty is measured will vary according to model type. It is difficult, for example, to aggregate the model risk associated with regulatory capital with that which arises from pricing models. However, many banks say that they do aggregate model risk, using both quantitative and qualitative metrics. When questioned more closely, though, the aggregation is rarely assigned a dollar value or performed across all model risk types. Indeed, some practitioners question the value of aggregating model risk, arguing that the techniques used may themselves create an additional tier of risk.
Determining the correct methodology and making available the necessary resources for MRM is a big challenge, and one that banks are meeting with varying degrees of success. In particular, we see significant differences according to geography, the size of the institutions and their regulatory environments. US banks are more likely than European to tell us that they are effective in managing model risk and more likely to say that they have adequately resourced it.
Enlightening comparisons are made by banks who are regulated in both the US, including the CCAR (Comprehensive Capital Analysis and Review), and by European supervisory regimes. They see the US regulators as being more prescriptive than the Europeans and say they provide fuller feedback to banks, which means that compliance is easier to demonstrate. Responding to complaints by some banks in Europe that regulatory requirements remain unclear and lacking specificity, one European regulator told us that in attempting to avoid compliance-by-tickbox they consciously aspire to a more nuanced approach than that adopted by the Fed.
There has been less recognition outside the US of model risk as a standalone risk type, but this is a gap that is closing. European regulators such as the Bank of England’s Prudential Regulation Authority (PRA), for example, take a close interest in SR 11-7 and are sharpening their focus on governance as well as the technical specifications of models.
The fact that many of the biggest banks are regulated by both the US and Europe is also pushing the standardisation of MRM. Inasmuch as effective MRM requires an enterprise-wide and consolidated function, it has to transcend geography. Hence the toughest requirements—traditionally in the US—provide a benchmark across the banks’ global operations. That has increased demand for resources in Europe as banking operations there play catchup. The number of model validations carried out by larger banks has risen dramatically. In addition, more extensive and better written documentation is required, and many are struggling to find enough experts with the necessary experience to do the job.
Nevertheless effective model risk management is now essential. While model risk can never be eliminated, it can be monitored and mitigated. The goal is knowing what you don’t know. And then getting to know it better.
This article draws on research carried out by Fintegral on behalf of members of the International Association of Credit Portfolio Managers (IACPM).