“So how can I tell if my engine is up to scratch?”

For most banks, selecting and purchasing a Basel II Credit Risk engine will be a relatively straightforward process. Unfortunately, the task of ‘validating’ the purchase is far less straightforward, in most cases takes more than a year, and in many cases, does not end in success. Many would consider themselves fortunate to cut the ribbon on a production system which could even qualify as a ‘quasi-success’.

Now there are 3 types of validation which will be crucial for all banks to perform:

At one very large bank, in order to ramp up towards validating its vendor supplied software solution, the bank’s quintessential modelling guru built a ‘mini-validation engine’ using Microsoft Excel. Now this was a very slick, very sophisticated mini-engine, but because it was built and maintained for a specific suite of test cases, it unfortunately required significant re-work on many occasions when new types of cases were added or an interpretation changed along the way.

In parallel to this, a team of 6 career bankers consumed more than a year developing and processing 600 ‘manufactured’ test cases, which were specifically aimed at compliance with the Accord, and not necessarily representative of the bank’s live data complexion. Comparing the purchased engine’s results to the mini-engine’s results uncovered HUNDREDS of bugs and design defects in its multi-million dollar vendor solution. (Yes, this same software has featured in Gartner’s ‘Magic Quadrant’ since it was launched and would be on nearly everyone’s RFP list and many of these defects were never able to be corrected because of poor underlying design choices.)

Next the bank began to pour its ‘live exposures’ into the purchased engine as well as its mini-engine. It took 3 months for the bank to acknowledge that this was an insurmountable problem because it had nearly 20 million exposures and an Excel worksheet can only hold 65,536 (or 2^16) rows, besides the fact that the exercise of comparing results between the two engines was very labour intensive and required a dizzying number of imports and exports around Excel. So the bank set out trying to establish a very small, representative set of candidate exposures per Regulated Entity, Legal Entity, Business Unit, Asset Class and Asset Subclass. They originally settled upon 52 test cases involving live exposures within the Retail and Small to Medium Corporate Bank and expanded outwards for nearly 2 years. Unfortunately, its most complex transactions existed in its Institutional Bank and many of these could never be modelled in Excel. This obviously falls short of a comprehensive, spearhead approach to testing, and in this bank’s case, it has now spent more than 3 years elapsed in testing a product which is still not in production. And because of a less than comprehensive approach, there is always the risk that their Home Regulator might arrive one morning and request a more in-depth assessment, and worse still, uncover problems which should have been detected through validation testing.

The third type of testing is performed by auditors. They are usually external. They are usually frighteningly expensive. And they usually tie up a good deal of your resources covering the same ground that you have covered yourself. But they represent a vital step towards satisfying your Home Regulator and so the trick is to supply them with the information they need as quickly and as transparently as possible. No double-handling, no long-winded explanations or education sessions, simply allow them to tap into the same tools that you have been using to assess your engine’s capability.

Now this case study depicts a bank which decided to go COTS software (Commercial Off The Shelf), but it is equally, if not more relevant for a bank which has conjured up its own solution in-house, since it will likely have to go a long way to prove to its Regulator that it knows everything there is to know about creating Basel software when several billion dollar software houses have gotten it so wrong.

But there may just be a full-scale Credit Risk engine which can also be licensed very inexpensively as your bank’s ‘validation engine’ with 3 massive benefits.

Why not talk to us at RegCap?

<--Previous                            Next-->