The Modelry development framework
adapted for a post-COVID assessment
The Modeling Engine
COVID-19 Assessment FAQ
The COVID-19 pandemic amounts to a real life economic scenario that is unlike any other in modern history, with several unique factors:
- Breakdown of many correlations that had been typically modeled as "stable"
- The speed of change - both on the way down as well as the recovery
- The extremely idiosyncratic behavior of unemployment, the most recurring variable among banking stress modes
All of this implies that models calibrated to pre-pandemic data are likely to significantly lose predictive power and fail to capture secular post-pandemic shifts.
While all models should be reviewed to ensure they can continue to perform, some classes will experience more severe impact. Banking stress test models in particular, which typically overweigh data from the single 2008 financial crisis are particularly vulnerable as the behaviors of many key scenario variables has been drastically different.
We have developed a systematic approach and a proprietary "scoring" mechanism for each model based on a large number of input sets and parameters. This approach is made possible by the power of our platform and its dependency graph which enables computations under thousands of scenarios in a fraction of the time it would take to run them otherwise.
For the assessment, we first review the characteristics of the development data, then, based on validated rules, create a set of "scenarios" each representing a different blend of pre-, post- (and in some cases "during") pandemic data. We then "score" the model performance under each such scenario against a set of standardized statistical tests appropriate for that model class. The resulting scores, with some additional expert judgement form the basis of the assessment.
In most cases we are able to connect clients' models to our platforms via wrappers or other techniques, and while there is some additional work involved it will typically still be a much faster process than a manual, unstructured review of each model one at a time.
The assessment classifies failing outcomes into three categories, in increasing order of severity:
- The model fails some statistical tests but remains fundamentally sound - this is comparable to failing performance monitoring which typically requires a recalibration or some other routine type of adjustment
- The drivers of the model no longer work and a full redevelopment is most likely the right approach (additional analysis may be needed in these cases to determine the appropriate action)
- The regime for this particular model has changed so drastically that an entirely new approach may be necessary, using different methodologies, data and/or segmentations
While the amount of work will vary substantially depending on the number, complexity and specific client implementations of various models, our systematic approach and technology specifically designed for the fast execution of large numbers of scenarios is certain to make the process as efficient as it can be.
Before any commitment, we perform an initial analysis of the clients' model inventory and implementation - typically a few days to a couple of weeks - at the end of which we are better able to estimate the effort required for the full assessment.