My 3-month hiatus has ended thanks to the Canadian long weekend. In these past few months I haven't had much time time to commit to posting new content, primarily because I started a new job. This has put my research interests on hold for the time being. So while I will likely not post as frequently as I have in…

# Model Validation

# Some Simple Measures of Forecast Accuracy

So you've built a model, the predictive plots look nice, but you want to synthesize this information down to a number. This is where measures of forecast accuracy come in. In this post we will recall some simple measures of forecast accuracy, and then I'll explain why I don't like them, unless you intend to use them to compare more…

# What is a Margin of Conservatism?

This question comes up a lot in my field of work, and unfortunately, there is no rigorous answer. Everybody seems to have a different (albeit loosely similar) definition of a margin of conservatism. However, we all need a definition, as it drives much of the model development and validation work at financial institutions. Furthermore, AIRB risk parameter estimation models require a margin of conservatism,…

# Weights of Evidence and the Information Value

In order to determine which variables demonstrate a high level of predictive power, we calculate the binned Weights of Evidence (WoE) and the associate Information Value (IV). Generally speaking, the IV provides a measure of how well an explanatory variable is able to distinguish between binary responses (e.g. "good" versus "bad") in some target variable. The idea is if a variable has…

# Age-Period-Cohort Model for Transition State Forecasting

You may be familiar with transition state models that are governed by the Age-Period-Cohort (APC) modelling framework. If you are, you've likely been witness to the recent insurgence of APC models in the literature -mainly being used to forecast mortgage and bond performance. The content related to APC modelling can be dense (literally, not figuratively) and you may find yourself confused when the the…

# Herfindahl–Hirschman Index

The Herfindahl–Hirschman Index is a measure of the size of subgroups in relation to its containing group. It is often used in econometrics to measure the market share of companies (subgroup) in relation to an industry (group). For simplicity, it can be interpreted as a measure of competition or concentration. The measure itself is quite simple , given by the expression:…

# Hypothesis Testing of Regression Coefficients

For a long time I resisted statistics (yes, the subject in general) because of what I deemed to be a lack of objectivity. It was this reason why I chose to study mathematics, and not statistics. As such, I've had little exposure to a formal statistical education, and much of my understanding stems from my own interpretation. At times I…

# Population Stability Index

One way to measure 'shifts' in the proportion of observations within subgroups is by the Population Stability Index (PSI). When a sample population is classified into various subgroups, one might want to establish whether those subgroups are stable with respect to a base population. The PSI is quantified by the formula: where and refer to the proportion of observations belonging to the subgroup…