Underwriting with Uncertainty: We're using ML wrong in underwriting.

Underwriting with Uncertainty: We're using ML wrong in underwriting.

There are a few more key topics I would like to share from my book... I believe one of the biggest errors in using Machine Learning models is not considering forecast uncertainty in underwriting.

We're still thinking in a regression world. If you compute the forecast uncertainty versus score for origination scores, you typically see a smooth increase as you move toward the low data extremes. ML doesn't work this way. The great advantage of ML is finding pockets of predictability. However, the regions outside those pockets have much lower predictability. In other words, forecast accuracy does not vary smoothly with score. In fact, it can vary widely for accounts with the same final score because of the unique attributes creating that score.

Underwriting systems ignore uncertainty, because they are built for a regression world. When deploying ML models, you really need to know whether the model is confident in the score. Specifically, uncertainty should be another dimension in your decision matrix so that the product offering can be changed based upon forecast uncertainty.

We have precedence for this: thin file / thick file. We all know that a bureau score is much more uncertain for a thin file than a thick file, and we adjust the product offering accordingly. As users of ML, we need to realize that this variation in uncertainty exists for every estimated score!

The long version of this idea can be found in my book. http://tiny.cc/tqf9001

Joseph Breeden
Posted on LinkedIn