Can a decision model make loan default prediction more accurate?

Traditional credit models can't read

Part 2 of 2. This series tests decision models on public data. Part 1 covered auto insurance fraud; this part covers loan default.

Lenders predict default with traditional machine-learning models built on structured fields: income, debt-to-income, credit history, the lender's own grade. But borrowers also explain, in their own words, why they need the money, and research has shown those words carry real signal about who will repay. A traditional model can't use any of it. It can count the characters in a description, but it can't read it.3

Grayson can. It reads the whole application, including the borrower's description, and answers plain-language underwriter questions about it with calibrated probabilities, without needing any of your labels. We wanted to know how much that adds to a traditional credit model.4

What is a decision model?

Lenders today rely on two kinds of AI, and each has a blind spot.

• ✣Traditional machine-learning models learn patterns from your own labeled history. They are fast and cheap, but they only see structured fields, and they need a lot of past outcomes to learn from.
• ✣Large language models, such as the ones behind ChatGPT and Claude, can read anything and reason about it. But they answer in text, they are slow and expensive at application volume, and the confidence they express isn't calibrated, so it is hard to plug them into a scoring system.
• ✣A decision model combines the two. Like a large language model, it reads any context, including the borrower's own words. Like a traditional model, it returns a number: a calibrated probability for each question you ask, fast enough to sit in a real-time decision.

Grayson is Finic's decision model. Because its answers are calibrated probabilities, they can be fed straight into the credit model you already run, which is exactly what we tested.

The test

We used Lending Club's public loan data: about 126,000 completed personal loans issued between 2007 and 2016 whose application includes the borrower's own description. A loan counts as a default if it was charged off.2

We split the loans by issue date, the way a lender would use them, and trained a traditional machine-learning model, the kind most lenders use, on the earliest 70%. We wrote 27 underwriter questions, such as “Is the stated income plausible for this job title?” and “Does the borrower describe current financial hardship?”, and sent 2,000 loans from the next period and 3,000 loans from the latest period to Grayson and, for comparison, to TypeSafe's Jev. Neither model ever saw whether a loan was repaid.5

On the earlier loans, we picked the answers that added most to the traditional model and combined them with its score. For comparison, we also trained a traditional text model (TF-IDF) on the borrowers' own words and combined it with the credit model the same way. Then we scored everything once on the later loans:

• ✣The traditional model on the application fields alone
• ✣The traditional model plus the TF-IDF text model
• ✣The traditional model plus Jev's answers
• ✣The traditional model plus Grayson's answers

Grayson made the traditional model more accurate

Adding Grayson raised the traditional model's accuracy score (ROC-AUC) from 0.681 to 0.689. It was the only improvement whose 95% confidence interval stayed above zero. Jev's gain was smaller and within the noise, and so was the text model's.

Why Grayson helps

Of the 27 questions, the one that added the most to the traditional model was “Does the borrower describe current financial hardship?”. That answer comes straight from the borrower's own description, the part of the application a traditional model can't read. Grayson turns it into a signal the model can use. A traditional text model trained on the same words picks up which words go with default, but it added only about half as much as Grayson.

The ROI, from real loan outcomes

Every loan in this data was one Lending Club approved. Suppose the lender had used the model to decline the riskiest 10% of them. A better ranking means more of the declined loans are ones that would have been charged off, and fewer are good loans turned away. Each such swap is worth $8,322: the average $5,114 lost on a charged-off loan plus the average $3,207 earned on a repaid one, both taken from the loans' actual repayments.

Grayson added the most of the three: about 19 fewer defaults and $155,000 per 10,000 loans.

Decision models are not interchangeable

Grayson and Jev saw the same applications and the same questions, and the selection even picked the same question for both. Grayson's answers were worth more. What a decision model brings depends on how well it reads the case in front of it, especially the parts that are written in plain language.

Run it yourself

Everything is on GitHub: the notebook, the 27 questions, and every Grayson and Jev answer. The notebook downloads the loan data from Kaggle, so you need a free Kaggle API token, but no Grayson or Jev keys.1

git clone https://github.com/finic-ai/fraud-benchmark-lending-club
cd fraud-benchmark-lending-club
python3.12 -m venv .venv && .venv/bin/pip install -r requirements.txt
cp .env.example .env   # add KAGGLE_API_TOKEN
.venv/bin/jupyter nbconvert --to notebook --execute --inplace benchmark.ipynb