Grayson makes XGBoost better at catching insurance fraud

XGBoost only knows what is in its columns

Most fraud teams score claims with gradient-boosted trees such as XGBoost. They are excellent at finding patterns in labeled history, but they have two gaps. They need a lot of confirmed fraud to learn from, which new products and smaller books of business don't have. And they see each field as a number, so they miss the judgment calls an experienced adjuster makes at a glance: does this coverage make sense for this car, does the timing of this claim add up?

Grayson fills those gaps. It reads a claim the way an adjuster would and answers plain-language questions about it with calibrated probabilities, without needing any of your labels. We wanted to know how much that adds to a standard XGBoost model.3

The test

We used Oracle's public CLAIMS data: 1,782 auto insurance claims from a US insurer, 923 of them fraud, with 27 fields each covering the accident, the vehicle, the policyholder and the policy.2

Every claim went to Grayson with the same eight yes/no questions, such as “Was the policyholder at fault?” and “Did the address change shortly before the claim?”. For comparison, we sent the same claims and questions to TypeSafe's Jev. Neither model ever saw which claims were fraud. Then we trained the same XGBoost model three ways:4

• ✣XGBoost on the 27 claim fields alone
• ✣XGBoost plus Jev's eight answers
• ✣XGBoost plus Grayson's eight answers

Grayson made XGBoost better

We scored each approach with 12 repeats of 5-fold cross-validation, so every claim was scored by a model that had never seen it. Adding Grayson raised XGBoost's ROC-AUC in 10 of the 12 repeats. Adding Jev didn't help.

Why Grayson helps

Grayson turns a claim's raw codes into the concepts an adjuster actually thinks in: does this policy pay for the policyholder's own car, were they at fault, did they move just before the claim? Those were the Grayson answers XGBoost relied on most. With only about 1,400 claims to learn from in each run, XGBoost struggles to assemble those concepts from raw categories on its own. Grayson hands them over ready-made.

The ROI: more fraud caught with the same team

Take an insurer with 100,000 claims a year and a typical fraud rate of about 6%, whose investigators review the riskiest claims. Ranking those claims with XGBoost + Grayson instead of XGBoost alone catches more fraud with exactly the same investigation workload. At a 10% review rate, that is about 165 more fraudulent claims caught and $410,000 saved a year.

The extra fraud caught is measured. The data has no claim amounts, so the dollar figures assume a $5,000 average claim and that half of each caught fraud's value is recovered.

This was the hardest case for a decision model

This dataset gives Grayson very little to work with. Every field is a pre-coded category, such as an age band of 26 to 30 or a vehicle price band, and there are no adjuster notes, claimant statements or other free text. Decision models do their best work on exactly that kind of narrative, subjective information.

Even so, Grayson improved a model trained directly on the insurer's own labels. On real claims data, with notes, descriptions and documents, we expect the gains to be larger.

Decision models are not interchangeable

Grayson and Jev saw identical claims and identical questions. Grayson's answers made XGBoost better; Jev's did not. What a decision model brings depends on how well it reasons about the case in front of it.

Run it yourself

Everything is on GitHub: the notebook, the eight questions, and every Grayson and Jev answer, so you can reproduce the results without API keys.1

git clone https://github.com/finic-ai/fraud-benchmark-vehicle-claims
cd fraud-benchmark-vehicle-claims
python3.12 -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/jupyter nbconvert --to notebook --execute --inplace benchmark.ipynb