Question-Bottleneck Scoring
A whitepaper on predicting customer outcomes by asking a decision model plain-language questions and weighting its calibrated answers: human-quality judgment calls at the speed and cost of code.
Abstract
Account decisions, such as whether a new account is high risk or fraudulent, usually rely on rules and gradient-boosted decision trees (GBDTs) over hundreds of engineered features, with uncertain cases sent to human reviewers. We present Question-Bottleneck Scoring (QBS), which breaks a decision into plain-language questions, such as whether the email, phone and address belong to one real person. Grayson, Finic’s decision model, answers each with a calibrated probability, and a logistic layer weights the answers into a score.
On 17,624 real accounts from a leading publicly listed financial services company, QBS, fitted on 2,000 labeled accounts with six questions, matched the company’s in-product GBDT model (300+ features). It was also more accurate where the GBDT was uncertain, raising above-chance discrimination there by 29%, and combining the two beat either alone. QBS’s decisions were 99.4% as accurate as human reviewers’, in under a second compared to an average of 7 minutes, and QBS could decide half the manual queue with no change in outcomes.
QBS also carries over from judging how likely an account is to be bad to judging how likely it is to be good. Predicting from a user’s first week who would become a power user, it edged out a GBDT trained on the same records (AUC 0.796 vs 0.790), and the two combined did better still (0.805).
Key results
- above-chance discrimination vs the production GBDT, in its uncertain segment (AUC 0.689 vs 0.647)
- +29%
- bad-account rate in the half QBS would approve, vs the accounts human reviewers approved
- −37%
- faster than manual review: 0.48 s median vs 7 minutes
- ~870×
- labeled accounts used to fit the score: six questions, no feature engineering
- 2,000
Read the full paper
The whitepaper covers the method in enough detail to replicate it, the full comparison with the production GBDT and with human reviewers, the six questions that were chosen, when to use QBS rather than a GBDT (and how to combine them), the power-user results, and the caveats.
A reference implementation of all four steps is on GitHub at finic-ai/question-bottleneck-scoring, under the MIT license.
Question-Bottleneck Scoring
Whitepaper · PDF · 11 pages
Frontier AI to stop fraud, abuse, and financial crimes.
Grayson decides in real time. Finic's agents investigate the rest.