Selective Prediction, Abstention, and Risk–Coverage

A model may improve accuracy from 85% to 97% after answering only half the requests. That can be valuable if the remaining cases receive a safe, timely fallback. It can also be misleading if evaluation silently drops them.

That silence has been counted. Ten influential language-model evaluations were checked one by one, drawn from the major leaderboards: GPQA, MMLU-Pro, IFEval, Omni-MATH, WildBench, BBH, MATH (L5), MuSR, SWE-bench and HLE. Nine of the ten grade strictly correct or incorrect, and give no credit at all for abstaining. Only WildBench offers partial credit for indicating uncertainty. Under that arithmetic a model that always guesses outscores an otherwise identical model that says “I don't know”. A guess can score. A refusal cannot.

That count comes from Kalai and colleagues, in a September 2025 paper called Why Language Models Hallucinate. Their remedy is a scoring instruction rather than a modelling one: “The numerous primary evaluations must be adjusted to stop penalizing abstentions when uncertain.” The fault they name is in the scoreboard, not in the model.

Selective prediction measures quality together with the fraction of cases receiving an automated answer.

A deferred queue can hold exactly the cases you least want to lose. FDA's De Novo summary for IDx-DR, granted on 11 April 2018, shows one. The pivotal study enrolled 900 participants at 10 primary care sites. Only 819 could be fully analysed — an analysable fraction of 92%, 95% CI 90–93%. The headline sensitivity of 87.4% and specificity of 89.5% are computed on those answered cases alone.

The interesting number is in the other column. For 38 participants the device returned an “insufficient image quality” output instead of a screening result. The disease was no rarer among them: “In the 38 participants with AI system insufficient image quality, the prevalence of mtmDR was 10/38 (26%), comparable to the mtmDR prevalence in the fully analyzable dataset.” Prevalence in the answered population was 23.8%. Abstaining removed the cases from the metric. It left the disease exactly where it was.

Abstention adds a pathway that changes the system. IDx-DR does not emit a binary. Its range of available device outcomes is three: mtmDR detected, mtmDR not detected, and insufficient quality. The third outcome is not an error state to be tidied away in analysis. It is a case the product still has to service, with a person attached to it.

Selective systems trade breadth for residual error, and that pair of quantities has a datable origin. The name “selective classification”, for classification with a reject option, was fixed in 2010 by El-Yaniv and Wiener in the Journal of Machine Learning Research. They traced the idea back to Chow in 1957 and 1970. They also named the trade-off the columns below describe: “The essence in selective classification is to trade-off classifier coverage for higher accuracy. We term this trade-off the risk-coverage (RC) trade-off.”

A decade later the definition was still the field standard. Kamath and colleagues call it exactly that in 2020: “The risk-coverage curve provides a standard way to evaluate selective prediction methods (El-Yaniv and Wiener, 2010).” Coverage and selective risk are citable constructs with a fifteen-year paper trail, not house vocabulary. Neither of them, by construction, says anything about the rejected cases.

A system may achieve equal accepted-case accuracy while deferring one language, device type, or demographic group far more often. Those users experience longer waits, more documentation, or less automation. The aggregate curve looks better while it happens.

Abstaining can widen a gap instead of closing it. That was measured across five vision and NLP datasets in 2021 by Jones and colleagues: “In this paper, we find that while selective classification can improve average accuracies, it can simultaneously magnify existing accuracy disparities between various groups within a population, especially in the presence of spurious correlations.” On MultiNLI, moving from full coverage to 50% average coverage took worst-group accuracy from 65% to 75%. The second-to-worst group went from 77% to 95%. Both groups improved. The distance between them grew.

On CelebA the worst group did not improve at all. Its accuracy fell as coverage fell. The model was most confident exactly where it was most often wrong for that group, so the rejection rule kept the answers it should have declined. An independent team found the same shape in regression: “the performance of a minority subgroup can decrease while we reduce the coverage”. That was Shah and colleagues, in 2022.

Report coverage, selective risk, fallback outcome, and delay by critical slice. Equal accuracy after filtering does not imply equal service, and lowering coverage is not by itself a safety measure.

Keep accepted and rejected populations connected throughout analysis. For one device class a binding regulation already demands it. FDA wrote the abstention output into the rule when it created the device type IDx-DR established: 21 CFR 886.1100, “Retinal diagnostic software device”, Class II. Special control (b)(2)(i) reads: “Clinical performance testing must evaluate sensitivity, specificity, positive predictive value, and negative predictive value for each endpoint reported for the indicated disease or condition across the range of available device outcomes.” For IDx-DR that range is three outcomes: mtmDR detected, mtmDR not detected, and insufficient quality. The abstention cannot be dropped from the evidence. It can only be reported alongside the rest.

The fallback pathway is regulated too. Another special control requires labelling to warn that the device “provides a screening diagnosis only and that it is critical that the patient be advised to receive follow-up care”. That is a requirement about what happens after the device speaks, or declines to. Steps 3 and 4 below are that regulation restated as ordinary evaluation practice.