Decide, Ask, or Defer: Clinical LLMs under Incomplete Evidence

AI summary

Overview

Research area: Evaluation methodology for clinical large language models (LLMs) that must act under incomplete medical evidence — specifically the decision of whether to diagnose, gather more information, or hand the case to a clinician.

Technical level: Intermediate. The core idea is conceptually simple, but the paper relies on matched experimental designs, case-clustered bootstrap confidence intervals, exact McNemar tests, and pairwise Stuart–Maxwell tests for marginal homogeneity.

Scope: An evaluation paradigm plus a controlled empirical study of Qwen, Gemini, and GPT on 200 matched synthetic clinical evidence states built from DDXPlus, comparing a new three-action Decide / Ask / Defer formulation against a matched binary Decide / Abstain baseline.

What This Paper Is About

Existing clinical LLM evaluations usually frame non-decision behavior as a single binary choice: decide or abstain. That framing collapses two very different situations into one label — a case where the model simply needs more information, and a case where the model should hand the case to a human clinician. The paper introduces a three-action formulation, Decide / Ask / Defer, together with a blinded protocol that hides evidence-completeness metadata, and asks whether separating information acquisition from clinician deferral exposes behavior that binary abstention conceals.

Key Contributions

1. A Decide / Ask / Defer three-action formulation. Decide produces an autonomous diagnosis from currently revealed evidence; Ask acquires one additional case-specific evidence item before a final Decide or Defer; Defer transfers the case to a clinician. This is paired with a blinded matched evaluation protocol in which the model never sees the evidence percentage, the number or identity of hidden items, or the ground truth diagnosis.

2. A controlled evaluation of information acquisition. The Ask mechanism draws from a case-specific hidden evidence pool H_t = E \ E_t. The model selects one item q_t, and the answer is retrieved from the underlying case record rather than generated by another model. A matched random comparator samples one item uniformly from the same pool, isolating the value of model-selected information from the value of receiving any information at all.

3. Cross-model and binary-baseline analysis. Qwen, Gemini, and GPT are evaluated on identical evidence, with pairwise Stuart–Maxwell tests for action-distribution homogeneity. A matched Decide / Abstain baseline for Qwen isolates the effect of the action space while holding model identity and revealed evidence fixed.

4. Identification of a safety–autonomy tradeoff. The three-action policy operationally rescues some erroneous autonomous decisions while also removing some correct ones, rather than being a uniformly better policy.

Main Findings

• Models disagree sharply under identical evidence. Across the 200 matched states, disagreement occurs in 137 of 200 states (68.5%). All three models agree in 63 states (31.5%), exactly two agree in 120 states (60.0%), and all three select different actions in 17 states (8.5%).

• Disagreement persists even at full evidence. At the 100% evidence state, at least two models select different actions in 37 of 50 cases (74%). The most common cross-model pattern is Qwen selecting Ask while both Gemini and GPT select Decide.

• Pairwise action distributions differ significantly. Stuart–Maxwell tests reject marginal homogeneity for every pair after Holm correction: Qwen vs. Gemini (50.5% disagreement, χ² = 93.15, df = 2, Holm-adjusted p = 1.78 × 10⁻²⁰), Qwen vs. GPT (61.5%, χ² = 89.00, p = 9.46 × 10⁻²⁰), Gemini vs. GPT (33.5%, χ² = 46.59, p = 7.63 × 10⁻¹¹).

• Distinct action profiles. Over the 200 states, Qwen selects Ask in 65.0% of states versus 33.5% for Gemini and 21.0% for GPT. Gemini selects Decide most often at 65.0%, versus 53.5% for GPT and 18.5% for Qwen. GPT selects Defer in 25.5% of states. Qwen's Ask rate declines from 80% at 25% evidence to 50% at full evidence; because the hidden pool is empty at 100%, that residual 50% consists entirely of Unfulfillable Ask. Gemini reaches a 92% Decide rate at both 75% and 100% evidence. At full evidence, Unfulfillable Ask occurs in 50% of Qwen states, 6% of Gemini states, and 2% of GPT states.

• Targeted information changes decision propensity more than correctness (Qwen only). Across 105 executable Ask states, targeted information raises the post-Ask Decide rate from 57.14% to 67.62% (+10.48 percentage points, 95% CI [0.95, 20.00]), with exact McNemar p = .052 — treated by the authors as inconclusive because the bootstrap interval and McNemar test diverge near the 0.05 threshold. Correct Decide outcomes rise from 38.10% to 43.81% (+5.71 pp, 95% CI [−2.86, 14.29], p = .263). Conditional accuracy among final Decide outcomes is 64.79% under targeted versus 66.67% under random information. Defer rates are 32.38% and 42.86%.

• The sensitivity analysis shows the same pattern. Restricting to the 81 distinct-item pairs (excluding 24 pairs where targeted and random selections coincided), the post-Ask Decide rate rises from 49.38% to 62.96% (+13.58 pp, 95% CI [1.23, 25.93], p = .05224), while correct Decide rises from 35.80% to 43.21% (+7.41 pp, 95% CI [−3.70, 18.52], p = .26318). Conditional accuracy is 68.63% targeted versus 72.50% random; Defer rates are 37.04% and 50.62%.

• Pair-level outcomes are balanced. Among the 81 distinct-item pairs, targeted information yields 16 wins, 49 ties, and 16 losses under the ordering correct Decide > Defer > incorrect Decide; a two-sided exact sign test over the 32 non-tied pairs gives p = 1.000.

• A safety–autonomy tradeoff against the matched binary baseline (Qwen only). The three-action policy reduces autonomous coverage by 14.5 percentage points and correct autonomous coverage by 13.5 percentage points. The overall wrong decision rate changes by −1.0 percentage point, with a confidence interval crossing zero. At full evidence, the wrong decision rate is 14 percentage points lower. Among 32 binary-baseline wrong decisions, 14 are rescues (ending in correct Decide or Defer); among 105 correct autonomous binary decisions, 43 lose correct autonomous status; and 15 of 63 binary abstentions become autonomous decisions (abstention recovery). The reduction in correct autonomous decisions is significant (McNemar p = .000584), while the overall wrong decision difference is not (p = .864166).

Methodology in Plain English

The researchers took 50 clinical cases from the DDXPlus synthetic diagnostic dataset (test split) and constructed four evidence states per case, corresponding to 25%, 50%, 75%, and 100% of available case evidence — 200 matched states in total. Cases were sampled in a pathology-aware way: one case per pathology group first, then additional cases from the remaining pool, using a fixed seed of 20260824. The final set covers 49 pathologies: 48 represented by one case each, and anemia by two cases.

Three models were evaluated on the same states: Qwen 3.8 27B (via Groq), Gemini 3.7 Flash (via Vertex AI), and GPT 5.6 Terra (via the OpenAI API), all at temperature 0. Each model call is blinded — the prompt shows only currently revealed evidence and the definitions of the three actions, never the evidence percentage, hidden item count or identities, completeness status, or ground truth. The model must return exactly one Stage 1 action. If it chooses Ask, a second stage shows only valid hidden questions, the model picks one, and the answer is pulled directly from the case record (no LLM generates it) and appended. A second Ask is not permitted; the final choice is Decide or Defer. If the model chooses Ask when the hidden pool is empty, the state is recorded as Unfulfillable Ask rather than being relabeled.

To isolate what model-selected information buys, one item is sampled uniformly from the same hidden pool for each executable Ask, creating a matched random condition. For Qwen only, a matched Decide / Abstain baseline runs on the same cases, evidence states, and revealed evidence, changing only the action space.

Statistical work is done offline on frozen outputs, with no additional model calls. Confidence intervals use 10,000 case-clustered bootstrap samples (resampling whole cases, keeping their evidence states together). Paired binary outcomes use exact McNemar tests; cross-model action distributions use pairwise Stuart–Maxwell tests with Holm correction. The case list was frozen before evaluation and verified against a manifest using case identifiers and a SHA-256 checksum.

Why This Matters

For research, the paper argues that binary abstention can conceal clinically different uncertainty states, and that a richer action space does not by itself yield a well-calibrated policy. Treating Ask as an action to be evaluated alongside Decide and Defer, rather than as an extra prompting trick, gives a cleaner way to study whether a model should gather information, commit, or escalate. The blinded protocol and matched comparators offer a template other clinical-LLM evaluations could reuse.

Real-world applications the formulation points toward (the paper itself describes only an evaluation paradigm, not a validated deployment policy):

• Clinical triage and intake tools that must decide when to commit to a differential, when to request one more patient detail, and when to route to a human clinician.
• Escalation design for LLM-assisted decision support, where the Defer action maps to a concrete handoff to a clinician rather than a silent non-answer.
• Benchmark design for health AI procurement, using matched evidence states and blinded prompts to compare vendor models on identical inputs rather than on vendor-reported scores.
• Safe-autonomy auditing, where the rescue-versus-loss accounting (14 rescues among 32 baseline errors, 43 correct decisions lost among 105) can be reused to characterize a system's tradeoff profile before deployment.

Industry relevance is that the results give vendors and health systems a way to compare models and policy designs on matched evidence rather than headline accuracy, and they highlight a concrete operational tension: a policy that reduces unsafe premature commitment can also suppress correct autonomous decisions, changing which cases stay autonomous without significantly reducing the overall wrong decision rate.

Future Directions

• Move beyond the finite hidden pool. The Ask setting currently restricts the model to selecting from a case-specific pool of pre-existing evidence items; open-ended clinical questioning, as in conversational history taking, remains unaddressed.
• Test on realistic clinical data. The evaluation uses 50 synthetic DDXPlus cases covering 49 pathologies and does not capture the variability of real clinical workflows, so prospective or real-world evaluation is an open requirement.
• Extend the matched analyses beyond one model. The targeted-versus-random and matched binary comparisons were conducted only for Qwen; whether the same safety–autonomy tradeoff holds for Gemini, GPT, or other models is unknown.
• Reduce comparator and configuration confounds. Each executable Ask used only one random comparator rather than repeated draws, and generation controls were not fully standardized across providers — GPT used reasoning_effort="none" while no corresponding reasoning setting was fixed for Qwen or Gemini. Both limit generalizability and could be tightened in follow-up work.
• Label deferral appropriateness. DDXPlus provides no ground truth labels for deferral appropriateness, so Defer is treated as a non-decision outcome rather than a clinically correct action; establishing such labels would let future work score deferral directly.

Target Audience

Researchers working on clinical LLMs, uncertainty quantification, abstention and selective prediction, and interactive information-seeking benchmarks; machine learning evaluation and safety researchers interested in blinded protocols and matched experimental designs; and clinical informatics or health-system teams designing triage, escalation, and human-in-the-loop deployment policies. The paper is most useful to readers comfortable with paired statistical testing and cross-model benchmarking, though the core conceptual argument — that Ask and Defer are different things — is accessible without that background.

Authors’ abstract

Clinical LLMs must decide not only what diagnosis to produce, but also whether the available evidence is sufficient for autonomous decision making. Binary DECIDE/ABSTAIN formulations merge distinct non decision states and do not explicitly evaluate information acquisition. We introduce a DECIDE/ASK/DEFER formulation together with a blinded protocol that prevents models from using evidence completeness metadata. We evaluate Qwen, Gemini, and GPT on 200 matched clinical evidence states constructed from DDXPlus. The models show substantial differences in action selection under identical evidence, with disagreement in 137 of 200 states. For Qwen, a matched targeted versus random analysis shows that selected information changes the likelihood of a subsequent autonomous decision more clearly than diagnostic correctness. Its matched DECIDE/ABSTAIN baseline further reveals a safety autonomy tradeoff: the three action policy rescues some erroneous autonomous decisions but also removes some correct autonomous deci sions. These results show that separating information acquisition from clinician deferral exposes behavior that binary abstention hides, without yielding a uniformly improved decision policy.