Beyond the Evidence Boundary: Behavioral Mirage Under Progressive Degradation in Medical VLMs

AI summary

Overview

Research area: Medical vision-language models (VLMs) — robustness evaluation, uncertainty quantification, abstention behavior, and mechanistic interpretability.

Technical level: Advanced. The paper combines forced-choice logit readouts, entropy-based thresholds, trajectory typologies, corruption sweeps, and linear probes on residual-stream activations.

Scope: A baseline-anchored evaluation framework that tracks only initially correct cases through progressive image and language degradation to locate an empirical "evidence boundary," then measures how often models keep committing beyond it and whether that commit/abstain decision is linearly decodable from internal activations.

What This Paper Is About

Medical VLMs are usually tested on clean images, even though real clinical deployments often lose visual evidence through hardware limits, acquisition conditions, storage, and transmission. Prior robustness work shows accuracy collapsing while confidence stays high, but it does not establish when a specific model-case pair stops behaving reliably, or what the model does afterward. This paper builds a framework that follows individually baseline-correct cases down a degradation ladder, marks the severity at which aggregate reliability falls to the level of a trivial fixed-answer predictor, and then characterizes the "mirage" behavior that follows.

Key Contributions

1. A baseline-anchored evidence-boundary framework. Cases are first filtered to a baseline-correct population (correct and with answer entropy below τ = 0.30 nats), then traced through a fixed 15-level severity schedule (s = 0 clean to s = 14 blackout). The population-level breaking threshold s* is defined as the first severity where the confident-correct share falls to or below p_max, the largest class share of the same class-balanced sample — i.e., the point where the aggregate is consistent with fixed-answer collapse.

2. A redefinition of mirage relative to that boundary. Rather than defining mirage as image-present/image-absent output alignment (Ben-Levi et al., 2026), the authors define operational mirage as continued substantive answering at s*, regardless of correctness. Mirage rate is reported as the post-boundary substantive commitment rate.

3. A behavioral trajectory typology. Each baseline-correct case is encoded as a length-15 state sequence over {correct, wrong, IDK/abstain} and assigned to exactly one of six mutually exclusive types by fixed precedence: persistent correct, direct mirage, correct recovery, mirage dissolves, consistent abstention, oscillatory.

4. A probe of the commit-versus-abstain axis. ℓ2-regularized, class-balanced logistic regression probes are fit on residual-stream activations at s*, with nested cross-validation (layer and regularization chosen inside training folds) across four token positions and from the clean input.

The paper also adds a language-only diagnostic (the Prior Harnessing Index, PHI, from Ben-Levi et al., 2026), an abstention-option ablation (IDK-enabled vs. no-IDK), and an option-order control.

Main Findings

• Only one of three models was analyzable. Across nine model-dataset pairs (MedGemma, LLaVA-Med, Ovis × APTOS, HAM10000, VQA-RAD), only MedGemma produced a usable baseline-correct pool on all three datasets, covering 33.1% to 58.9% of items. LLaVA-Med met the minimum pool criterion only on VQA-RAD (n = 24, descriptive only); Ovis met it on none. On APTOS, LLaVA-Med assigned 100.0% of items to high-entropy commitment and Ovis abstained on 100.0% of items.

• Boundaries differ sharply by dataset and modality. Median breaking severity was earliest on HAM10000 (2 for noise, 5 for blur), then APTOS (4, 6), then VQA-RAD (9, 8). Noise produced an earlier boundary than blur on APTOS and HAM10000; the ordering reversed on VQA-RAD.

• Mirage rates span 16.0% to 100.0%. At each pair's s*, MedGemma committed rather than abstained at rates of 64.8% (APTOS noise, s* = 4), 16.0% (APTOS blur, s* = 7), 100.0% (HAM10000 noise, s* = 3), 100.0% (HAM10000 blur, s* = 5), 84.4% (VQA-RAD noise, s* = 8), and 86.7% (VQA-RAD blur, s* = 7). The APTOS contrast is the largest modality gap in the study.

• Abstention propensity drives much of the heterogeneity. MedGemma abstained on a mean 56.6% to 62.3% of degraded APTOS outputs, 12.4% to 17.7% of VQA-RAD outputs, and almost never on HAM10000 — only 35 of 9,688 degraded HAM10000 outputs (0.4%) selected the abstention option.

• Post-boundary trajectories are qualitatively different per dataset. On VQA-RAD, correct recovery was the largest trajectory under both modalities (42/128 noise, 47/128 blur), followed by persistent correctness (37, 32) and direct mirage (27, 20). On HAM10000, all 64 recoveries were the 64 actinic keratosis (class A) examples, and without the blackout step the dataset was mostly direct mirage (251 and 258 of 346). APTOS was dominated by abstention-ending trajectories.

• Persistent correctness at blackout is not robustness. No APTOS or HAM10000 example stayed correct through blackout, whereas 37/128 (28.9%) of VQA-RAD examples under noise and 32/128 (25.0%) under blur did — but 34 of the 37 and 28 of the 32 had the correct answer "No," making the answer recoverable without the image.

• Errors concentrate on specific answer classes, and this concentration is order-dependent. At s* on VQA-RAD, every non-ground-truth commitment under noise was class B (a Yes-to-No flip), and 24/25 were class B under blur. Under reversed option order this concentration did not persist.

• Mirage rates depend on how the boundary is anchored. With the natural-correct share substituted for the confident-correct share, rates became 43.8, 16.0, 99.7, 98.3, 78.9, and 71.1%. Taking p_max over the full baseline-correct pool raised APTOS blur from 16.0% to 71.9% and APTOS noise to 85.2%, while HAM10000 stayed at or above 98%. Varying τ between 0.05 and 0.30 left HAM10000 at 100% but moved APTOS noise from 93.6% to 64.8% and APTOS blur from 92.6% to 16.0%.

• Output entropy is only a partial safeguard. Wrong answers given with H < τ made up 1.2% to 17.1% of the sample at s*, and 78.5% of wrong HAM10000 blur answers beyond s* still had H < τ. Median entropy was higher for wrong than correct answers in most pairs (HAM10000 noise 0.72 vs. 0.39; VQA-RAD blur 0.36 vs. 0.11) but not on APTOS noise (0.44 vs. 0.50).

• Option order changes both pool membership and behavior. Under reversed order, only 85.5% (APTOS), 39.6% (HAM10000), and 67.2% (VQA-RAD) of sampled examples remained confident-correct at s = 0. APTOS blackout abstention went from 100% to 0% under both alternative orders. The same 120 clean VQA-RAD items returned No for 65.8% in the original order and Yes for 77.5% in the reversed order.

• Language-only answerability tracks behavior on VQA-RAD. Mean text-only PHI over full MedGemma populations was 0.07 nats (APTOS), 2.08 (HAM10000), and 1.84 (VQA-RAD). With PHI as a predictor of baseline correctness, AUROC was 0.677 on VQA-RAD (95% bootstrap interval 0.646 to 0.706) but 0.294 on APTOS and 0.464 on HAM10000. For commit-versus-abstain at s*, PHI AUROC was 0.670 (noise) and 0.653 (blur) on VQA-RAD, and 0.406 and 0.395 on APTOS.

• Commit-versus-abstain is linearly decodable internally. Nested AUROC at the last prompt token was 0.895 to 0.953 (APTOS noise), 0.994 (APTOS blur), 0.966 to 0.970 (VQA-RAD noise), and 0.896 (VQA-RAD blur) — all above chance and above class-only baselines. It was lower at the last image token (0.396 to 0.905) and question token (0.747 to 0.983), and highest at the answer prefix (0.991 to 1.000), indicating decodability where the letter is produced rather than prediction ahead of time. Clean-input activations still predicted s* behavior at 0.713 to 0.825.

• Language degradation pushes toward wrong commitment, not abstention. On VQA-RAD, WordNet substitution was mild (87.5% natural accuracy at r = 0.75) and character corruption was more damaging and non-monotonic (43.8% at r = 0.50, 53.1% at r = 0.75), yet IDK peaked at only 13.3%. On HAM10000, blackout accuracy was 18.5% with no protection and 73.4% with answer-choice descriptions protected, indicating reliance on the option descriptions.

Methodology in Plain English

Step 1 — Build a trustworthy starting population. Each dataset is turned into a closed-ended multiple-choice task with a fixed answer set plus one explicit "I don't know" option. The model's answer is read directly from the output distribution in a single deterministic forward pass (no sampling), so uncertainty is a precise number. Anything that is correct and below an entropy threshold of τ = 0.30 nats is called "baseline-correct" and only those cases move forward. This matters because it means the study is not measuring raw failure rates — it is measuring what happens to cases that once worked well.

Step 2 — Degrade the evidence in a controlled way. Visual degradation uses two families on a shared 15-step ladder from clean (s = 0) to an all-black image (s = 14): Gaussian noise of increasing standard deviation, and Gaussian blur of increasing kernel width. Language degradation instead mutates the prompt text (WordNet synonyms, random character corruption, blackout masking) under four "protection" settings that shield the abstention instruction, the answer descriptions, both, or neither. On VQA-RAD only the question text is mutated.

Step 3 — Define the boundary using a fair null. The threshold s* is where the share of confident-correct cases drops to p_max — the accuracy a lazy predictor would get by always choosing the most common class. This anchors the boundary to something interpretable: the point where the model's aggregate output is no better than a constant answer.

Step 4 — Classify what happens next. Each case becomes a sequence of states across all 15 severities and gets sorted into one of six trajectory types. The authors deliberately treat a low-confidence correct answer as IDK-like while treating a low-confidence wrong answer as a commitment, reasoning that a wrong commitment is still a commitment.

Step 5 — Look inside the model. At s*, they take the residual-stream activation at the final prompt token and ask whether a simple logistic regression can separate "answered" from "abstained" cases. They report both standard 5-fold cross-validated AUROC and a stricter nested version where the best layer is chosen only inside training folds.

Step 6 — Stress-test the conclusions. They rerun with reversed and rotated answer orders, resample with bootstrap, vary τ, use four noise seeds, and apply degradations after resizing to the model's input resolution (896 pixels) to check whether the boundary and mirage numbers are artifacts.

Why This Matters

Impact on research. The paper argues that accuracy-versus-severity curves hide the thing that matters clinically: whether a model's commitment is still supported by available evidence. By conditioning on baseline-correct cases and anchoring the boundary to a fixed-answer null, it converts a vague robustness question into a measurable per-case event. It also separates a behavioral signal from an internal one — behavior is heterogeneous across datasets and option orders, yet the commit/abstain state is highly decodable from activations. The option-order results are a methodological warning for anyone benchmarking abstention: pool membership itself shifted, with VQA-RAD flipping from 65.8% "No" to 77.5% "Yes" on identical clean items.

Real-world applications:

• Low-resource and point-of-care deployment, where imaging hardware, acquisition conditions, storage, transmission, and compute can each progressively strip visual evidence from a case.
• Clinical decision support triage, where a system should route to a human reviewer when evidence is insufficient rather than emit a confident but ungrounded suggestion.
• Regulatory and procurement evaluation of medical foundation models, where boundary-aware reporting

Authors’ abstract

VLMs for medical image interpretation are typically evaluated on clean inputs, despite the progressive evidence loss that can occur in high stakes clinical settings. Existing robustness studies show that performance can degrade while confidence remains high, but do not establish when a previously reliable model-case pair ceases to behave reliably. We introduce a baseline-anchored framework that traces baseline-correct cases through progressive image degradation to identify an empirical evidence boundary, and define mirage as continued commitment at this boundary, regardless of correctness. Across three VLMs on three medical datasets, only MedGemma provides a sufficiently large baseline-correct pool for trajectory analysis. On VQA-RAD, the only dataset in which the question text varies, language-only answerability is associated with baseline correctness (AUROC 0.677) and with commitment at the boundary (AUROC 0.653 to 0.670). MedGemma's post-boundary behavior is heterogeneous, including direct mirage, correct recovery, mirage dissolution, consistent abstention, and oscillation, with mirage rates of 16.0 to 100.0% across dataset-modality pairs. Despite this behavioral heterogeneity, residual-stream probes decode commit-versus-abstain behavior at the boundary with AUROC of 0.895 to 0.994 under nested cross-validation. These results show that progressive evidence loss can produce persistent commitment that output entropy and abstention behavior do not consistently reflect, motivating boundary-aware evaluation, option-order controls, and improved abstention mechanisms for reliable medical foundation models operating under partial observation.