When Derived Measurements Mislead: Quantifying and Mitigating LLM Over-Trust with Privileged-Modality Reliability Evidence

AI summary

Overview

Research area: Artificial intelligence — specifically the interface between upstream machine-learning estimators and downstream large language models (LLMs), evaluated in a physiological-sensing case study using photoplethysmography (PPG) and electrocardiography (ECG).

Technical level: Intermediate. The paper is readable without specialized signal-processing knowledge, but it assumes familiarity with LLM pipelines, knowledge distillation, and basic statistical estimation (confidence intervals, bootstrap resampling).

Scope: The paper names, formalizes, measures, and begins to mitigate a downstream failure mode it calls derived-feature over-trust (DFOT), in which an LLM treats an upstream-derived measurement as a direct fact rather than as an instance-dependent estimate.

What This Paper Is About

Modern AI systems often chain a specialized upstream model (which converts raw signals into a compact derived value) to a downstream LLM that reasons over that value. The LLM usually never sees the raw data or any indication of whether the derived value is trustworthy for the particular instance, so a coherent-sounding LLM can still be wrong because it assigned an uncertain estimate the status of a fact. The paper defines this as derived-feature over-trust (DFOT), builds two controlled challenge scenarios plus five linked metrics to quantify it, and demonstrates the framework on a 50,000-record paired PPG–ECG dataset where ECG is available only for training and offline reference — never to the LLM.

Key Contributions

1. A new failure mode. DFOT is introduced as a distinct downstream failure: an LLM assigns an upstream-derived estimate the epistemic status of a direct fact, or uses it outside its valid scope. The authors distinguish it from hallucination, from conventional prediction error, from probability calibration, and from selective prediction.

2. A five-estimand metric chain. Conflict over-trust rate (COTR) and context-induced error rate (CIR) characterize failure induction in the two challenge scenarios; correct repair rate (CRR) measures repair on frozen error sets; evidence-specific repair margin (ESRM) contrasts matched evidence against patient-disjoint shuffled evidence; and utility harm rate (UHR) measures unnecessary verification among cases that were already usable at baseline. Crucially, the estimands use fixed ("frozen") denominators set before evidence conditions are compared.

3. A matched-versus-shuffled intervention. Matched evidence comes from the current record; shuffled evidence is patient-disjoint but preserves the case, baseline answer, timing, treatment arm, Student seed, and prompt template, changing only the evidence source. This tests whether revision depends on information specific to the instance rather than on the mere presence of reliability language.

4. A protocol-locked benchmark and reference baseline. A patient-disjoint PPG–ECG benchmark with a privileged ECG-to-PPG distillation baseline (called K2) compared against a conventional PPG-only reliability model (called B2), providing a reproducible point of comparison for future DFOT mitigation methods. Code is released at https://github.com/Zongheng-Guo/When-Derived-Measurements-Mislead.

Main Findings

• DFOT can be reliably induced. On validation, D1 COTR is 1.000 (300 records; 37 patients) and D2 CIR is 0.942 (292/310 neutral-correct records). In the protocol-locked E3 replication, all 300 D1 records from 44 patients are over-trusted, and severe history induces an error in all 297 neutral-correct D2 records from 97 patients, giving CIR = 1.000. The authors stress that these are controlled manipulation checks, not prevalence estimates.

• The proposed baseline improves repair and specificity on the locked test. The abstract reports the baseline improves four repair and specificity endpoints by 1.82–6.69 percentage points, with all paired confidence intervals excluding zero. Specifically, D1 matched CRR rises from 40.11% (B2) to 44.83% (K2), a gain of +4.72 [1.67, 7.28] percentage points; D1 ESRM rises from 21.67% to 26.17%, +4.50 [0.98, 8.18] points; D2 matched CRR rises from 94.39% to 96.21%, +1.82 [1.51, 2.91] points; and D2 ESRM rises from 15.82% to 22.51%, +6.69 [2.60, 7.19] points.

• Substantial headroom remains. On D1, 55.17% of the deliberately induced errors remain unrepaired, which the authors describe as leaving clear mitigation headroom. D2 matched repair is already near ceiling, making its larger ESRM improvement more informative than its smaller CRR gain.

• Interface harm is small but not conclusively bounded. UHR increases by 0.67 percentage points (95% CI −0.4 to +1.7), from 2.89% for B2 to 3.56% for K2. The stated UHR criterion is an exploratory interface-engineering limit (upper CI < 2 pp), not a clinical non-inferiority margin.

• Effects hold across models and prompts. Under prompt P1, the K2–B2 contrasts remain positive for Qwen3-8B, DeepSeek-V4-Pro, and GPT-5.5 with no directional reversal. Within Qwen3-8B, effects remain positive across prompts P1–P3; pooled D1 and D2 ESRM gains are +0.064 [0.029, 0.097] and +0.071 [0.035, 0.108]. The authors note this is not a complete LLM-by-prompt factorial evaluation. Every Student seed preserves the direction of effect.

• Upstream gains are real but incomplete. The 17-arm development study selects K2 at AUROC .9042 versus B2 at .8927, while within-case shuffled-target K5 remains close to .9003, suggesting both privileged regularization and correct teacher–student correspondence contribute. On the locked upstream test, K2 exceeds B2 by .0250 AUROC [.0118, .0274] and globally shuffled ECG by .0115 [.0095, .0455]. Extreme low-FPR effects remain unresolved.

• A negative control disciplined the conclusions. A tail-aware objective improves the development tail mean by 3.06 percentage points (95% CI 1.28–5.56) over K2 and by 2.59 points (95% CI 0.85–4.53) over hidden-weighted BCE, but shows no reliable advantage over a patient-deranged shuffled-tail control (+1.31 points, 95% CI −1.70 to +4.50; pAUC difference 0.0057, 95% CI −0.0105 to 0.0212). That control preserves the number of positive assignments (346) and all training mechanics while retaining the true hidden label for only 10.69% of examples — so the authors abandoned further tail-specific optimization, concluding the extra gain could not be attributed to label-specific privileged information.

• Exact original-recipe validation numbers. B2 reaches AUROC .8926, AUPRC .7466, Brier .1018, ECE15 .0689, TPR@1% 19.05%, TPR@2% 25.87%, TPR@5% 36.80%. K2-global-shuffled reaches .8909, .7509, .1000, .0710, 17.53%, 22.62%, 37.34%. K2 matched reaches .9018, .7642, .0984, .0577, 17.86%, 23.81%, 36.15%. K2 improves global ranking and calibration but not hidden-irregularity sensitivity in the extreme low-FPR region.

• The ECG teacher is a strong reference. Averaged over three random seeds, the hierarchical ECG teacher achieves record-level rhythm AUROC 0.9912 ± 0.0007, heart-rate MAE 1.241 ± 0.004 bpm, RMSSD correlation 0.9868 ± 0.0011, and SDNN correlation 0.9831 ± 0.0014. The authors explicitly note these values qualify it as a privileged reference model but do not by themselves demonstrate transfer.

Methodology in Plain English

The authors separate their problem into layers and measure each one separately.

The setup. An upstream estimator turns deployment-modality raw signals into a derived feature D. A separate reliability model produces an estimate R, which is rendered into human/language-readable evidence E. A downstream LLM M receives a task Q, the derived feature D, context C, and optionally the evidence E — but never the privileged modality.

The two challenges. D1 presents cases where the PPG-derived rhythm looks regular but the offline ECG says irregular ("hidden irregularity"); the failure is the LLM accepting the derived value despite the conflict. D2 fixes a currently regular segment but supplies a misleading severe atrial fibrillation history; the failure is the LLM rejecting a current derived measurement that PPG and ECG agree on. Error sets are "frozen": D1 freezes the subset of conflict cases the LLM got wrong under no evidence; D2 first restricts to cases answered correctly under neutral history, then freezes those that severe history broke. All evidence conditions reuse these fixed denominators.

The evidence intervention. Reactive mitigation simply appends reliability evidence to the frozen baseline interaction. Matched evidence comes from the current record; shuffled evidence comes from a patient-disjoint donor while holding the case, baseline answer, arm, seed, timing, and prompt template constant. The difference in CRR between these two conditions is ESRM.

The four experiment stages. E0 evaluates Qwen3-8B under the primary prompt P1. E1 holds P1 fixed across Qwen3-8B, DeepSeek-V4-Pro, and GPT-5.5. E2 tests three semantically equivalent prompts within Qwen3-8B. E3 applies the frozen Qwen3-8B/P1 protocol to patient-disjoint test patients with no prior downstream LLM queries. Qwen requests use vLLM 0.25.1, temperature 0, at most 96 output tokens, disabled thinking, and deterministic parsing. E3 uses 5,000 patient-cluster bootstrap replicates, resampling three Student seeds with replacement and sharing each seed draw across paired arms and evidence conditions. Four efficacy gates require the lower confidence bounds for K2–B2 differences in matched CRR and ESRM to stay positive in both D1 and D2.

The data. From the MIMIC-III Matched Waveform Database v1.0, a channel-level scan identified 1,297 candidate patients with simultaneous PLETH and Lead-II recordings. Requiring a complete 240-s paired interval, at most 20% missing samples in either raw channel, and valid ECG and PPG beat-interval extraction yielded 50,000 synchronized four-minute records from 1,275 patients: 36,115 records from 925 patients for training, 6,464 records from 163 patients for validation, and 7,421 records from 187 patients for the locked test — all patient-disjoint. PPG is resampled at 50 Hz (12,000 samples per segment) and Lead-II ECG at 125 Hz (30,000 samples per segment). Reliability is defined over four clear rhythm cases: both regular and both irregular are HIGH, while PPG-regular/ECG-irregular and PPG-irregular/ECG-regular are LOW. The locked test contains 3,733 clear cases (897 LOW, 2,836 HIGH), including 401 hidden-irregularity and 2,151 both-regular cases. MIMIC clinical tables are not used, so cohort age, sex, and race summaries are unavailable.

The models. The ECG teacher processes eight contiguous 30-s windows with a shared encoder, gated multi-view fusion, and a temporal Transformer, producing window-level and record-level three-class predictions. The PPG student is initialized from SIGMA-PPG, pools 120 PPG patches into eight windows aligned with the teacher, and predicts p_low; B2 optimizes reliability BCE while K2 adds alignment to teacher outputs with L_K2 = L_rel + 0.15·(L_winKD + L_globalKD) at temperature T = 2. B2, K2, and a K2-global-shuffled control (which globally permutes teacher targets) share training schedule, optimization budget, data order, and random seeds (42, 123, and 2026). The 0.15 weights and T = 2 were chosen on validation and frozen before test access. The language-facing block contains only a HIGH/LOW label, p_low rounded to four decimals, and an explanation that the score estimates current-segment surrogate reliability; LOW means p_low ≥ 0.5, and the frozen interface contains no label–score mismatches. No ECG waveform, rhythm value, diagnosis, teacher logit, or embedding is ever shown to the LLM.

Why This Matters

Impact on research. Modular AI systems are usually evaluated either by upstream prediction performance or by downstream task accuracy, leaving the interface between them largely unexamined. This paper makes that interface a measurable object. It introduces a claim hierarchy — accessibility (does the reference channel contain learnable information?), transfer (does it improve a deployment-time reliability signal?), utilization (does that signal repair downstream decisions?), and specificity (does repair depend on the current instance?) — and shows that these claims require different evidence. It also deliberately avoids a single weighted composite score, arguing acceptable tradeoffs among repair, specificity, abstention, workload, delay, and missed detection are application dependent, and offering a metric vector for auditable Pareto comparisons instead.

Real-world applications

• Wearable cardiac monitoring. PPG-derived heart-rate and rhythm labels are widely deployed, but PPG reliability varies with rhythm, motion, perfusion, device, and acquisition conditions. The paper's D1 scenario models the dangerous case where a regular-looking PPG trace hides an irregular rhythm.
• Clinical decision-support pipelines. The paper's D2 scenario models how stale or severe patient history can cause a model to override a currently valid measurement — a failure mode relevant to any system that mixes historical context with fresh sensor data.
• Any LLM-over-specialist-model architecture. The authors point to vision systems connecting LLMs to specialized visual models or computer-aided diagnosis outputs, audio systems coordinating speech interfaces with specialized audio models, and signal-language systems passing physiological summaries or encoded time series into language reasoning.
• Safety-critical automation generally. Wherever an upstream estimator's validity is instance-dependent — higher-fidelity modalities, repeat measurements, or expert adjudication could all serve as the reference — the DFOT evaluation template applies.

Industry relevance. Teams shipping any multi-stage AI product (a specialized model feeding an LLM) get a concrete, low-cost protocol: define an independently verified reference, construct a controlled challenge, run matched-versus-shuffled evidence, and report absolute CRR alongside ESRM and a cost endpoint such as UHR. The paper's insistence on freezing cases, baseline decisions, parsers, and denominators before comparison, and on reporting negative controls, is directly transferable quality-assurance practice. The released code and protocol-locked benchmark give a shared comparison target rather than every team inventing its own.

Important caveats. The authors state the framework evaluates controlled LLM behavior and is not intended for direct clinical decision-making. D1 and D2 are controlled stress tests rather than prevalence estimates. ECG is treated as a higher-fidelity rhythm reference rather than infallible ground truth. K2 is explicitly framed as a reference baseline for a newly defined task, not a definitive solution.

Future Directions

• Compare stronger reliability generators under the same frozen protocol. The paper names calibrated uncertainty heads, ensembles, conformal or selective predictors, retrieval-based verification, and stronger multimodal students as candidates. New methods need not reuse the same teacher, architecture, loss, or evidence representation.

• Resolve the extreme low-FPR region. The authors note that global ranking can improve without improving the stringent operating region a particular workflow requires, and that extreme low-FPR effects remain unresolved after their tail-aware optimization failed its negative control. This is stated as an open problem rather than a solved one.

• Extend beyond physiological sensing. A new instantiation requires a derived measurement, an independently defined criterion for appropriate reliance, a controlled challenge exposing inappropriate reliance, and a matched–shuffled or equivalent counterfactual control. The reference may come from a higher-fidelity modality, repeated measurements, expert adjudication, or another independently specified source — but the specific reliability generator and numerical effects remain application-specific.

• Run the factorial evaluation the paper did not run. The authors explicitly state that their model-and-prompt comparison is not a complete LLM-by-prompt factorial evaluation, leaving a broader robustness study open.

Target Audience

This paper is most useful to researchers and practitioners working on multi-stage AI systems in which an upstream estimator feeds a downstream LLM — particularly in medical AI, wearable sensing, and clinical decision support. It will also interest evaluation and safety researchers concerned with calibration, selective prediction, learning to defer, and knowledge-conflict benchmarks, since the paper positions DFOT against each of those literatures. Engineers building or auditing modular pipelines will find the metric definitions, the matched-versus-shuffled design, the protocol-lock discipline, and the released code directly actionable. Readers looking for a finished solution should note the authors' own framing: this is a problem definition, measurement framework, and reference baseline, with substantial residual error left for stronger methods to address.

Authors’ abstract

Derived measurements increasingly enter large language model (LLM) pipelines as direct facts despite their instance-dependent validity. We define derived-feature over-trust (DFOT) as the failure in which a downstream LLM assigns such a measurement the epistemic status of a direct fact or uses it outside its valid scope. Using physiological sensing as a case study, D1 tests acceptance of a PPG-derived rhythm contradicted by offline ECG, whereas D2 tests rejection of an offline-confirmed reliable PPG rhythm under misleading severe history. ECG supplies training supervision and offline reference construction but is never shown to the LLM. Five estimands quantify this chain: conflict over-trust rate (COTR) and context-induced error rate (CIR) characterize D1/D2; correct repair rate (CRR) measures frozen-error repair; evidence-specific repair margin (ESRM) contrasts matched and patient-disjoint shuffled evidence; and utility harm rate (UHR) measures unnecessary verification among HIGH-reliability cases used without verification at baseline. The framework does not depend on a particular reliability generator. We demonstrate it on 50,000 paired PPG-ECG records using ECG-to-PPG privileged distillation as an illustrative baseline and PPG-only inference. On a protocol-locked 187-patient test, the baseline improves four repair and specificity endpoints by 1.82-6.69 percentage points, with all paired confidence intervals excluding zero; UHR increases by 0.67 percentage points (95% CI: -0.4 to +1.7). DFOT provides a common evaluation target for stronger mitigation methods. The code is available at https://github.com/Zongheng-Guo/When-Derived-Measurements-Mislead.