Evidence-Bounded Mental Health Reasoning from Heterogeneous Speech Protocols

AI summary

Overview

Research area: Natural Language Processing / multimodal clinical NLP, specifically speech-based mental health screening and evidence-grounded reasoning over heterogeneous clinical speech protocols.

Technical level: Advanced. The core idea (only reason from evidence the recording actually contains) is intuitive, but the paper formalizes it with a permission matrix, an evidence-package schema, and three new boundary-audit metrics.

One-sentence scope: The paper reframes multimodal mental health screening as "evidence-bounded reasoning," introduces a benchmark of 1,870 evidence packages from six clinical speech sources with explicit modality masks and claim permissions, and proposes EviBound, a protocol-aware harness that reaches a held-out depression AUROC of 0.8658 with zero claim violations.

What This Paper Is About

Mental health screening systems built on speech and text usually treat every recording as equally informative, even though a fixed reading task, a prompted narrative, a free interview, and a text-only transcript actually license very different kinds of conclusions. Forcing all of them into one unconstrained reasoning space causes what the authors call epistemic flattening: models hallucinate symptoms from scripted text, make acoustic claims when no audio exists, or attribute conclusions to modalities that were never collected. The paper's goal is to make the boundaries between protocols explicit and executable, so a system can only claim what its evidence supports.

Key Contributions

1. Evidence-Bounded Reasoning. The authors formalize protocol-aware mental health reasoning by representing each screening instance as an evidence package ℰ = (P, M, Π, 𝒳), where P is the protocol profile, M the modality mask, 𝒳 the observed inputs, and Π the admissible evidence boundary for that protocol. A report is only "evidence-bounded" if every emitted claim satisfies permit(c, M, P) = 1.

2. Evidence Package Benchmark. A unified benchmark of 1,870 standardized packages derived from six heterogeneous mental health resources — CMDC, DAIS-C, E-DAIC, EATD, MMPsy, and MODMA — with modality masks and claim permissions, frozen subject-independent splits (1,284/214/372 train/validation/test), and evaluator-only labels kept hidden from inference.

3. EviBound Framework. A three-stage protocol-aware harness: a profile-aware planner that restricts reasoning scope, parallel evidence tools including a five-way acoustic consensus, and a boundary validator that suppresses unsupported claims.

4. Boundary-Aware Evaluation. Three new metrics — Claim Violation Rate (CVR), Missing-Evidence Handling Rate (MHR), and Evidence-Bound Consistency Pass (EBCPass) — that measure evidence consistency separately from predictive accuracy.

Main Findings

• Predictive gains over direct omni-modal prompting. EviBound reaches a pooled depression-eligible AUROC of 0.8658 versus 0.6716 for Qwen3-Omni-Flash (+0.1942), and F1 rises from 0.5032 to 0.6557. Against the strongest direct LMM baseline, Gemini 3.5 Flash at 0.7848, the AUROC gain is +0.0811 with a 95% bootstrap interval of [0.0476, 0.1175].

• Zero boundary violations. EviBound achieves 0% CVR and 100% EBCPass on held-out packages. Direct omni-modal baselines still violate: Qwen3-Omni-Flash records CVR 0.271 and EBCPass 0.714, Qwen3.5-Omni-Plus 0.246 and 0.736, Gemini 3.5 Flash 0.070 and 0.931.

• More reasoning does not fix the problem. The long-reasoning baseline (Qwen3-Omni-Flash Thinking) scores 0.6217 AUROC on the depression-eligible split with CVR 0.318 and EBCPass 0.661 — the lowest consistency pass rate in the main table. The evaluated long-reasoning baseline is lower by 0.0499/0.0322 AUROC on depression/anxiety-eligible tasks relative to direct Qwen3-Omni-Flash.

• The biggest gains come under restrictive protocols. Stratified AUROC on prompted speech rises from 0.5993 to 0.9779 (+0.3787) and on fixed-reading depression rows from 0.5000 to 0.7333 (+0.2333). Interview-based depression packages (n=325, 73 positives) improve from 0.7504 to 0.8456 (+0.0952), and the MMPsy-dominated anxiety interview split (n=256, 30 positives) from 0.8777 to 0.8864 (+0.0086).

• Improvement over the acoustic backbone alone is not statistically established. EviBound's delta against the 5-way acoustic consensus is +0.0007 with a 95% CI of [-0.0155, 0.0170] on depression-eligible packages and +0.0056 with [-0.0168, 0.0267] on anxiety-eligible packages. The authors interpret this as EviBound preserving the acoustic backbone's predictive strength while adding routing and validation, not exceeding it.

• Ablations attribute most predictive signal to acoustic evidence. Base Harness reaches 0.7270 AUROC; the strong openSMILE route 0.8565; segmented openSMILE 0.8503; wav2vec2 0.8390; HuBERT 0.8507; WavLM 0.8316; 5-way consensus 0.8652; EviBound 0.8658. The validator does not change scores because risk estimates are frozen before report repair.

• A shortcut risk is documented and controlled. In an E-DAIC transcript control, interviewer prompts alone reach AUROC 0.6410, rivaling participant answers at 0.6471. The benchmark therefore separates participant evidence from prompt/control text at the manifest level.

• Boundary replay audits are clean. The claim-permission audit covers 1,870 packages and 1,870 report outputs with zero failed records, zero score changes, and zero permission violations. Held-out exact match, macro F1, and micro F1 on the 786 held-out action-labeled instances are all 1.0000; the cleanup guard changed 84 rows and removed 168 unsupported acoustic route/tool actions while removing zero gold actions.

• MHR values are not reported. The metric is formally defined but no numeric result for Missing-Evidence Handling Rate appears in the paper content provided.

Methodology in Plain English

The authors start by describing what kind of recording they are dealing with. Each item in their benchmark is packaged with a protocol profile (free interview, prompted speech, fixed reading, or text-only discourse), a modality mask saying which channels exist (raw audio, transcript text, pre-extracted acoustic features), and a list of what claims that combination is allowed to support. A compact permission matrix encodes rules such as: fixed-reading text cannot support symptom-history inference, and a feature-only record cannot license a "direct listening" claim.

EviBound then works in three stages. A deterministic planner reads the package and decides which evidence routes are even legal, registering missing modalities up front so absence becomes part of the report contract. The evidence tools run in parallel — for audio-supported records the main route is a five-way acoustic consensus over openSMILE/eGeMAPS, segmented openSMILE, wav2vec2, HuBERT, and WavLM, each producing calibrated risk estimates and uncertainty signals. Applicable routes are combined by a weighted average using weights derived from historical calibration performance, and frozen omni-modal LLM outputs are used only as auxiliary evidence rather than as authoritative decision makers.

Finally, a boundary validator checks the draft report for modality hallucination, protocol misuse, and out-of-scope clinical claims, removing or rewriting offending claims and re-running schema validation. Crucially, the risk score is frozen before this repair step, so report cleanup cannot inflate AUROC or F1. Evaluation is then run on two separate tracks: predictive metrics (AUROC, Macro-F1, QWK) and boundary-consistency metrics (CVR, MHR, EBCPass), with uncertainty estimated from 2,000 paired bootstrap resamples.

Why This Matters

Impact on research. The paper argues that the field's central problem is not insufficient reasoning capability but the absence of explicit evidence-bound control. It turns dataset documentation and benchmark principles into executable runtime constraints rather than prose caveats, and it introduces boundary-aware metrics so accuracy and evidence validity can be reported side by side instead of collapsed into one score.

Potential real-world applications (the paper explicitly frames these as benchmark-level research evaluation, not clinical deployment or diagnosis):

• Screening and triage tooling that must refuse to infer from evidence it does not have.
• Tele-mental-health intake pipelines that mix free interviews, prompted speech, and text-only records under one system.
• Auditable report generation where unsupported claims must be blocked and missing evidence disclosed.
• Dataset documentation and model reporting workflows that need machine-checkable claims about modality availability and data provenance.

Industry relevance. Health technology companies, speech-analytics vendors, and clinical AI auditors care about systems that produce zero unsupported claims rather than marginally higher scores. The paper's funding includes Beijing Renyixun Health Technology Co., Ltd., and the release levels reported in Appendix B — 622 bounded automatic reports, 97 evidence-scoped reports, and 1,151 manual-review reports — illustrate how a deployment-oriented team might tier outputs by how much evidence backs them.

Future Directions

• Scale and protocol coverage. The authors note the benchmark's scale remains moderate, anxiety evaluation is heavily dominated by MMPsy interview records, and restrictive profiles such as fixed reading serve only as boundary-stress tests with limited sample sizes. They call for longitudinal sampling, multilingual coverage, and broader psychiatric conditions.

• Learned boundaries instead of deterministic rules. EviBound's rules are explicitly encoded for formal verifiability, which guarantees zero claim violations for known protocols but cannot catch failures outside the defined manifest schema, such as open-domain conversational hallucinations. The authors suggest neuro-symbolic approaches that learn boundary constraints while preserving verifiable safety guarantees.

• From offline screening to prospective validation. The work targets offline, structured screening under controlled package interfaces. Transitioning to real workflows requires clinician-in-the-loop auditing, robustness to noisy real-world inputs, and rigorous prospective validation.

• Cultural and linguistic bias. The benchmark pools English and Chinese resources acquired under different protocols and population characteristics, so both the predictive routes and the permission matrix may inherit cultural and linguistic biases that do not transfer to other languages, cultures, or acquisition settings.

Target Audience

Clinical NLP and multimodal machine learning researchers, particularly those working on speech-based mental health screening or benchmark design. Also relevant to applied scientists and product teams building health screening systems who need to reason about what a model is entitled to claim from a given recording, and to researchers working on factuality, hallucination mitigation, and evaluation of trustworthy medical AI. Readers interested only in predictive modeling will find the boundary-and-permission framing to be the distinctive part; readers interested in evaluation methodology will find the CVR, MHR, and EBCPass definitions and the separation of risk scoring from report validation to be the most transferable contribution.

Authors’ abstract

Computational mental health screening using multimodal speech and text has shown great promise. However, existing models often assume all clinical speech protocols carry equivalent evidentiary validity. In reality, heterogeneous protocols, from free interviews to fixed reading tasks, support fundamentally different evidence. Forcing uniform reasoning flattens these boundaries, causing models to hallucinate symptoms from irrelevant text or overclaim support. Even advanced long chain-of-thought LLMs fail to resolve this issue, as free-form reasoning can exacerbate boundary violations. To address this, we reformulate multimodal screening as an evidence-bounded reasoning problem. We introduce the Evidence Package Benchmark, integrating 1,870 packages across six heterogeneous sources with explicit modality masks and evidence permissions. We further propose EviBound, a protocol-aware evidence control framework. Unlike direct LLM prompting, EviBound uses a profile-aware planner to restrict reasoning scope, orchestrates evidence tools via five-way acoustic consensus, and enforces a boundary critic to suppress unsupported claims. Empirical results show EviBound achieves a held-out test Depression AUROC of 0.8658, exceeding the strongest direct omni-modal baseline by +0.0811 AUROC while maintaining zero claim violations. Our work moves beyond unconstrained accuracy toward evidence-consistent, protocol-aware systems for safer clinical NLP research.