ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation

AI summary

Overview

Research area: Clinical natural language processing and multimodal large language models (MLLMs) applied to electrocardiography (ECG) interpretation, with supporting theory on robustness and cross-modal consistency under missing modalities.

Technical level: Advanced. The paper combines corpus construction, a dual-encoder MLLM architecture, reinforcement learning with custom rewards, and formal guarantees (Theorems 2.2 and 2.3) about worst-environment risk and cross-modal discrepancy.

One-sentence scope: ECG-R1 is a reasoning ECG MLLM whose interpretation corpus, architecture, and reinforcement-learning rewards are all designed to reduce hallucination and to keep performance stable when either the ECG signal or the ECG image is missing.

What This Paper Is About

Existing multimodal large language models remain unreliable at ECG interpretation: they can produce fluent, structurally complete analyses that are clinically incorrect, and most support only ECG images, not the raw time-series signal. The authors argue two root causes: training corpora generated by "purely prompting" general-purpose LLMs (so interpretations reflect pretrained priors rather than explicit diagnostic criteria), and architectures that assume paired image-plus-signal inputs, which break down when a modality is absent. ECG-R1 is proposed to address both, and the paper additionally provides "the first quantitative evidence that severe hallucinations are widespread" in current proprietary, open-source, and medical MLLMs, concluding that the public should not directly trust these outputs without independent verification.

Key Contributions

1. ECG-R1, described as the first reasoning ECG MLLM for ECG interpretation, further optimized with reinforcement learning using ECG Diagnostic Evidence Rewards (EDER) to reward structured intermediate reasoning, not only final-answer correctness.
2. Protocol-Guided Instruction Data Generation, which combines a deterministic grounding feature extractor (FeatureDB) with a five-phase protocol derived from Chapter 23 ("How to Read an ECG") of ECG from Basics to Essentials: Step by Step (Stroobandt et al., 2015), producing structured six-step analyses with a final summary and diagnosis while surfacing plausible unannotated abnormalities.
3. Interleaved Modality Dropout (IMD), a theoretically motivated training strategy that randomly drops either modality and optionally swaps the order of the token blocks, with stated theoretical guarantees for robustness and cross-modal consistency.
4. A systematic evaluation of ECG interpretation across proprietary, open-source, and medical MLLMs, plus evaluation by four licensed cardiologists comparing GEM (the strongest baseline) and ECG-R1 on 100 randomly sampled test cases.

Main Findings

• Non-specialized MLLMs fail on ECG. Among all non-ECG-specialized models, GPT-5.1-Instant achieves the highest Diagnosis Accuracy, yet only 31.48. Existing medical MLLMs stay consistently below 30.00 Diagnosis Accuracy (for example MedGemma-4B 27.34, MedGemma-27B 25.23, QoQ-Med-VL-7B 27.01, HuatuoGPT-Vision-7B 29.27, Chiron-o1-8B 21.20, MedVLM-R1 16.62).

• Structural fluency masks systematic hallucination. Many non-specialized models score relatively high on Analysis Completeness but poorly on Analysis Relevance, Lead Evidence Validity, and ECG Feature Grounding — the paper reads this as generating plausible-sounding but incorrect content that leads to wrong diagnoses.

• ECG-R1 leads on all seven grounded interpretation metrics. ECG-R1 (RL) reaches Diagnosis Accuracy 80.29 versus GEM 74.70 and PULSE 66.13; Analysis Completeness 6.51 (GEM 4.25), Analysis Relevance 4.74 (GEM 3.79), Lead Evidence Validity 5.81 (GEM 4.41), ECG Feature Grounding 80.57 (GEM 65.34), Evidence-Based Reasoning 79.08 (GEM 63.15), and Clinical Diagnostic Fidelity 84.20 (GEM 62.90). The SFT model scores 79.33, 6.36, 4.58, 5.53, 79.92, 78.08, and 83.51 respectively.

• Reported gain over GEM. ECG-R1 yields a +17.49 average absolute gain over GEM on ECG Feature Grounding, Evidence-Based Reasoning, and Clinical Diagnostic Fidelity.

• RL improves over SFT on every metric, which the authors attribute to evidence-rewarded reinforcement learning strengthening reasoning.

• GEM degrades sharply under modality missing. When only the time-series is provided and the image is entirely missing, GEM's Diagnosis Accuracy suffers a maximum relative drop of 28.0%, and Analysis Relevance a maximum relative drop of 44.9%. ECG-R1 shows consistently smaller relative drops and still surpasses GEM-with-both-modalities when one modality is entirely absent.

• ECG-R1 is more cross-modally consistent. Comparing time-series-only versus image-only interpretations: GEM scores BLEU-4 0.33, ROUGE-L 0.43, SBERT-Score 0.92; ECG-R1 scores 0.69, 0.73, and 0.97.

• Cardiologists rate ECG-R1 highest. Four licensed cardiologists scored 100 randomly sampled cases across seven criteria. ECG-R1's seven ratings were 4.55/5 (0.53), 4.34/5 (0.66), 4.43/5 (0.58), 4.48/5 (0.57), 3.25/5 (1.78), 4.38/5 (0.64), and 4.38/5 (0.63); GEM's were 4.16/5 (0.78), 3.89/5 (0.89), 4.05/5 (0.71), 4.03/5 (0.73), 2.82/5 (1.59), 3.84/5 (1.00), and 3.84/5 (0.99). The text identifies the second value as Analytical Accuracy (ECG-R1 4.34 vs. GEM 3.89), a criterion where lower scores indicate more frequent or severe factual errors. The table's extracted header lists Reliability, Usefulness, Analytical Relevance, Analytical Accuracy, Analytical Completeness, Reasoning Quality, Findings Novelty, Clinical Value, and Overall Satisfaction, but each row contains only seven values and the paper states there are seven criteria, so the exact value-to-label mapping beyond Analytical Accuracy cannot be resolved from the provided content. GPT-5.1-Instant and PULSE were also rated, with PULSE scoring slightly higher than GEM on Analytical Accuracy but lower on Completeness and Reasoning Quality.

• Findings Novelty remains case-dependent. The paper reports that this criterion improves but shows greater inter-rater variability.

Methodology in Plain English

Building the supervision. Rather than asking a general-purpose LLM to describe an ECG from its labels, the authors first run a deterministic, non-trainable feature extractor (FeatureDB) over the raw ECG time-series. For each lead and detected heartbeat it measures fiducial-point amplitudes, waveform morphology, and key intervals, organized into beat-wise, time-ordered sequences — 14 sequences per lead across 12 leads, covering heart rate, RR intervals, P amplitude and duration, PR interval, QRS amplitude and duration, T amplitude and duration, ST descriptors, and QT/QTc intervals. These features are combined with a protocol-aware prompt built from the monograph's procedure, reorganized into five phases: (i) Technical, Rate & Rhythm; (ii) Conduction, Axis & Intervals; (iii) Chamber Hypertrophy & Voltage; (iv) Ischemia, Infarction & Mimics; and (v) Electrolytes & QT, with explicit negatives to rule out key mimics. DeepSeek-V3.1-Terminus acts as the interpretation generator over MIMIC-IV-ECG, producing responses with a <think> block containing a six-step reasoning trace (five phases plus a final medical reasoning step), a brief summary, and an <answer> block with the final diagnosis. Repeating this yields 30,000 protocol-guided instruction samples.

Making the model handle either modality. Instead of the coupled design used by prior ECG omni-perception MLLMs — where time-series embeddings are aligned to images and then pushed through a single image-to-language projector into the <image> token space — ECG-R1 uses Qwen3-VL-8B (LLM plus visual encoder) and ECG-CoCa (time-series encoder) with two independent projectors. An explicit <ecg> tag, placed before <image>, marks where the time-series token block is injected, so the model encodes the signal only when that tag is present. Only the projectors and the LLM are trained.

Training in two stages. A one-epoch supervised fine-tuning pass runs on the union of the protocol-guided samples and the public ECGInstruct dataset, with ECG images synthesized from raw signals using ECG-image-kit. During both SFT and RL, IMD injects transformations drawn from a small environment set of four possibilities: drop the image, drop the time-series, swap block order one way, or swap it the other. Practically, a modality-drop trial happens with probability p_d, and, conditioned on keeping both modalities, a token-swap trial happens with probability p_s. The authors show that optimizing this mixture risk bounds the worst-environment risk (Theorem 2.2, with α = min{p_d/2, (1−p_d)p_s, (1−p_d)(1−p_s)}) and, because ECG images and signals are two renderings of the same waveform, the intrinsic view asymmetry is negligible, which lets excess risk control cross-modal discrepancy (Theorem 2.3).

Reinforcement learning with evidence rewards. The RL set is built by taking the top-500 most frequent report texts, keeping up to 10 uniformly sampled examples per report type (all when fewer than 10 exist), and globally shuffling with a fixed seed of 42, giving 3,948 training samples. DeepSeek-V3.1-Terminus extracts step-specific key diagnostic evidence phrases relevant to the ground-truth labels, retaining up to three salient phrases per step and capping each phrase at six words. Rewards are: a step reward measuring the fraction of those evidence phrases that appear in the generated step text, averaged over the K=6 steps to form the EDER reward; a diagnosis accuracy reward computed as set-level Jaccard similarity between the predicted and ground-truth diagnosis label sets; and a format reward of 1 if the output contains a non-empty <think> block. The total reward is format + accuracy + λ·EDER, optimized with DAPO using group-normalized advantages and decoupled clipping with ε_low = 0.2 and ε_high = 0.3.

Evaluation. Grounded ECG interpretation uses seven rubric-based metrics on the ECG-Grounding test set of 2,381 samples. To avoid depending on the older closed-source grader GPT-4o, the authors score outputs with DeepSeek-V3.1-Terminus under the same rubric and additionally introduce GLM-5 as an independent verifier. Robustness is tested by randomly dropping either modality under the full-modality setting; consistency is measured by scoring time-series-only and image-only inputs on the same cases with three text-similarity metrics.

Why This Matters

Research impact. The work reframes ECG MLLM quality as a data-provenance problem and a process-supervision problem, not only a scale problem: explicit monograph thresholds replace LLM priors in corpus construction, and stepwise evidence rewards replace answer-only rewards. It also supplies quantitative hallucination evidence across proprietary, open-source, and medical MLLMs, and offers formal statements linking a training-time dropout mixture to worst-case robustness and cross-modal agreement.

Real-world applications.

• Clinical decision support and triage in cardiology, where an automated reading must be cross-checked against leads and protocols before being confirmed by a clinician.
• Handling incomplete records: many workflows have either a digitized waveform or a rendered ECG image but not both, so a model that stays reliable with one modality missing is more deployable.
• Automated quality-checking or pre-screening of existing ECG reports to flag implausible statements before clinician review.
• Structured reporting that mirrors the monograph's five-phase sequence (rate and rhythm, conduction and intervals, hypertrophy and voltage, ischemia and infarction, electrolytes and QT), which aligns output with how clinicians already read ECGs.

Industry relevance. Two co-authors are Tencent employees (Jarvis Lab) and one is affiliated with HeartVoice Medical Technology, indicating commercial interest in deployable ECG interpretation products. The explicit caution that the public should not directly trust MLLM ECG outputs without independent verification is directly relevant to any product that exposes ECG interpretation to non-specialists, and to regulatory expectations for clinical AI.

Future Directions

• Generalization beyond the current corpus. All training data derives from MIMIC-IV-ECG and a single monograph protocol; whether the protocol-guided pipeline and IMD guarantees hold across other hospitals, devices, populations, and alternate labeling schemes is not established.
• Extending the theoretical guarantees to genuinely asymmetric views. The consistency result leans on ECG image and signal being two renderings of the same waveform with negligible intrinsic view asymmetry; multi-view settings where each view carries distinct information (their example contrasts speech semantics with spatial vision) are left open.
• Broadening evaluation coverage. Cardiologist review covered 100 sampled cases and four raters, with Findings Novelty remaining case-dependent and showing high inter-rater variability; larger and more varied human evaluation could clarify where the model still fails.
• Reducing dependence on LLM-generated supervision. Both the reference interpretations and the extracted evidence phrases come from DeepSeek-V3.1-Terminus, and grading is done by DeepSeek-V3.1-Terminus with GLM-5 as an independent verifier; whether the approach can be grounded in expert-written or fully verifiable supervision is not reported.

Target Audience

Researchers and engineers working on medical multimodal LLMs, ECG or time-series foundation models, and clinically deployable reasoning systems will benefit most, along with clinicians and health-AI reviewers who need to judge how much to trust automated ECG

Authors’ abstract

Electrocardiography (ECG) serves as an indispensable diagnostic tool in clinical practice, yet existing multimodal large language models (MLLMs) remain unreliable for ECG interpretation, often producing plausible but clinically incorrect analyses. To address this, we propose ECG-R1, the first reasoning ECG MLLM designed for reliable ECG interpretation via three innovations. First, we construct the interpretation corpus using \textit{Protocol-Guided Instruction Data Generation}, grounding interpretation in measurable ECG features and monograph-defined quantitative thresholds and diagnostic logic. Second, we present a modality-decoupled architecture with \textit{Interleaved Modality Dropout} to improve robustness and cross-modal consistency when either the ECG signal or ECG image is missing. Third, we present \textit{Reinforcement Learning with ECG Diagnostic Evidence Rewards} to strengthen evidence-grounded ECG interpretation. Additionally, we systematically evaluate the ECG interpretation capabilities of proprietary, open-source, and medical MLLMs, and provide the first quantitative evidence that severe hallucinations are widespread, suggesting that the public should not directly trust these outputs without independent verification. Code is available at \href{https://github.com/PKUDigitalHealth/ECG-R1}{here}.