ECG-RSI: Recursive Self-Improvement through Targeted Electrocardiogram Waveform Evolution

AI summary

Overview

• Research area: Artificial intelligence for medical signal understanding — specifically multimodal large language models that interpret electrocardiogram (ECG) waveforms, combined with recursive self-improvement (RSI) and synthetic training-data generation.
• Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, benchmark evaluation protocols, multimodal model architectures, and self-improvement/data-evolution literature.
• Scope: The paper introduces ECG-RSI, a framework that couples verified ECG waveform synthesis with feedback-driven model improvement, and reports its effects across seven benchmarks plus two validation sets over three improvement rounds.

What This Paper Is About

ECG interpretation models need training data that targets whatever the model is currently getting wrong, while every question and answer must stay faithful to the actual waveform shown. The problem is that turning a specific learner weakness into a controlled, verifiable training example is hard: a generated waveform must express the intended ECG feature without disturbing other features, and the accompanying question and answer must agree with the resulting signal. The paper's goal is to build a loop in which construction failures repair the data generator, learner errors steer the next round of data synthesis, and candidate models are compared on a held-out selection set before any update is accepted.

Key Contributions

1. A framework separating two feedback loops. ECG-RSI distinguishes construction feedback (used to repair the data-generation pipeline) from learner feedback (used to propose and compare subsequent training interventions), with model promotion decided on a separate selection set.

2. Target-driven reverse waveform synthesis. Desired ECG feature contrasts are translated into constrained signal perturbations, the edited waveform is remeasured, and supervision is accepted only when target and protected-feature errors fall within their tolerances — yielding measurement-grounded supervision.

3. An analysis of how intervention utility changes with the learner. Across the retained trajectory, the preferred supervision shifted from direct reinforcement toward confusion-focused contrasts, and continued feedback correction produced diminishing gains on external benchmarks.

4. An empirical separation of error diagnosis from intervention selection, including evidence that the most prominent weakness does not by itself identify the best training direction.

Main Findings

• Large gains over three rounds on the starting model. Across seven benchmarks, three rounds raised Qwen3.5-35B-A3B's mean question-answering accuracy from 34.57% to 56.04% (a 21.47-percentage-point gain) and its mean classification F1 from 24.27 to 34.03 (a 9.77-point gain).

• Highest aggregate among compared models. ECG-RSI R3 achieves the highest S7 score (46.61) among the compared proprietary, open-source, and ECG-specific models, starting from R0's S7 of 30.15 — a total difference of +16.46.

• Individual metrics do not all peak at the last round. Along series R-A, S7 rises from 30.15 (R0) to 43.22 (R1) to 46.23 (R2) to 46.61 (R3), but G12EC accuracy, ECG-QA accuracy, CPSC2018 F1, and CODE-15% F1 peak at R2 and decline slightly at R3.

• Validation accuracy improves overall. MedX accuracy rises from 51.57% to 57.24% (despite a small R2 decline of -0.08), and LUDB accuracy improves at every round from 53.45% to 67.24%.

• Feedback error rates fall across all difficulty levels, but L3 stays hardest. L1 errors fall from 26.11% to 12.97%, L2 from 38.52% to 24.43%, and L3 from 58.20% to 38.17%; L3 has both the largest reduction and the highest R3 error rate.

• Errors keep falling while benchmark gains diminish. The paper reports that continued feedback improvement yields diminishing external gains, with errors falling as S7 gains diminish.

• Synthesis has verified numerical fidelity. In a single run of 10,000 samples, all six measured features met the adopted MAE criteria: HR MAE 2.14 (limit ≤3), RR 15.28 (<20), PR 7.71 (≤25), QRS 4.58 (≤30), QT 9.71 (≤30), and QTc 10.22 (≤40).

• Physician review supports waveform and answer quality. Two independent physicians reviewed 240 examples (40 per edit type): target attainment 0.82, non-target preservation 0.79, morphology consistency 0.86, lead consistency 0.83, answer accuracy 0.88, and explanation support 0.76, with inter-rater κ = 0.78. Explanation support was lower than answer accuracy for every edit type.

• Reverse synthesis and feedback-guided selection both add value. From a shared checkpoint at S7 = 43.22, ordinary training-side questions reached 44.15, forward-only synthesis 44.27, forward plus reverse synthesis with fixed selection 45.56, and forward plus reverse with feedback-guided selection 46.23.

• Finer feedback produces more useful training data. With the same 10,000 added questions, S7 rose from 45.56 with no feedback, to 45.60 with ECG category, 45.98 with category plus question type, and 46.23 with category, type, difficulty, and confusion pairs.

• Freezing generator updates hurts both efficiency and results. Reaching 10,000 accepted questions required 16,193 candidates for the full method, versus 36,723 with operator updates frozen (2.27 times as many) and 22,398 with skill updates frozen (1.38 times as many); downstream S7 dropped to 39.29 and 40.15 respectively.

• The dominant confusion does not determine the best intervention. In the R0→R1 comparison, direct reinforcement won with S_sel = 42.66 despite a 52% Mobitz I/II confusion error in the parent, beating the contrast-focused branch (40.51) and the curriculum branch (38.27). In the later comparisons the confusion-focused branch won: 45.08 versus 44.12 and 43.55 in R1→R2, and 46.03 versus 45.90 and 44.71 in R2→R3.

Methodology in Plain English

The system runs as a repeating loop with two distinct feedback channels.

The data factory. Rather than only generating questions from existing ECG recordings, the framework also works in reverse: it starts with a waveform and a requested change (for example, "make this PR interval prolonged"), applies a constrained edit, then re-measures the signal with a delineator that extracts heart rate, RR, PR, QRS, QT, QTc, and amplitudes. A candidate edit is accepted only if the intended change lands inside its target range and the protected features drift no more than their tolerances, with additional checks on boundary continuity, lead consistency, morphology, leakage, diversity, and cost. Only then are the question, answer, context, and explanation generated from the resulting waveform. When things fail, the failure type points to a specific repair: signal errors prompt operator revisions, missing morphology prompts retrieval changes, ambiguity prompts question/context revisions, and unsupported explanations prompt prompt revisions. Old and revised versions are compared on matched sources, requests, seeds, and budgets before adoption.

The training loop. The model's errors, invalid outputs, and regressions on a feedback set are diagnosed by category, learner-relative difficulty, and supervision type. Difficulty is measured empirically by sampling K independent answers per training question under fixed decoding settings and grouping examples into reliably solved, mixed-success, and unresolved. Recipes then specify sampling quotas and construction requests.

Comparing candidates fairly. From the same parent model, three recipe families are built: A (direct reinforcement of weak categories), B (confusion contrast, targeting recurring confusion pairs), and C (difficulty and task curriculum). Each branch adds 10,000 previously unused accepted questions on top of the same historical manifest, and all branches use three epochs of supervised fine-tuning with matched settings, updating the language model while freezing the vision encoder and multimodal aligner. Candidates are scored on a separate selection set, and only the highest-scoring checkpoint passing the predesignated output-validity and non-target regression tolerances is retained; otherwise the parent stays. Independent evaluation data is held out from both generation and selection.

Why This Matters

Impact on research. The paper makes an argument that error diagnosis and intervention selection should be treated as separate problems, and that gains measured on a feedback set must be checked against independent benchmarks. It also supplies a concrete case where the most salient weakness (a 52% Mobitz I/II confusion) did not point to the best intervention, which challenges the intuition that the biggest error cluster should drive the next training round.

Real-world applications.

• Training ECG interpretation assistants that remain grounded in the actual rendered waveform rather than in memorized clinical text.
• Generating targeted, clinically reviewed practice material for cardiology trainees, stratified by rhythm, interval, and morphology findings.
• Building condition-controlled ECG data for rare or underrepresented diagnostic categories where real recordings are scarce.
• Auditing existing ECG models by measuring how their predictions change under controlled, verified waveform edits.

Industry relevance. The framework targets a practical bottleneck in medical AI: obtaining supervision that is both adaptive to a specific model's weaknesses and verifiable against the underlying signal. The reported efficiency numbers — 16,193 candidates for 10,000 accepted questions with the full method, versus 36,723 when operator updates are frozen — speak directly to the cost of synthesizing regulated-domain training data. The paper also reports that closed-source models (GPT-5.6-sol at S7 = 44.50 and Gemini-3.8-Flash at 44.49) are outperformed by ECG-RSI R3 at 46.61 on these seven benchmarks.

Future Directions

• Determine why external gains diminish while feedback errors keep falling. The paper reports that errors continue to fall as S7 gains shrink, which raises the question of whether the model is approaching a benchmark ceiling, overfitting the feedback distribution, or learning behaviors that the seven benchmarks do not measure.

• Explain why the salient confusion is not always the best intervention. The first round favored direct reinforcement over the confusion-contrast branch despite a 52% Mobitz I/II confusion error. Understanding when contrast-based supervision overtakes reinforcement would make recipe planning more principled.

• Test whether these results transfer beyond the starting model. All reported evolution results center on Qwen3.5-35B-A3B; the paper does not report whether the same trajectory, branch preferences, or efficiency ratios hold for other base models.

• Extend beyond three rounds and beyond the current task coverage. The retained trajectory covers three update rounds and 23 ECG categories, and the paper does not report how the selection preferences or diminishing-returns pattern would evolve with longer horizons or an expanded task set.

Target Audience

Researchers and practitioners working on multimodal medical AI, synthetic data generation, and recursive or self-improving training pipelines will get the most from this paper. It is also relevant to clinical informatics teams building ECG interpretation tools, and to evaluation-focused researchers interested in how feedback-set improvement relates to independent benchmark performance. Readers need a working knowledge of supervised fine-tuning and benchmark methodology to follow the experimental sections.

Authors’ abstract

Electrocardiogram (ECG) models need supervision that targets evolving weaknesses and remains faithful to waveform evidence. We introduce ECG-RSI, a recursive self-improvement framework combining verified waveform synthesis with feedback-guided intervention search. Signal remeasurement verifies waveform edits; construction errors guide generator repair, while learner errors motivate interventions compared on a separate selection set. Across seven benchmarks, three rounds raise Qwen3.5-35B-A3B's mean question-answering accuracy from 34.57% to 56.04% and mean classification F1 from 24.27 to 34.03. Across the retained trajectory, the preferred supervision shifts from direct reinforcement to confusion-focused contrasts. Continued feedback correction yields diminishing external gains. These results motivate separating error diagnosis from intervention selection and checking whether gains on the feedback set transfer to independent benchmarks.