M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents

AI summary

Overview

Research area: Clinical informatics / AI agent evaluation — specifically, whether AI coding agents can reproduce standard EHR-derived research variables (severity scores, phenotypes, rolling-window features) from raw clinical databases, and whether giving them written procedural guidance helps.

Technical level: Intermediate. Readers need some familiarity with electronic health record (EHR) databases, SQL, and LLM coding agents; no deep machine-learning background is required, but the clinical concept definitions and evaluation metrics take careful reading.

Scope: A 28-task, 15-family benchmark over MIMIC-IV and eICU that measures how clinician-reviewed markdown "skills" change an agent's agreement with declared reference derivations, with matched controls that decompose where the gain comes from.

What This Paper Is About

Clinical research on EHRs depends on derived variables — SOFA, KDIGO AKI staging, MELD, suspected infection, vasopressor equivalents — whose real definitions live outside the database schema, in item codes, time windows, unit conversions, missingness rules, and output-grain choices. Those choices change cohorts and downstream conclusions, yet no existing benchmark represents that procedural knowledge as an auditable artifact an agent can be evaluated against.

M4Bench asks whether a coding agent can reproduce declared EHR derivations, and whether supplying explicit, clinician-reviewed procedural specifications (as markdown "skills") measurably improves that reproduction.

Key Contributions

1. A benchmark object: 28 clinical EHR derivation tasks across 15 task families over MIMIC-IV and eICU (26 MIMIC-IV tasks, 2 eICU tasks; 11 standard and 17 raw modes), with clinician-reviewed skills, OS-level Docker isolation, contamination linting, sanitized per-run audit logs, and a family-level reference-validity audit.

2. A decomposition apparatus: decoy-skill, raw-reference-SQL, operational-spec, and schema-perturbation controls designed to separate specification completeness from delivery format, reference-code structure, and schema familiarity.

3. A Codex finding under that apparatus: targeted skills raise reward by +0.136 [0.085, 0.191] on the 28 native tasks (26/28 improving), while an eight-task control subset shows an operational spec alone captures +0.162 [0.072, 0.260] and skills add only +0.024 [-0.034, 0.099] on top.

4. A descriptive cross-provider replication: a full-coverage Claude Code run on all 28 tasks (Opus 4.7 and Sonnet 4.6, 560 canonical runs) shows a directionally consistent +0.120 [0.067, 0.177] with 21/28 tasks improving, reported as descriptive rather than provider-general evidence.

Main Findings

• Skills improve reward, but modestly: task-balanced reward rises from 0.703 without skills to 0.839 with them, a delta of +0.136 [0.085, 0.191], with 26 of 28 tasks improving. Per model, GPT 5.4 Mini moves from 0.639 to 0.802 (+0.162 [0.099, 0.232], 26/28) and GPT 5.5 from 0.767 to 0.876 (+0.109 [0.060, 0.166], 24/28). Family-cluster sensitivity is [0.065, 0.207], the family-balanced bootstrap gives +0.120 [0.057, 0.190], and the sign-flip test gives p < 10⁻⁴.

• Specification completeness, not packaging, drives the gain: on an eight-task control subset, an operational spec (complete non-SQL clinical rules without item identifiers or dataset recipes) lifts reward from 0.606 to 0.767 (+0.162 [0.072, 0.260]); adding the clinician-reviewed skill on top yields 0.792, a +0.024 difference [-0.034, 0.099] the authors call indistinguishable from zero.

• Generic context and file format are ruled out as explanations: on a ten-task subset, decoy skills score 0.554 versus 0.553 for no skill, and raw reference SQL nearly matches targeted skills (0.837 versus 0.833, delta 0.003).

• Schema familiarity explains only part of the effect: on GPT 5.4 Mini across the 26 MIMIC tasks, schema obfuscation preserves 86% of the skill gain (+0.135 [0.068, 0.208]) and schema restructuring preserves 70% (+0.110 [0.057, 0.170]). The authors explicitly describe these as descriptive robustness checks, not bounds on memorization.

• Strict end-to-end success is rare: only 87 of 760 runs pass every strict diagnostic — 13/280 no-skill and 25/280 with-skill in the primary comparison, plus 49/100 raw-SQL control runs against 7/100 targeted-skill runs on the same ten tasks. Of the 522 failing primary-comparison runs, 468 meet the key-coverage threshold and miss only a value-match threshold (0.85 to 1.0 per task, exact on 10 of 11 standard tasks).

• Row grain barely moves: task-balanced key F1 changes by only +0.003 [-0.010, 0.018], improving on 5/28 tasks and falling on 3, even though many skill cautions concern intervals and grain. The headline gain is concentrated in numeric value agreement on rows the agent already keyed correctly; ordered nMAE falls from 0.116 to 0.060.

• Gains concentrate in implementation-heavy families: rolling urine-output windows, MELD edge cases, suspected-infection timing, baseline-creatinine hierarchy, and ventilation status improve most. KDIGO, Charlson, standard vasopressor equivalents, SOFA, and several already-high-baseline scores stay near zero or negative. Grain errors concentrate in raw Sepsis-3, raw ventilation, and raw suspected infection (key F1 0.68 to 0.87).

• Claude replication is directionally consistent: reward goes from 0.747 to 0.867 (+0.120 [0.067, 0.177], 21/28 positive). Opus shows +0.083 [0.038, 0.135] (25/28) and Sonnet +0.156 [0.087, 0.231] (20/28). Restricting to 19 tasks with matched CLI version and routing profile gives +0.119 [0.054, 0.192] (14/19). Strict passes rise from 25/280 to 37/280 and key F1 by +0.008. The authors note the harness was not uniform (167/280 no-skill versus 100/280 with-skill runs on Claude Code 2.1.119) and label this a descriptive check, not a provider ranking.

• Control subsets over-represent large-effect tasks: the targeted-skill delta is +0.280 on the ten decoy/raw-SQL tasks versus +0.056 on the other 18, and +0.186 on the eight operational-spec tasks versus +0.116, which the authors disclose as a limitation on generalizing the mechanism conclusions.

• Base capability appears necessary: an exploratory local gpt-oss-20b arm, non-publishable under the release criteria, shows a near-zero paired delta of +0.008 [-0.072, 0.088], suggesting procedural specs do not substitute for coding capability.

Methodology in Plain English

Each task gives an agent a clinical derivation target and a DuckDB clone of the relevant database with the reference target tables removed (standard tasks retain some intermediate feature tables; raw tasks also remove the mimiciv_derived schema). The agent must write an output.csv with required identifier columns and value columns. The evaluator joins the agent's rows to hidden reference rows on the configured keys, penalizes missing, extra, and duplicate keys in every column's denominator, and computes reward as the mean match rate over scored value columns within task-specific numeric or timestamp tolerances. It also reports key precision and F1, row coverage, and strict pass/fail.

The core comparison is no-skill (task instruction plus database access) versus with-skill (the same inputs plus a task-specific markdown artifact summarizing clinical rules, item identifiers, time windows, missingness conventions, and output grain, but containing no executable SQL or answers). Skills and task specifications were each reviewed by two clinicians drawn from reviewers C1–C4, all of whom are co-authors, so the review is internal rather than external validation.

To find out why skills help, the authors added matched controls: a task-mismatched decoy skill (tests whether any extra context helps), raw reference SQL (tests whether exposing reference-code structure is the cause), an operational spec without database-specific recipes (tests whether completeness of the rules is the cause), and schema obfuscation/restructuring (probes schema-name familiarity). Runs execute inside Docker with fail-closed isolation, and post-run auditing excludes traces that mention answer paths, evaluator internals, answer-copying behavior, or byte-identical outputs. Uncertainty is reported as 95% non-parametric task-cluster bootstrap intervals, with family-cluster and family-balanced sensitivity analyses and a Monte Carlo sign-flip test.

Why This Matters

EHR-derived variables underpin cohort definitions, severity adjustment, and observational study conclusions, and small implementation choices — an item code, a time window, a missingness default — change who ends up in the cohort. M4Bench turns the tacit procedural knowledge held by clinicians into versioned, reviewable artifacts and measures whether agents can use them, giving the field a way to audit both the specification and the agent's residual errors.

Real-world applications:

• Cohort and phenotype construction: teams rebuilding sepsis, AKI, or ventilation cohorts can use the released skills as documented, reviewable starting points rather than re-deriving conventions from scratch.

• Agent development and validation: builders of clinical coding agents get a controlled test bed with isolation, contamination auditing, and per-column diagnostics instead of pass/fail unit tests.

• Research reproducibility: the per-run evidence packages, hash manifests, and reference-validity audit let others inspect exactly which derivation choices a benchmark target encodes and where public concept code diverges from clinical intent.

• Institutional governance: because skills are markdown artifacts with clinical provenance, hospital data teams can review and version them before an agent touches credentialed data.

Industry relevance: vendors building clinical data platforms, EHR analytics tools, and agentic coding assistants for health systems need exactly this kind of evaluation — one that separates "the model wrote plausible SQL" from "the derivation matches a declared, clinically reviewed convention" and that reports strict pass rates alongside partial-credit scores. The finding that a complete written specification captures most of the gain, while output-grain errors persist, is directly actionable for anyone shipping these systems into supervised research workflows.

Future Directions

• Run the mechanism controls on all 28 tasks. The decoy/raw-SQL and operational-spec subsets were purposive, fixed before first runs, and over-represent large-effect tasks (+0.280 versus +0.056), so the mechanism conclusions currently apply only to those subsets.

• Add clinician-adjudicated and multiple accepted reference targets. The current targets are public MIT-LCP and eICU concept code, which clinician review flagged as diverging from clinical definitions in places (for example, broad culture inclusion overcalling suspected infection, SOFA defaulting missing organ components to zero, and vasopressor mapping omitting angiotensin II).

• Broaden beyond MIMIC-IV. With 26 MIMIC-IV tasks and two eICU tasks used as cross-database probes, external validity requires more eICU and cross-site coverage, plus care-process task families.

• Run a causal Claude contrast. The full-coverage replication mixed Claude Code 2.1.119 and 2.1.220 partly by condition, recorded model-slot overrides for 230 runs, and shows ancillary claude-haiku-4-5 usage in 330 runs; a declared rerun under one frozen CLI version and routing policy would make the cross-provider comparison causal rather than descriptive.

Target Audience

Clinical informatics and EHR research teams who build or audit derived variables; AI researchers and benchmark designers working on coding agents and agent skills; developers of clinical data platforms and agentic tooling for health systems; and biostatisticians or study leads who need to understand how derivation choices propagate into cohorts and conclusions. Clinicians involved in phenotyping or concept curation will find the reference-validity audit and the skill artifact format most directly useful.

Authors’ abstract

Clinical research over electronic health records depends on derived variables whose definitions sit outside the schema, in item codes, time windows, missingness rules, and output-grain choices that change cohorts and downstream conclusions. We introduce M4Bench, 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU, to test whether AI coding agents can reproduce such derivations and how clinician-reviewed agent skills change their results. Its controls decompose the value of skills into specification completeness, delivery format, and schema familiarity. In a Docker-isolated Codex campaign (GPT 5.4 Mini and GPT 5.5), skills raise reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounts for most of this gain: on an eight-task control subset, an operational spec alone raises reward by 0.162, and skills add only 0.024 on top, indistinguishable from zero. On a ten-task subset, decoy skills score like no skill and raw reference SQL nearly matches targeted skills, arguing against generic context or the skill format as the explanation. Schema obfuscation and restructuring preserve 86% and 70% of the skill gain (GPT 5.4 Mini, 26 MIMIC-IV tasks), a descriptive robustness check rather than a bound on memorization. A descriptive Claude Code replication on all 28 tasks (Opus 4.7 and Sonnet 4.6, 560 runs) is directionally consistent (+0.120, 21 of 28 tasks improving). These gains are partial agreement with the reference, not completed tasks: only 25 of 280 with-skill and 13 of 280 no-skill Codex runs pass every strict check, and skills barely change which rows agents return (key F1 +0.003). Skills are released as a versioned, clinician-reviewable markdown delivery format, with a family-level reference-validity audit. M4Bench releases tasks, skills, evaluation code, and sanitized per-run evidence as an open-source benchmark.