Auto-MD-bench: Evaluating Autoresearch Agents for Real-World Medical Tasks

AI summary

Overview

Research area: Autonomous machine learning agents (LLM-driven "autoresearch" systems) applied to clinical prediction modeling, and benchmark design for evaluating them.

Technical level: Intermediate. The core ideas are accessible, but readers benefit from familiarity with standard ML evaluation concepts such as AUROC, survival concordance, and probability calibration.

Scope: The paper introduces Auto-MD-bench, a 15-task benchmark built from 10 clinical datasets, and uses it to show that optimizing agents for a single primary metric produces models that score well but are poorly calibrated and inconsistent under population shift.

What This Paper Is About

Autonomous agents can now build machine learning models end to end, including for medicine, but they are typically guided by one headline number such as AUROC. In real clinical use, a model also needs to give trustworthy absolute risk estimates and keep working when applied to a different hospital, cohort, or time period. The paper asks whether the standard task objective is enough to steer agents toward models that are actually suitable for medical use, and builds a benchmark to test this directly.

Key Contributions

1. A new benchmark, Auto-MD-bench, covering 15 real-world medical prediction tasks drawn from 10 clinical datasets, spanning survival analysis, classification, and sequential prediction. Ten tasks use internal validation; five use external validation across a different cohort, hospital, or time period.
2. Multi-dimensional evaluation: each agent-developed model is scored on primary task performance, risk calibration (expected calibration error), performance under population shift, and subgroup performance, rather than on the primary metric alone.
3. A controlled objective experiment on UCI Diabetes showing that explicitly writing calibration into the task description changes agents' modeling choices and sharply improves calibration while preserving discrimination.
4. Process-level analysis: the authors collect and annotate the agents' code and development records to link specific modeling practices (using averaged ranks as probabilities, class-imbalance handling without recalibration, censoring errors, real-time rule violations) to the observed performance gaps.

Main Findings

• Agents beat the baseline on the headline metric but not on calibration. Codex and Autoresearch reach mean normalized scores of 0.517 and 0.518 versus 0.487 for the simple baseline models, yet their mean ECE values are 0.086 and 0.075 versus 0.027 for the baselines — roughly three times as large. AIDE scores 0.474 with ECE 0.128; AI Scientist v2 scores 0.355 (0.467 excluding invalid runs) with ECE 0.072.
• Nearly identical discrimination can hide very different calibration. Two individual UCI Diabetes submissions from Codex and AI Scientist v2 have AUROC values of 0.670 and 0.671, but ECE values of 0.006 and 0.154.
• Two modeling practices explain most of the calibration gap. In the analysis of contributions to the ECE difference from baseline (Figure 2), experiments that used averaged ranks as probabilities account for 75% of the difference and 14% of valid experiments; experiments handling class imbalance without recalibration account for a further 18% and 9% of valid experiments. Among experiments handling class imbalance without averaged ranks, mean ECE is 0.008 with subsequent recalibration and 0.160 without.
• Stating calibration in the objective helps. On UCI Diabetes, an AUROC-only objective yields AUROC 0.676 ± 0.023 and ECE 0.172 ± 0.211; adding ECE yields AUROC 0.683 ± 0.016 and ECE 0.016 ± 0.023. A Brier-score objective lowers ECE to 0.061 ± 0.157 but lowers AUROC to 0.640 ± 0.107.
• Degradation under population shift varies widely by task. On Rotterdam → GBSG, scores under shift are 85–86% of the no-shift scores for the research agents and 97–99% for the coding agents. Ratios fall to 41–63% on Sepsis, new hospital and 34–67% on Sepsis, 500 local labels, while SEER, later period and eICU, new hospitals stay close to 100%. Codex and Autoresearch beat the simple baseline on all five external tasks and on four of five respectively.
• Subgroup gaps are masked by aggregate scores. In the four ICU mortality tasks, the oldest age tertile is the lowest-scoring age group in every valid experiment, with a highest-to-lowest age-group difference of 0.065 to 0.084 (baselines: 0.065 to 0.078). Agents checked performance by age or sex in only 6 of 168 valid experiments.
• Modeling choices are fairly uniform, but validation discipline is not. Gradient-boosted trees appear in 76–89% of valid experiments. Codex and Autoresearch average across seeds or folds in 89% of experiments versus 62% for AIDE and 31% for AI Scientist v2; AI Scientist v2 uses a single model in 36% of experiments and a single holdout split in 37%. Explicit probability calibration appears in only 2–22% of experiments across agents.
• Correct scores can coexist with incorrect modeling. Eight of 79 valid survival submissions mishandle censored outcomes (e.g., treating censored patients as observed deaths), and all eight still exceed their task's simple baseline score. A masking test found real-time rule violations in three of the 13 valid sequential prediction experiments tested for AIDE and AI Scientist v2, all three from AIDE.
• The underlying LLM shifts calibration more than discrimination. Holding Codex fixed, Qwen3.8 Flash, gpt-6-luna, and Gemini 3.8 Flash score 0.517, 0.517, and 0.520, while their mean ECE values are 0.086, 0.041, and 0.029 respectively — the last close to the baseline's 0.027.
• Comparison to prior benchmarks. Of the benchmarks listed in Table 1 (MLE-bench, HealthAgentBench, MedAgentBoard, MedAgentGym, AutoHealth, ACP-Bench, BioML-bench, BioXArena), only AutoHealth and ACP-Bench report calibration, only BioXArena reports external validation, and Auto-MD-bench is the only one reporting both.

Methodology in Plain English

The team assembled 15 prediction tasks from 10 clinical data sources: survival tasks such as SUPPORT, METABRIC, Rotterdam, SEER, and NHANES; classification tasks such as UCI Diabetes readmission, PhysioNet 2012, MIMIC-IV, and eICU mortality; and three sepsis-onset sequential prediction tasks derived from PhysioNet/CinC 2019. Each task gives the agent a written description, labeled training data, test features without labels, a null prediction file, and a format checker. Agents write and run code inside sandboxed Apptainer containers with no internet access, and the test labels never enter that environment.

Four agents were tested: Codex, Autoresearch (Codex following a step-by-step "change one thing at a time" protocol), AIDE (greedy tree search), and AI Scientist v2 (best-first tree search over four stages). All four used Qwen3.8 Flash in the main experiments, ran three repetitions per task with a 12-hour wall-clock budget on a machine with eight CPU threads and 16 GB of RAM, and had their submitted predictions scored afterwards outside the agent environment.

Scores are normalized against each task's null predictions using a formula that maps the raw score onto a 0–1 scale, with 0.5 treated as the null value for concordance and AUROC and 0 for the other metrics. Survival is scored with adjusted Antolini time-dependent concordance, classification with AUROC except PhysioNet 2012 which uses the minimum of sensitivity and positive predictive value, and sequential prediction with the PhysioNet/CinC 2019 normalized utility. Comparisons are made against deliberately simple baselines: Cox neural networks from pycox for survival and scikit-learn histogram-based gradient-boosted trees for the other tasks, each trained five times and averaged.

Calibration is measured with expected calibration error, adapted for survival (comparing grouped mean predictions against Kaplan–Meier estimates at three time points) and for sequential prediction (computed over patient-hours). Beyond the numbers, the authors had Claude Opus 5.5 annotate the agents' code by file and line number to identify model families, validation strategies, and class-imbalance and censoring handling, and they also compared code across successive submission versions to see how search behavior evolved.

Why This Matters

Impact on research. The paper reframes how autonomous modeling agents should be judged in high-stakes domains: a leaderboard number on a single metric is a partial description of model quality. It provides an evaluation template that combines discrimination, calibration, external validation, and subgroup performance, and it links outcome differences back to concrete, inspectable modeling choices.

Real-world applications:

• Clinical risk scoring tools used for screening or treatment decisions, where an inaccurate absolute risk estimate can misdirect care even when ranking is good.
• Hospital deployment across sites, where a model trained at one institution is applied at another with different coding practices and patient mix — the sepsis tasks show scores can fall to 34–67% of their in-domain value.
• Regulatory and procurement review of AI-based medical software, where documentation of calibration and external validation is a recognized requirement rather than an optional extra.
• Model monitoring and governance, where subgroup performance checks (such as the age-tertile gaps found here) expose disparities that aggregate scores conceal.

Industry relevance. Teams building or purchasing automated ML pipelines, whether in health systems, insurers, or medical AI vendors, can use these findings to set task specifications that include calibration targets, and to treat agent-generated code as something that needs auditing rather than just benchmarking.

Future Directions

• Objective design at scale. The calibration-in-the-objective experiment was run only on UCI Diabetes; extending it across all 15 tasks, and testing objectives beyond AUROC, Brier score, and AUROC-plus-ECE, is the natural next step.
• Default calibration behavior. Since explicit probability calibration appears in only 2–22% of experiments, an open question is whether agent scaffolds should be prompted or constrained to calibrate by default, rather than depending on the task description.
• Auditing agent-generated clinical code. The censoring-handling errors and real-time rule violations found here suggest a need for automated correctness checks on the modeling logic itself, not just on the output predictions.
• Model and scaffold sensitivity. With three LLMs producing near-identical primary scores but ECE values from 0.029 to 0.086, further work could disentangle how much of the calibration gap comes from the base model versus the agent's search strategy.

Target Audience

Researchers working on autonomous ML agents and agent benchmarks; clinical informatics and biostatistics groups who build or review prediction models; medical AI developers and regulators concerned with calibration and external validity; and machine learning practitioners interested in how the choice of optimization target shapes the properties of the resulting model.

Authors’ abstract

Autonomous agents are increasingly capable of developing machine learning (ML) models end to end. In medical practice, however, strong performance on the target metric does not fully characterize how good the models are in real-world use cases. For example, medical ML models should also provide well-calibrated risk estimates and remain robust across patient populations. Yet autonomous model development is typically guided by task objectives that emphasize simple predictive discrimination (e.g. AUROC). This raises a key question: are these objectives sufficient to guide agents toward models that meet the broader requirements of real-world medical use? In this study, we introduce Auto-MD-bench, a benchmark of 15 real-world medical tasks from 10 clinical datasets, spanning survival analysis, classification and sequential prediction, with external validation across cohorts, hospitals and time periods. We evaluate four agents: AIDE, AI Scientist v2, Codex, and Autoresearch. The strongest agents achieve competitive primary scores, yet their predicted risks are less calibrated than those of the simple baseline models, which could lead to severe medical accidents. Moreover, models with nearly identical primary scores can exhibit markedly different calibration, while performance varies substantially across population shifts. These findings show that primary task performance alone is insufficient to assess autonomous clinical modeling and multiple different metrics widely used in healthcare practice need to be taken into consideration.