ReX-MLE: The Autonomous Agent Benchmark for Medical Imaging Challenges

AI summary

Overview

Research area: Autonomous AI agents and machine learning engineering benchmarks, applied to medical imaging (computer vision for healthcare).

Technical level: Intermediate. The paper is readable without deep medical imaging background, but assumes familiarity with concepts such as segmentation, detection, Dice scores and ML training pipelines.

Scope: The paper introduces ReX-MLE, a benchmark of 20 medical imaging challenges from 10 competitions used to measure whether autonomous coding agents can complete full end-to-end medical imaging workflows.

What This Paper Is About

Autonomous coding agents built on large language models can now handle many general machine learning and software tasks, but they appear to break down on complex, domain-specific scientific problems. This paper asks whether the leading agents (AIDE, ML-Master, R&D-Agent) can independently solve real medical imaging challenges — from data preprocessing through training to producing submission-ready prediction files — under realistic compute and time limits. The answer the authors report is largely no: most agent submissions rank in the 0th percentile against human competitors.

Key Contributions

1. A new benchmark, ReX-MLE. The authors assemble 20 challenges derived from 10 high-impact medical imaging competitions on the Grand Challenge platform, spanning 8 imaging modalities and task types including segmentation, detection, classification, image quality assessment, and generative enhancement.
2. An end-to-end evaluation protocol. Agents must produce complete submission-ready prediction files (for example NIfTI volumes, masks, and JSON detection files) — not just text or a single CSV — under a strict 24-hour wall-clock budget, mirroring real competition conditions.
3. Extensive agent evaluation. AIDE, ML-Master, and R&D-Agent are evaluated with GPT-5, Gemini 3 Pro, and Claude 4.5 Sonnet backends, with additional ablations extending the budget to 72 hours.
4. A capability-level failure analysis. Instead of only reporting scores, the authors audit each agent's execution traces against 13 "Winning Strategies" identified by Eisenmann et al. (2023) to explain why agents fail, using an automated LLM-as-a-judge pipeline.

Main Findings

• Severe performance gap. Across all 20 ReX-MLE challenges, agents failed to reach expert-level performance, with most submissions scoring in the 0th percentile relative to human competitors. Overall mean percentiles were 9.05% for AIDE, 4.53% for ML-Master, and 12.15% for R&D-Agent.
• Segmentation was the hardest category. In pathology segmentation (PUMA), AIDE failed to produce valid submissions, while ML-Master and R&D-Agent achieved Dice scores of zero, which the authors attribute to difficulty handling gigapixel whole-slide image data. In volumetric neurovascular tasks (TopBrain, TopCoW), most mean Dice scores remained below 0.3 while winners exceeded 0.85.
• Detection and classification were slightly better but still far off. On TopCoW-CTA-Det, R&D-Agent reached an IoU of 0.70 (56th percentile), one of the few non-zero-percentile outcomes. On TopCoW-CTA-Cls, AIDE reached accuracy of 0.33 against a human 0.73.
• Generative tasks performed poorly. On USenhance, agents scored around 0.11 against a human baseline of 0.91; the authors suggest agents treated the task as simple style transfer without accounting for ultrasound speckle noise physics.
• LDCT-IQA was an outlier success. R&D-Agent reached a score of 2.66 (50th percentile) against a winning score of 2.74. The authors caution this may reflect the metric's sensitivity to global image statistics, and note LDCT-IQA is the only challenge that does not require submitting separate prediction files beyond a submission.csv.
• Solution reports did not close the gap. On the medical subset of MLE-Bench, providing agents with the winning solution reports ("w/ sol."), excluding HCD, left the performance gap largely intact, suggesting the limitation is engineering execution rather than domain knowledge or hypothesis generation alone. ML-Master did surpass the human gold standard on HCD (AUROC 0.992 versus human 0.983).
• More time helped only marginally. Extending the budget from 24 to 72 hours yielded overall mean percentiles of 6.25% for AIDE, 14.5% for ML-Master, and 33.25% for R&D-Agent. The largest gain was R&D-Agent on LDCT-IQA at the 83rd percentile; complex 3D tasks such as ISLES'22 and DENTEX remained at the 0th percentile for most agents.
• Model backend was second-order. Swapping GPT-5 for Gemini or Claude changed absolute scores but not the overall pattern of failure. For example, Claude produced the highest ISLES'22 Dice score (0.65) yet still ranked in the 25th percentile; Gemini let AIDE reach the 8th percentile on DENTEX against a winning 0.40 AP.
• Capability analysis shows missing scientific practices. R&D-Agent showed the highest coverage of winning strategies (domain knowledge 0.663, reflecting metrics in method design 0.638, postprocessing results 0.614, analyzing and handling failure cases 0.601). AIDE and ML-Master were considerably lower. All three agents converged near or at zero on optimizing the augmentation method, ensembling, and leveraging external data.
• Inefficient compute use. On the 80GB NVIDIA H100 GPUs, agents typically used only 10–20% of available memory during training, and their inherently sequential workflows prevented them from running parallel experiments the way human practitioners do.
• A retrieval-relevance gap. The authors argue agents can find relevant methods but cannot judge whether they apply — for example, whether a task is truly 3D, whether the data support a given approach, or whether compute is sufficient. They cite nnU-Net, with over 5,000 citations, as a framework agents often overlook.

Methodology in Plain English

The authors built their benchmark by adapting challenges from the Grand Challenge platform, selecting tasks that had substantial community participation (typically over 200 registrants or extensive submissions), open licensing for research use, and publicly released evaluation metrics and scripts. For each challenge they prepared standardized materials following the MLE-bench framework: a concise competition description, automated dataset-preparation scripts, sample submissions, Python implementations of the official evaluation metrics for local validation, and a YAML metadata file with challenge identifiers, task types, data paths, and grading parameters.

Each agent received five standardized inputs per challenge: a competition description, the full training data with annotations, test data without ground truth, a sample submission showing the expected output format, and a local evaluation script. Agents worked with no human intervention on NVIDIA H100 GPUs (80GB memory), 64 CPU cores, and 128GB RAM, with a strict 24-hour wall-clock budget per challenge. GPT-5 was the primary model unless otherwise noted.

Performance was measured two ways. First, raw challenge-specific scores on each leaderboard's own metrics. Second, leaderboard-relative ranking: the authors collected the top 10 human competitor scores from each public test leaderboard, reconstructed positional rankings, averaged them into a mean ranking, and converted that to a percentile. An agent with mean ranking 1 of 10 competitors sits at the 100th percentile; a mean ranking of 11 corresponds to the 0th percentile. The overall ReX-MLE Rank is the mean of all competition percentiles, with failures assigned 0%.

To understand why agents failed, the authors examined full execution logs — reasoning traces, planning decisions, code generation, debugging attempts, and intermediate outputs — against 13 Winning Strategies from Eisenmann et al.'s meta-analysis of biomedical competition winners. Because there were 60 execution traces across 20 challenges, manual annotation was replaced with an automated pipeline using GPT-5 as a technical evaluator, assigning a binary score of 1 when explicit evidence of a strategy was present and 0 otherwise.

Why This Matters

Impact on research. The paper challenges the assumption that scaling general-purpose agents will naturally produce scientific competence. By showing that failures stem from missing domain-specific engineering practices rather than insufficient time or model scale, it argues that progress requires architectural innovation — domain-specific reasoning and competence self-awareness — not just bigger models.

Real-world applications:

• Clinical imaging pipelines. Segmentation, detection, and quality assessment tasks in CT, MRI, ultrasound, pathology, and microscopy are core to clinical workflows; the benchmark shows agents are not yet reliable enough to automate them.
• AI-assisted radiology and pathology. The near-zero Dice scores on volumetric and gigapixel tasks highlight concrete gaps in handling DICOM, NIfTI, voxel spacing normalization, and whole-slide image patching.
• Medical research acceleration. Automated end-to-end pipelines could speed up model development for neurovascular imaging, oncology, and dentistry if the engineering gaps were closed.
• Safety screening for high-stakes autonomy. The paper notes that agents cannot reliably recognize when they lack domain knowledge, producing "confident but meaningless submissions" — directly relevant to deployment in drug discovery, materials science, and genomics.

Industry relevance. Companies building ML-agent frameworks and medical AI products now have a standardized, leaderboard-grounded benchmark for measuring whether their agents work on real clinical data rather than simplified pipelines. For regulated medical AI, the finding that agents cannot self-assess competence boundaries is a direct input to validation and oversight requirements.

Future Directions

1. Domain-aware agent architectures. The authors call for mechanisms supporting domain-specific reasoning rather than retrieval pattern-matching, plus competence self-awareness so agents recognize when human expertise is required.
2. Better compute and workflow management. Agents used only 10–20% of H100 memory and ran strictly sequentially. Future work could target larger batch sizes, model ensembles, hyperparameter sweeps, and parallel experiment execution.
3. Deeper capability diagnostics. The authors note their automated LLM-as-a-judge assessment is scalable but lacks the depth of manual expert analysis; more thorough expert annotation could refine the failure taxonomy.
4. Broader expert-domain benchmarking. The paper suggests the failure modes — retrieval-relevance gaps, domain-specific engineering deficits, and inability to apply documented best practices — likely extend to other scientific domains beyond medical imaging. Given that ablations covered only 4 representative challenges due to computational constraints, expanding those studies is another open direction.

Target Audience

This paper is most useful to researchers and engineers building autonomous ML agents, benchmark designers evaluating agent capabilities, and medical AI practitioners interested in whether current agents can handle clinical imaging workflows. It also speaks to research groups considering automated scientific discovery pipelines, and to anyone assessing the reliability limits of LLM-based agents in high-stakes domains. Readers need only a general understanding of machine learning metrics and medical imaging task types.

Authors’ abstract

Autonomous coding agents built on large language models (LLMs) can now solve many general software and machine learning tasks, but they remain ineffective on complex, domain-specific scientific problems. Medical imaging is a particularly demanding domain, requiring long training cycles, high-dimensional data handling, and specialized preprocessing and validation pipelines, capabilities not fully measured in existing agent benchmarks. To address this gap, we introduce ReX-MLE, a benchmark of 20 challenges derived from high-impact medical imaging competitions spanning diverse modalities and task types. Unlike prior ML-agent benchmarks, ReX-MLE evaluates full end-to-end workflows, requiring agents to independently manage data preprocessing, model training, and submission under realistic compute and time constraints. Evaluating state-of-the-art agents (AIDE, ML-Master, R&D-Agent) with different LLM backends (GPT-5, Gemini, Claude), we observe a severe performance gap: most submissions rank in the 0th percentile compared to human experts. Failures stem from domain-knowledge and engineering limitations. ReX-MLE exposes these bottlenecks and provides a foundation for developing domain-aware autonomous AI systems.