The Pulse
In GLM 5.2 MLE-bench Test, Single Agent Posts 62.5% Medal Rate
In the paper’s GLM 5.2 MLE-bench comparison, the Malena coding agent earned medals on 62.5% of competitions, compared with 47.1% for AiScientist. The study also reports that the agent matched or beat tested harnesses across its frontier-mod
AI.info Team · Written with AI ·
On MLE-bench, a single coding-agent session earned medals on 62.5% of competitions with the GLM 5.2 model, compared with 47.1% for AiScientist, the best-performing external harness in that comparison. The paper, posted to arXiv on September 30 and featured by Apple Machine Learning Research, argues that strong models with direct access to files and code execution can match or beat systems built around search trees, specialist agents and other layers of orchestration.
The agent, called Malena, works in one long-running session with a filesystem and shell. The researchers compared it against four open-source systems—MLEvolve, AiScientist, Arbor and ScienceFlow—while holding the models, compute resources and time budgets constant. The result challenges the assumption that a more elaborate software wrapper necessarily makes an autonomous machine-learning engineer more effective.
What Malena did with a shell
The study separates access to an execution environment from the orchestration layered around it. In an initial comparison, a model using a chat interface had to return a complete solution script as text; a coding agent could instead inspect files, write code, run it and debug errors. Both approaches had six hours to produce a submission, and the researchers found that direct access to the environment produced the largest improvement they measured.
They then tested added search methods and autonomy. Across the tested search strategies, paired comparisons found no statistically significant advantage in percentile or medal rate. Malena went further: rather than ask a harness to select and schedule successive attempts, it remained in one session for the full time budget and decided what to test and refine.
“In this paper we find that, under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline, pointing to the backbone as the primary driver for performance.”
Kirill Brilliantov, Alejandro Hernández-Cano and Emmanuel Abbé, the paper’s authors
Four systems, matched conditions
For the external comparison, the researchers ran Malena against MLEvolve, AiScientist, Arbor and ScienceFlow. On MLE-bench, the 62.5% figure for Malena and 47.1% for AiScientist are medal rates in the GLM 5.2 results. The authors report that, with frontier models, none of the tested systems significantly outperformed Malena; they also tested six harness-model pairings on 40 NatureBench tasks, a separate set of open scientific-research problems.
The authors’ trace analysis offers a possible explanation. Malena explored approaches and reused code and other artifacts from its earlier work, then shifted toward performance-focused refinements such as ensembling and model selection as runs progressed. In effect, the agent performed parts of the search that other systems encode in their harnesses.
The result is conditional, not a finding that orchestration never helps. The paper says weaker models can still benefit from hand-designed workflow guidance, and its comparison does not establish that every agent task or deployment setting favors a minimal harness.
Benchmark results come with limits
Most of the experiments rely on MLE-bench, a collection based on public Kaggle competitions. The authors identify possible training-data contamination and task-preparation problems as concerns; they report fixing affected tasks, checking for contamination and adding NatureBench as a second evaluation. Those steps address some concerns, but the paper’s conclusions remain bounded by the tasks and systems tested.
The authors also caution that some confidence intervals are too wide to support precise claims about the size of performance differences. Their multi-agent tests cover a limited set of coordination methods, leaving more sophisticated ways of directing agents for future investigation. And performance is not the only trade-off: at GLM 5.2, the paper estimates Malena’s inference cost at $12.12 versus $1.95 for AiScientist, attributing the gap largely to the growing context in Malena’s long-running session; the authors say hardware dominates total cost.
Whether more advanced coordination can improve on that result remains untested.