The Pulse
Study Finds Strong AI Agents Need Less Harness Machinery
A study published by Apple Machine Learning Research finds that a single coding-agent session matched or beat four specialized harnesses on tested machine-learning engineering benchmarks. The authors caution that the results are limited by
AI.info Team · Written with AI ·
Across 17 comparisons with four established AI harnesses, a single coding-agent session matched or beat every system on the tested machine-learning engineering tasks. A paper posted September 30 by researchers whose work Apple published argues that, when a strong model can inspect files and run code itself, added layers of agent orchestration often bring little measurable benefit.
The result challenges a common design choice in autonomous machine-learning engineering: wrapping a model in specialized agents, search systems and memory tools to manage its work. The researchers do not argue that all scaffolding is useless. Their evidence concerns a defined set of benchmarks, models and compute budgets, and leaves open whether more elaborate coordination helps in settings they did not test.
A single coding session held its own
The study, “How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?”, compares a minimal setup called Malena with four open-source systems: MLEvolve, AiScientist, Arbor and ScienceFlow. Malena gives one coding agent a working directory and basic tools to read and write files, run commands and debug; it continues in a single session rather than handing work among a collection of specialized agents.
For the main comparison, the researchers held the model backbone, hardware and time budget constant. They tested 30 MLE-bench tasks with a 24-hour budget, reporting results across 17 combinations of harness and model. The authors write: “under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline, pointing to the backbone as the primary driver for performance.” Kirill Brilliantov et al., “How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?”
Across those comparisons, Malena matched or beat each tested harness on the frontier backbones. The paper notes a qualification at the smaller Gemma 4 31B backbone: MLEvolve had a higher mean estimate, although the medal-rate confidence intervals overlapped. The experiments used one Nvidia A100 with 80GB of memory, 12 CPU cores and 144GB of RAM.
Direct access mattered more than orchestration
The strongest shift came not from adding more agents, but from letting the model work inside a coding environment. The researchers compared that setup with a chat-based approach in which the model had to return a complete script in one response and could receive an error message if the script failed. Giving models direct access to the files and execution environment produced the largest improvement among the interventions they tested.
Other additions did not produce statistically significant gains over the minimal coding-agent setup in the study’s controlled ablations. Different search strategies showed no significant advantage over one another; adding subagents or parallel workers also failed to reliably improve the final, self-selected submission. Parallel agents sometimes found stronger candidates, but the system’s choice of which candidate to submit could erase that advantage.
That distinction matters because the benchmark rewards the solution an agent selects, not merely the best result it happens to generate along the way. The paper also finds that Malena’s long-running session can cost more in model inference than systems that reset into shorter threads, even though the study says hardware costs remain the larger expense overall.
The benchmark boundary matters
The researchers tested a second benchmark, NatureBench, which asks agents to work on scientific tasks tied to published research. On that evaluation, Malena matched or exceeded the compared systems for the two tested model backbones. The authors also acknowledge a possible mechanical advantage for Malena and AiScientist: NatureBench scores the best result achieved during a run, and those systems called the evaluator more often than MLEvolve.
Most of the paper’s results come from MLE-bench, a collection of public Kaggle competitions. The authors fixed task-preparation problems, checked generated code for similarity to known solutions and added NatureBench as a second test, but say public benchmark tasks remain a plausible target for training-data contamination. They also report only three to four seeds per task for the expensive 24-hour runs, limiting the precision of some comparisons.
The findings apply most directly to agents working on the benchmark tasks with modern coding tools and a single main worker. The authors say their study does not deeply test coordination among concurrent workers—a separate problem that may reward different design choices. For now, the paper’s clearest result is narrower: in its 30-task MLE-bench comparison, a single coding-agent session did not need the extra machinery of four specialized harnesses to keep pace.