The Pulse
Auto-Research Arena Finds Simple Search Ranks Second
In Auto-Research Arena’s Opus 5 results, GEAR produced the smallest model, while AutoScientists led overall and Sequential Search ranked second. With GPT-6 Astra, GEAR ranked first and Sequential Search ranked second.
AI.info Team · Written with AI · fact-checked ·
A smaller model in the Opus 5 tests
GEAR reduced a language model from 50.33 million parameters to 13.20 million in Auto-Research Arena, a benchmark designed to test whether AI research systems can make useful changes to model-training code. The result describes the complete program GEAR produced, not the effect of any single technique. The comparison puts seven search methods through nine efficiency objectives, including model size, training cost and inference latency.
The paper appeared on arXiv on October 26, 2026. The paper’s authors write:
Arena clearly differentiates existing harnesses across research objectives, with several sophisticated methods outperformed by our simple sequential search baseline, indicating substantial headroom in current auto-research systems.
Quality is a gate, not the score
Each search starts from a shared language-model training program and receives an allowance of up to 100 charged experiments per task. Rather than asking every method simply to reduce validation loss, Arena assigns one resource target per run. A proposed change counts only if it meets the benchmark’s quality and resource limits.
For the language-model tests, candidates had to keep validation bits per byte below 1.05, a threshold slightly above the starting model’s score. The nine targets cover total parameters; training tokens needed to reach the quality bar; training FLOPs, memory, time and updates; and inference measures for GPU key-value cache, decode latency and request latency. A system earns credit for efficiency only when the resulting model remains within the stated quality requirement.
Sequential Search ranks second
AutoScientists led the aggregate comparison with an Elo rating of 1729.4 in the Opus 5 results. Sequential Search ranked second at 1687.6, ahead of Arbor at 1632.4. The paper reports that with GPT-6 Astra, GEAR ranked first and Sequential Search ranked second.
That result complicates the idea that a more elaborate agent workflow will necessarily produce stronger research. The tested methods included multi-agent coordination, a hypothesis tree, evolutionary search and a diversity-preserving archive. Yet the simpler sequential method outscored several of those systems overall. No method recorded the best value on every objective: GEAR produced the smallest model in the Opus 5 comparison, while AutoScientists led on both latency measures.
Different goals produced different designs
The winning programs changed more than tuning settings. The paper describes discoveries including looped Transformers, grouped-query and multi-query attention, shared token representations, activation-memory changes and inference optimizations. In its parameter-efficiency discussion, the paper describes recurrent weight sharing that turns a model into a looped Transformer, along with depth-specific residual gains. The benchmark’s catalogue describes the full program, not a controlled test of each component.
Arena’s evaluation rules determine which results qualify. A candidate must meet the BPB quality threshold and its applicable resource limits; the benchmark scores progress on the selected objective among qualifying results. This means a lower resource figure alone does not count as a successful result if the candidate fails the quality requirement.
A language-model test, not a general proof
The study’s evidence comes from one instantiation of the benchmark in language modeling. Its results show how search methods fare under these objectives and constraints; they do not establish that the same rankings or discoveries will carry over to other research fields. The authors present Arena as a framework for later tests across objectives and research settings.
For now, the comparison is bounded by nine language-model efficiency tracks and a maximum allowance of 100 charged experiments per task. Within those limits, the central result is concrete: AutoScientists finished first overall in the Opus 5 comparison, while a simple sequential search finished second; with GPT-6 Astra, GEAR led and Sequential Search again ranked second.