InnoGym: Benchmarking the Innovation Potential of AI Agents

AI summary

Overview

Research area: Evaluation methodology for large language model (LLM) agents, sitting at the intersection of natural language processing, AI benchmarking, and automated scientific/engineering problem solving.

Technical level: Intermediate. The conceptual framing is accessible to anyone familiar with how ML benchmarks work, though the formal metrics (performance gain, novelty distance) and the agent-execution infrastructure assume some background in agent systems and evaluation design.

Scope: The paper introduces InnoGym — a benchmark (iBench) and execution environment (iGym) that score AI agents not just on whether they solve a task correctly, but on how much they improve over the best human solutions and how methodologically different their approach is.

What This Paper Is About

Existing benchmarks for AI agents almost universally measure one thing: correctness. An agent that reproduces a known solution by copying a textbook method scores identically to an agent that finds a genuinely new algorithmic approach. The authors argue this conflates two distinct abilities — being effective and being creative — and propose a framework that separates them. Their goal is to build the first standardized, reproducible testbed where an agent's innovation potential can be measured along two axes at once.

Key Contributions

1. A formal definition of innovation for AI agents. Each task is modeled as a quadruple (P, S, V, D) — problem instance, solution space, performance measure, and a dissimilarity function between solutions. Innovation is decomposed into two orthogonal metrics: Performance Gain (G), how far a solution exceeds the best-known human result, and Novelty (N), how methodologically distant it is from all known solutions.

2. iBench, a benchmark of 18 "Improvable Tasks." Curated from 197 candidate tasks drawn from NeurIPS Competitions, KDD Cup, ROADEF, GMCM, MLArchSys, and classic NP-hard problems. Tasks are standardized through resource-availability filtering, evaluator validation, solution collection, and data partitioning into visible/hidden splits.

3. iGym, a unified agent execution environment. An SDK providing an asynchronous tool dispatcher, recovery from failures across long-horizon runs, native concurrency, and a common abstraction layer so that different agent frameworks (MLAB, CodeAct, AIDE, etc.) can be compared fairly on identical infrastructure.

4. A systematic evaluation revealing a creativity–robustness gap. Experiments across three agent frameworks and three backbone LLMs show that agents can generate novel methods but routinely fail to convert that novelty into reliable performance.

Main Findings

• No agent surpassed human state of the art. Across all 10 tasks in the main evaluation, every agent framework averaged a negative performance gain: MLAB −24.32, CodeAct −41.58, AIDE −42.68. The best leaderboard human scores averaged 57.94 versus the agents' 23.79 or below.

• Two tasks produced zero valid submissions. On Cross-Domain-Meta-Learning (CDML) and the Perception-Test-Temporal-Action-Localisation-Challenge (PTTALC), all tested agents failed to produce executable solutions, indicating brittleness on tasks with intricate data formats or complex requirements.

• MLAB was the strongest overall. It led on both performance gain and novelty, suggesting a rare combination of execution quality and creative variation. CodeAct and AIDE underperformed on both axes, apparently due to weaker handling of complex file structures and tool use.

• Novelty does not imply effectiveness. On RCIC and TrojanDetection, frameworks with mid-to-high novelty scores still returned among the lowest performance scores. The authors conclude the primary bottleneck is not a shortage of novel ideas but the inability to implement them correctly and robustly.

• Base model strength dominates agent framework design. On the Circle Packing analysis task, Gemini-2.5-Pro reached 2.49 and a GPT-5 configuration reached 2.44, close to AlphaEvolve's 2.65, while DeepSeek-v3.1 reached 2.40. Agent scaffolds amplify a model's intrinsic reasoning ability rather than substituting for it.

• Iterative refinement shows diminishing returns. Over extended runs, performance gain tends to improve while novelty declines as solutions converge toward a local optimum. The authors visualize this as a trajectory in a complex plane where gain sets the vector magnitude and normalized novelty sets the angle.

• A temperature sweet spot exists. Low sampling temperatures maximize performance (exploitation), high temperatures maximize novelty (exploration), and the mid-range of 0.5–0.75 achieves near-optimal performance while meaningfully raising methodological novelty.

Methodology in Plain English

The researchers began by collecting 197 real tasks from competitions and classic hard problems where a clear human baseline exists and there is still room to improve. They filtered this down twice: first removing tasks that lacked accessible datasets, validators, leaderboards, or affordable compute; then removing tasks whose evaluators were broken or that duplicated other domains. Eighteen tasks survived.

For each surviving task, they rewrote the description in Markdown, packaged dependencies into containers, built validators to check that submissions are legal and feasible, collected known human solutions from leaderboards and papers, and normalized the scoring so results are comparable across tasks. Notably, relative or participant-dependent scores were converted into absolute instance-level scores while verifying that the conversion preserved the original rankings.

Agents are given only the visible part of a task (descriptions, examples, development data) plus the validator. The hidden part — evaluation data, the evaluator, and the set of known solutions — stays out of reach. After an agent submits a solution, the pipeline runs three steps: check feasibility, score performance, then measure novelty. Novelty is computed by having Codex extract a structured profile of the agent's strategy, then asking GPT-5 to rate how different that strategy is from each known reference solution along six rubric dimensions on a 0–4 scale. The minimum distance across all known solutions, rescaled to 0–100, becomes the novelty score.

Experiments ran three agent frameworks on 10 tractable tasks, with DeepSeek-v3.1 as the backbone, allowing up to 12 hours per run and three repetitions per configuration, reporting the best valid submission.

Why This Matters

Impact on research. Correctness-only benchmarks create a blind spot: they reward agents for rediscovering textbook methods and offer no signal about whether an agent can produce a conceptually new approach. InnoGym supplies the missing measurement axis and a reproducible platform for tracking progress on it. It also reframes what a "good" result looks like — a novel solution that fails to run is now explicitly categorized as unsuccessful exploration rather than success.

Real-world applications:

• Algorithm and optimization competitions. As AI agents enter ROADEF-style challenges and operations research contests, organizers need scoring that distinguishes a novel heuristic from a well-tuned conventional one.

• Drug discovery and materials science. Tasks like the BELKA molecular prediction challenge benefit from evaluation that rewards genuinely different modeling strategies, not just higher AUC.

• Hardware and systems design. With MLArchSys-derived tasks in the benchmark, the framework applies to problems where the design approach matters as much as the final metric.

• Scientific paper review and ideation. The novelty metric offers a possible mechanism for assessing whether proposed research methods are genuinely distinct from prior work.

Industry relevance. Companies deploying coding agents and ML-engineering agents need to know whether their systems can handle unfamiliar, high-stakes problems or merely replay known recipes. The finding that base model quality dominates scaffold design is directly actionable for teams deciding where to invest. iGym's recovery and concurrency features also address practical pain points in long-running agent deployments, where a crash at hour ten currently wastes the entire run.

Future Directions

• Extending to solved and exploratory problems. The benchmark currently excludes tasks with known optima (like MATH or SWE-Bench) and open exploratory problems with no human baseline. Adding them — especially exploratory tasks where the first feasible solution counts as a major innovation — would broaden applicability.

• Capturing innovation dimensions beyond performance and novelty. The authors acknowledge that efficiency, interpretability, and long-term scientific impact are not measured. Multi-dimensional innovation metrics could close this gap.

• Improving novelty estimation under sparse prior knowledge. Novelty is computed relative to a finite set of known solutions, which biases results when that set is incomplete. Better methods for estimating the true diversity of the known-solution space would strengthen the metric.

• Scaling to compute-heavy tasks and closing the robustness gap. Computational limits currently exclude very large tasks. Separately, the central empirical finding — that agents produce novel ideas they cannot reliably implement — points toward research on verification, debugging, and self-correction as the highest-leverage direction.

Target Audience

AI evaluation researchers and benchmark designers will find the formal framework and task taxonomy most useful. Agent developers building long-horizon systems benefit from the iGym infrastructure and the empirical comparison of MLAB, CodeAct, and AIDE. Competition organizers and industrial R&D teams evaluating whether agents can contribute to genuinely open problems should read the main results and the Circle Packing analysis. Graduate students entering the agent-evaluation space will find the paper a well-structured entry point, though the appendices hold the implementation details needed to actually use the benchmark.

Authors’ abstract

LLMs and Agents have achieved impressive progress in code generation, mathematical reasoning, and scientific discovery. However, existing benchmarks primarily measure correctness, overlooking the diversity of methods behind solutions. True innovation depends not only on producing correct answers but also on the originality of the approach. We present InnoGym, the first benchmark and framework designed to systematically evaluate the innovation potential of AI agents. InnoGym introduces two complementary metrics: performance gain, which measures improvement over the best-known solutions, and novelty, which captures methodological differences from prior approaches. The benchmark includes 18 carefully curated tasks from real-world engineering and scientific domains, each standardized through resource filtering, evaluator validation, and solution collection. In addition, we provide iGym, a unified execution environment for reproducible and long-horizon evaluations. Extensive experiments show that while some agents produce novel approaches, their lack of robustness limits performance gains. These results highlight a key gap between creativity and effectiveness, underscoring the need for benchmarks that evaluate both.