Adversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities

AI summary

Overview

Research area: Evaluation and benchmarking of AI scientist systems (large language model agents that generate scientific or engineering hypotheses), using real-world competitive domains with time-delayed ground truth.

Technical level: Intermediate. The framework and results are readable without specialist knowledge, though the statistical tests and the domain-specific mechanics (FIA technical regulations, competitive card-game deck construction) add complexity.

Scope: The paper proposes and instantiates a proof-of-concept benchmark in which frontier LLMs must generate candidate ideas before a fixed information cutoff, which are then compared against innovations independently produced by human experts afterward, in Formula 1 2026 car design and Magic: The Gathering deck construction.

What This Paper Is About

Benchmarking whether AI systems can generate genuinely novel scientific ideas is hard, because existing benchmarks rely on synthetic tasks or on targets that already predate model training data, so results can be confounded by prior exposure. The authors propose using adversarial, fast-moving real-world domains, where competing experts independently produce publicly observable outputs after a known cutoff, as a test bed for AI scientist capabilities. They demonstrate the idea in two structurally different domains and compare model outputs against real expert artifacts revealed after the models' knowledge cutoffs.

Key Contributions

1. A portable evaluation framework for AI scientist capabilities based on time-delayed generation: a fixed pre-cutoff input corpus, model-generated structured candidate artifacts, and post-cutoff expert artifacts as ground truth. The paper lists four suitability criteria for a domain: a constrained but non-trivial design space, adversarial expert innovation, publicly accessible observable downstream artifacts, and a clear information cutoff.

2. Two domain instantiations with different structure: Formula 1 car design under the 2026 FIA technical regulations (free-form ideation evaluated against 40 real pre-season innovations) and Magic: The Gathering's Lorwyn Eclipsed set (discrete deck construction against 19 Pro Tour decklists).

3. A three-agent pipeline that separates strategic intent, technical analysis, and generation, run in two configurations per domain (general vs. component-focused in F1; one-shot vs. tool-use in MTG) against six frontier LLMs, with deterministic and human-reviewed evaluation metrics.

4. Identification of specific failure modes for AI scientists, including plausible-but-misaligned ideas, systematic blind spots for value arising from surrounding context, and the finding that the key gap is not idea generation but filtering, prioritization, and coherent novelty.

Main Findings

• Models generate plausible output but rarely match experts. Across both domains models produced plausible candidates, but few aligned with real-world expert solutions. In F1, the six models proposed 881 candidate technical innovations, and 19 of 40 (48%) real 2026 pre-season innovations were independently suggested by at least one model.

• Strong model-to-model variation in F1. GPT-5.2 matched 10 of 40 real innovations with 166 ideas proposed across runs, while Qwen3 235B thinking partially matched 3 real-world innovations. Across the six models, idea count was not significantly associated with the number of distinct matches (Spearman rho = 0.33, p = 0.52, n = 6); GPT-5.2 ranked highest on the volume-normalized measure at 6.0 matches per 100 ideas.

• Quality signals predict real-world matches. Rule compliance carried the strongest signal: legally compliant or grey-zone ideas (n = 542) had 9 times higher odds of matching a real innovation than non-compliant ideas (n = 339; OR = 8.92, 95% CI [2.74, 29.08], p = 2.9 × 10^-6). Citation accuracy (OR = 3.11, 95% CI [1.30, 7.46], p = 0.007) and engineering plausibility (OR = 4.65, 95% CI [1.11, 19.44], p = 0.022) showed weaker but significant associations. Pooled, ideas rated PASS on rule compliance matched at 19.4% (14 of 72), GREY at 5.5% (26 of 470), and FAIL at 0.9% (3 of 339).

• Run-to-run output is noisy. Match counts varied across independent runs: the mean within-model standard deviation was 0.78, and five of six models had at least one run with zero matches. GPT-5.2 produced the most consistent output (cross-run best-match cosine similarity mu = 0.70), while o3 explored more widely (mu = 0.62).

• MTG: partial recovery of expert card choices. Across six models and 108 generated decks, 14 of 19 (74%) PT new-set cards were selected. The strongest single deck-pair match was Gemini 3 Flash in one-shot mode, recovering 5 of 7 new-set cards (71%) from the third-place PT deck, against a per-pair expectation of 0.32 cards (15.5 times). That same configuration was the only one whose nine-deck total exceeded the upper quartile of its random-baseline band in both configurations (8 of 19 one-shot, 1.34 times expected; 3 of 19 tool-use, 2.13 times expected).

• Models converge on widely adopted cards. Across the 19 new-set PT cards, the number of generated decks playing a card correlated strongly with the number of PT decks containing it (Spearman rho = 0.74, p = 0.0003, n = 19; Kendall tau = 0.62, p = 0.0008).

• Generic effects over-predicted, build-arounds missed. Spell Snare appeared in 33 of 108 generated decks but only 9 of 19 PT decks, and Sear in 14 generated decks against 4 PT decks. Conversely, Formidable Speaker appeared in 7 of 19 PT decks but in zero generated decks.

• Scaffolding has a trade-off. Tool-use significantly reduced new-set discovery for every model (paired Wilcoxon p = 0.031; 45/54 [83%] one-shot decks beat random versus 15/54 [28%] for tool-use decks), while substantially increasing main deck validity for GPT-5.2 (11% to 100%) and Qwen3 (0% to 89%). No statistical differences were observed across all models on legality (Friedman p = 0.24).

• Loophole type matters more than frequency. INTERFACE_COUPLING was the most frequently flagged category by agent 2 (378 of 1311 mappings, 28.8%) but matched real innovations at only 5.0%, whereas BROAD_FREEDOM was less frequently flagged (20.7%, 271 mappings) but produced the highest match rate (7.7%). Overall mapping match rate was 5.4%.

• Evaluation pipeline itself is noisy. Of 519 pairs flagged by the LLM novelty judge for human review, 89% were downgraded to NO_MATCH. Cosine similarity between generated and real mechanisms did not discriminate matched (n = 55, mean = 0.628) from unmatched (n = 464, mean = 0.621) pairs (Mann-Whitney one-sided p = 0.22).

• Coverage was uneven across car areas. F1 recall was highest in cooling (4/5) and bodywork (6/9), with suspension also strong (3 of 4, 75%), while front wing (0 of 3) and electronics (0 of 1) were entirely uncovered. The two pipeline configurations were complementary: three innovations were found only via general runs and eight only via component-focused runs.

• Overall capability gap is filtering, not generation. In MTG, aggregate discovery within any single configuration only modestly exceeded its random baseline, and 60 of 108 generated decks (56%) achieved a best-match new-set-card recall above the per-pair random expectation. The authors conclude the gap lies in over-emission and selective omission rather than absent understanding.

Methodology in Plain English

The authors pick domains where the future is genuinely unknown at a fixed point in time. They assemble a fixed input corpus containing only material available before that cutoff, ask an AI system to produce structured candidate artifacts, then compare those artifacts against what human experts actually did afterward. Because every model's reported knowledge cutoff predates the release of both corpora, the authors argue there is no risk of information leakage.

For F1, the corpus was a 264-page PDF of the FIA 2026 Section C Technical Regulations dated 10 December 2025, and the task was to produce technical innovations specific to the 2026 cars. The ground truth is a curated set of 40 real F1 concepts drawn from publicly available pre-season analysis published between 28 January and 23 February 2026.

For MTG, the corpus combined the 267 cards of the Lorwyn Eclipsed set (released 23 January 2026) with 4,168 existing Standard-legal cards, and the task was to produce a tournament-legal deck prioritizing the most viable new cards. Ground truth is 19 decklists: the Top 15 finishers by Day 2 standings at Pro Tour Lorwyn Eclipsed (held 30 January to 1 February 2026) plus four featured builds.

Both domains used a three-agent pipeline. For F1, agent 1 turns a 2026-context summary into performance goals; agent 2 maps the regulations into degrees of freedom tagged with one of five loophole categories (INTERFACE_COUPLING, DEFINITION_EDGE_CASE, BROAD_FREEDOM, NOT_PROHIBITED, EXCEPTION_ZONE) plus cited article numbers; agent 3 receives the prior outputs, the regulatory text for cited articles, and historical few-shot examples, and produces roughly 15 car design ideas. For MTG, agent 1 labels each new card as STAPLE, ROLE_PLAYER, BUILD_AROUND, or UNPLAYABLE; agent 2 produces a combined shortlist with strategy notes using pre-tournament metagame context; agent 3 generates three decks, each for a different strategic angle.

Six frontier LLMs were tested: GPT-5.2 (August 2025 cutoff), o3 (June 2024), Gemini 3 Flash and Gemini 3.1 Pro (both January 2025), and Qwen3 235B and Qwen3 235B (thinking) (both June 2025). Each model ran in two configurations per domain: general (whole-car) versus by-component in F1, and one-shot versus tool-use in MTG. For MTG, four model names map to 108 total decks, i.e. 9 generated decks per model per configuration.

Evaluation ran along two axes: outcome similarity (does the generated artifact correspond to an independent expert innovation under the same constraints?) and intrinsic quality (is it well-grounded, rule-compliant, and plausible?). F1 novelty matching used a dual top-k (k = 3) embedding shortlist, an LLM judge (GPT-5.5) assigning MATCH, PARTIAL, or NO_MATCH, and human review by a single author adjudicator (who is not a domain expert in either field) to verify whether the same physical effect was described. Quality was scored on citation accuracy (PASS/PARTIAL/FAIL), rule compliance (PASS/GREY/FAIL), and engineering plausibility (PASS/FAIL). MTG evaluation was deterministic: Jaccard, precision, and recall over non-land cards, plus separate analysis of new non-land Lorwyn Eclipsed cards. Two random baselines were computed over the 261 non-land new-set card pool: a per-pair hypergeometric expectation (kK/N), and an aggregate expectation across the d = 9 decks per configuration using T = 19 distinct PT new-set cards, computed analytically and via 10,000 Monte-Carlo resamples.

Statistical tests included Fisher's exact tests on 2x2 contingency tables pooled across all six models (n = 881 ideas), paired Wilcoxon signed-rank and Friedman tests for MTG configuration comparisons, and Spearman's rho and Kendall's tau for card-level rank correspondence. The authors describe these as exploratory associations rather than model-level causal comparisons, because outputs are clustered by model, run, and pipeline configuration.

Why This Matters

Impact on research. The paper argues that benchmarking AI scientists and verifying their discoveries will become the bottleneck to safely deploying autonomous research systems. It offers a practical intermediate test bed between decomposed synthetic benchmarks (DiscoveryBench, HypoBench, PaperBench, ScienceAgentBench) and full prospective evaluation, and it suggests that open-ended ideation benchmarks should explicitly measure filtering, prioritization, and meta-alignment with experts rather than rewarding fluency and volume. It also notes that historical replay suffers from unclear knowledge cutoffs, expensive experiments, and ambiguous negative results, whereas prospective time-delayed evaluation avoids many of these problems.

Real-world applications (from the paper's own framing):

• Pre-registering experimental designs before a study runs.
• Shortlisting candidate targets before a drug-discovery campaign.
• Selecting proof-search strategies before a conjecture is formally settled.
• Other adversarial domains with periodic reveal events, including legal and financial regulatory cycles.

Industry relevance. The failure modes identified — over-predicting generically attractive effects, missing value that only emerges from composition with surrounding context, and a scaffolding-exploration trade-off where more structure improves legality but reduces discovery — are directly relevant to teams building agentic research or design systems. The authors suggest future systems may benefit from being coupled to simulation, CFD, or laboratory automation, since several F1 failure modes appear to reflect the absence of physical-validation tools. The work was funded by ARIA, DSIT and Pillar VC under the Encode: AI for Science Fellowship, and code, data, prompts, and generated outputs are released in two GitHub repositories for the F1 and MTG domains.

Future Directions

• Pre-register the evaluation. The domains were designed to be run prospectively, but the human artifacts in this study had already been generated at the time of evaluation. Future instantiations should pre-register prompts, corpus, models, and runs before expert artifacts are released.

• Move closer to real science. The paper suggests benchmarks that preserve complex design spaces, independent expert search, and delayed verifiable outcomes while incorporating richer process data: what information was available at the cutoff, what candidate ideas were considered, why some were discarded, and what evidence later confirmed or falsified them.

• Solve cross-domain aggregation. How to combine metrics from heterogeneous domains into a single interpretable score remains an open question.

• Test whether scaffolding-exploration trade-offs generalize. The paper found no consistent differentiation between reasoning and non-reasoning models in this work, notes that only a small set of models was used, and states this requires further investigation. Coupling systems to simulation, CFD, or laboratory automation is offered as a way to address failure modes that stem from lacking physical validation.

Target Audience

Researchers and engineers working on AI scientist systems, agentic research pipelines, and LLM evaluation methodology; benchmark designers interested in contamination-resistant, time-delayed evaluation; and domain practitioners in motorsport engineering or competitive card games who want to understand where current frontier models converge with, or miss, expert judgment. The paper is also relevant to science-funding and research-policy audiences interested in how autonomous research systems might be validated before deployment in higher-stakes settings. Readers should note the authors' caveats: a match shows convergence with a human idea under shared constraints, not that the model reasoned the same way or that the idea would provide genuine value; expert artifacts are an incomplete ground truth; and the domains studied are proxies for scientific discovery and should be complemented by, not substituted for, expert judgment and external validation.

Authors’ abstract

Benchmarking the ability of AI scientists to generate novel ideas is notoriously difficult. Existing benchmarks in this field have made progress in evaluating scientific reasoning and research replication, but often rely on synthetic tasks or retrospective targets, which may be confounded by prior exposure. We hypothesize that complex, adversarial, fast-moving real-world domains where expert practitioners independently generate observable outputs can provide a practical solution to fill this gap and evaluate the capabilities needed for AI scientists, including reasoning, novelty, and hypothesis formulation. We instantiate this framework in two structurally different domains, Formula 1 (F1), where models ideate around car design concepts for the 2026 season, and real pre-season innovations provide a ground truth, and Magic: The Gathering (MTG), where models propose decks from a recently updated card pool and are evaluated against 19 Pro Tour (PT) decklists. Across both domains, models produce plausible outputs, but few align with real-world expert solutions. In F1, the best model, GPT-5.2 matched 10 of 40 real innovations with 166 ideas proposed across runs. In MTG, the best deck from Gemini 3 Flash recovered 5 of 7 new-set cards from the third-place PT deck, and across all 108 decks, the cards models selected most often were also the cards most widely adopted by PT decks (Spearman $ρ= 0.74$, $p = 0.0003$). These results suggest that a key capability gap for AI scientists is not idea generation, but filtering, prioritization, and coherent novelty.