WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents

AI summary

Overview

Research area: Evaluation of AI research agents, specifically benchmarking whether agents that run experiments actually learn the quantitative effects of the changes they make. It sits at the intersection of agent benchmarking, adaptive experimental design, response-surface reconstruction, and hyperparameter/component analysis.

Technical level: Advanced. The paper assumes familiarity with factorial designs, ridge regression, Gaussian processes, conditional effects and interactions on a binary cube, response surfaces, and acquisition functions.

Scope: A single-sentence scope: the paper defines and builds an executable benchmark that grades an agent's submitted prediction table for every binary configuration of a computational workflow, using exhaustive CPU execution as ground truth.

What This Paper Is About

AI agents increasingly plan, run, and interpret computational experiments, and the paper argues that a system can pick a good configuration without actually understanding how each component changes the outcome. WhatWorkedBench measures that gap directly: an agent inspects workflow code, buys a limited number of measurements under a budget, and must submit a complete predicted score table (a response surface) for every configuration of component settings.

The benchmark then compares that table against exhaustive native execution, so it can score the accuracy of every conditional component effect, every mean pair interaction, the configuration chosen, and whether the artifact was delivered at all. The goal is a reproducible target on which measurement policies, numerical estimators, and uses of program structure can be compared on identical evidence.

Key Contributions

1. An executable evaluation of experimental understanding. The target is a complete field of conditional component effects rather than a single preferred configuration, so it covers arbitrary combinations of each task's documented changes, alongside configuration choice and final artifact delivery. The reference comes from exhaustive native execution of each workflow.

2. A catalog of 36 task conditions on 30 sources across 8 computational workflow families. Exhaustive native execution yields 1,248 configuration records, 3,392 conditional effects, and 342 mean pair interactions, with standalone execution and grading.

3. Controlled comparisons of three algorithmic decisions. Measurement selection, numerical inference, and use of program structure are separated using 4,206 numerical-control records, 108 core agent episodes, eight structure-diagnostic episodes, and six completed submissions from added workflow families. The results separate optimization success from intervention knowledge.

4. Native-verified code equivalences. The benchmark encodes rules from visible workflow code (for example, a parameter becoming inactive when its operation is disabled) as equivalence classes, and checks every claimed equivalence against native prediction hashes and scores.

Main Findings

• Optimization success and intervention knowledge are distinct. On the 22 four-factor sources, pair-effect design/ridge at B = 8 achieves 0.612 family-macro recovery, 15 exact configuration choices, and three strict reconstructions. Thirteen instances combine an exact configuration choice with an effect error above the strict tolerance.

• Estimators trade off mean recovery against worst-case error. The effect-variance GP reaches 0.701 recovery, 16 exact choices, and one strict reconstruction, so the GP ranks higher on mean recovery and configuration choice while pair ridge ranks higher on the maximum-error criterion. Under maximum-edge tolerances of 2.5%, 5%, 10%, 20%, and 40% of response range, pair ridge passes on 1, 1, 3, 8, and 20 of 22 instances. Among its 15 exact-choice cases, mean continuous recovery is 0.634 with minimum 0.361.

• Conditional sign reversals are pervasive in the catalog. At a numerical floor of 10⁻¹², 35 of 36 task conditions contain a sign reversal. Requiring both positive and negative effects to exceed 1%, 5%, or 10% of the task response range gives 33, 24, and 16 conditions respectively. The paper's Figure 2 example: binary term counts raise SciFact NDCG@10 by 0.160 with sublinear TF off and lower it by 0.084 with TF on, a −0.245 interaction, with both right-column cells scoring 0.544.

• Shared inference extracts extra value from the same observations. Fitting a Gaussian process to the identical agent observations raises recovery from 0.632 to 0.698 in the original Flash cohort and from 0.621 to 0.720 in the additional cohort. Shared GP recovery is 0.698 and 0.720 on the two Flash cohorts compared with 0.669 on D2-design observations and 0.640 with GP's own effect-variance acquisition; the original-cohort GP gain stays positive after omitting each source in turn, ranging from 0.021 to 0.096.

• Agent recovery depends heavily on budget. Original Flash rises from 0.178 at B = 4 to 0.632 at B = 8; the additional Flash cohort rises from 0.375 to 0.621. Pro rises from 0.043 to 0.366 in the original cohort and from 0.308 to 0.503 in the additional cohort. Across all 18 Flash B = 8 episodes, seven select an exact optimum and one passes strict reconstruction.

• Much of the prediction task is extrapolation. Across the original Flash B = 8 trials, 67.4% of conditional effects involve an unmeasured configuration. Filling unmeasured utilities with the mean of the same observations gives recovery 0.287 versus the delivered 0.632.

• Code equivalences raise recovery. Across six paired workflows at B = 20, encoding equivalence classes raises pair-ridge recovery from 0.157 to 0.389 and effect-GP recovery from 0.248 to 0.462. At B = 32, the gains are 0.445 to 0.650 and 0.537 to 0.697. All four sources with nontrivial classes improve; the two identity controls retain their predictions.

• Agents can state program rules but do not apply them. In eight B = 20 episodes on four sources, the four code-only agents report all six registered rules before measurement, and the four rules-supplied agents also report them, yet all eight final tables use public ridge helpers and violate the equivalences. Supplied rules change mean recovery from 0.247 to 0.429, and an offline post-hoc projection raises recovery from 0.338 to 0.507 while lowering raw effect error in all eight runs.

• More code information did not help in these cohorts. Full-code and opaque agent views yield recovery 0.268 and 0.257 in the original cohort, then 0.185 and 0.234 in the additional cohort; shared GP favors opaque observations in both cohorts, with full-minus-opaque differences of −0.024 and −0.064.

• The shared-inference pattern extends to added families. On six completed B = 32 Flash submissions covering beat detection and graph link prediction, three sources per family, shared GP raises family-macro effect recovery from 0.303 to 0.455, with positive gains on all six sources. Same-source numerical controls reach 0.230 for D2, 0.414 for grid GP, and 0.404 for effect GP.

Methodology in Plain English

Each task is a small computational workflow with either four or six binary options, called factors. Together the options form a mask, and every mask is legal, so the full space has 16 or 64 configurations. Native code executed on fixed data, seeds, and evaluation cohorts produces a deterministic score in [0, 1] for each mask.

The agent gets two free anchor measurements, the all-zero and all-one masks, and can purchase at most B additional distinct measurements, adaptively or in batches. At the end it must submit a predicted score for every legal mask. Grading then inserts the authoritative measured values at the cells the agent actually measured, derives predicted conditional effects from the single submitted table, and compares them with the true effects.

The primary number is the error of each component's effect in each background, where a background is a fixed setting of all other components. Because effects along a switching path determine the whole surface, a table with small effect error automatically yields small error for any multi-component intervention. The paper reports raw effect MAE, a relative recovery score, pair-interaction MAE, grid MAE, selection regret, and a strict reconstruction score that requires every conditional-effect error to fall within a tolerance of the response range, with the development tolerance set to 0.1 and a five-tolerance profile also evaluated.

Reference reconstruction methods include main-effect ridge (D1), pair-effect ridge (D2), sequential ridge designs, a random design with 20 seeds, a Gaussian process over the number of differing options, and grid GP and effect-variance GP acquisition. Code-equivalence controls measure one representative per equivalence class and assign that prediction to every class member. Agent studies use the identifiers deepseek-v4-flash and deepseek-v4-pro, abbreviated Flash and Pro, with reasoning enabled, a 32,768-token output cap, a disclosed 1,800-second session deadline, and a maximum of 20 calculation calls at 30 CPU seconds and 45 wall seconds each. Averaging is family-macro: repetitions within source, sources within family, then equal family weights.

Why This Matters

Impact on research. The paper provides a reusable, executable target for a question that agent benchmarks usually handle informally: did the agent actually learn how the components interact, or did it just land on a good setting? Because references are exhaustive and deterministic, and because observations can be replayed identically, measurement policies and estimators can be compared on the same evidence instead of on different experiments. The reported separation between exact configuration choice and effect reconstruction gives the field a concrete failure mode to measure.

Real-world applications (areas the benchmark's workflow families touch):

• Machine-learning pipeline tuning, where classification, regression, and clustering workflows are scored with balanced accuracy, transformed MAE, and normalized mutual information.
• Time-series and forecasting pipelines, using one-step ridge forecasts on sources such as CO₂, El Niño, Nile, and sunspots.
• Image restoration pipelines, using native image filters on sources such as astronaut, coffee, hubble deep field, and rocket.
• Ranked retrieval pipelines, using TF-IDF retrieval scored by NDCG@10 on ArguAna and SciFact (8,567 and 5,183 documents, 64 queries each).
• Signal processing, where beat detection uses SciPy signal processing with event F1 on archived MIT-BIH waveforms, and graph link prediction uses NetworkX features with logistic regression scored by ROC AUC.

Industry relevance. Teams that build automated experimentation, AutoML, or research-agent tooling face the same question the benchmark poses: whether an agent's final artifact carries transferable knowledge or only a lucky pick. Because the benchmark separates acquisition from inference, it can be used to decide whether to invest in better experiment selection, better numerical reconstruction, or better use of program structure. The finding that agents already state code rules but then violate them in their deliverables points

Authors’ abstract

AI research agents need reliable knowledge of how their experiments change outcomes. We introduce WhatWorkedBench to measure experimental understanding, the accuracy of predictions about component changes after budgeted experimentation. Agents inspect code, select measurements, and submit a response surface, a table predicting scores for every configuration of component settings. Exhaustive CPU execution supplies reference effects for changing each component while holding the others fixed. These effects capture combinations of changes across 36 tasks from 30 data sources and 8 workflow types, with 1248 configuration records. Core evaluation combines 4,206 numerical-control records across all eight families and 108 agent episodes across the original six. At eight new measurements, pair-effect ridge selects an optimum on 15 of 22 sources and limits every effect error to 10% of score range on three. Fitting a Gaussian process (GP) to the same agent observations raises effect recovery, accuracy relative to true effect magnitude, from 0.632 to 0.698 in the original Flash cohort and from 0.621 to 0.720 in an additional cohort. On six completed beat-detection and graph submissions, the same-observation GP raises family-macro recovery from 0.303 to 0.455. On six workflows with six binary options at 20 new measurements, encoding code equivalences, configurations with identical behavior, raises GP recovery from 0.248 to 0.462. WhatWorkedBench supports research on experimental agents, adaptive experimental design, numerical inference, and use of program structure.