Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts

AI summary

Overview

Research area: Evaluation methodology for AI research agents and scientific forecasting — specifically, how to measure whether an agent's written explanation of a planned experiment actually improves predictions of the experimental outcome.

Technical level: Advanced. The paper assumes familiarity with proper scoring rules (interval scores), paired experimental contrasts, bootstrap inference, equivalence margins, and intention-to-treat analysis.

Scope: The paper introduces "predictive credit" as a paired-forecast measurement of what a scientific explanation adds to forecasting the result of a fixed intervention, and reports three prospective studies (120, 72, and 144 states) plus post-outcome replays and controls totalling 3,708 prospective slots.

What This Paper Is About

Automated research systems produce both an experimental result and a written account of why a proposed change should work, but final performance only evaluates the result. This paper asks a narrower, testable question: does supplying that explanation help a fixed forecaster predict the executed outcome of a fixed intervention, compared with supplying only the public state and the planned intervention? To answer it, the authors build paired forecasts that share an intervention, a forecaster, and an outcome while varying only the description, a matched explanation, and a donor explanation from another state.

Key Contributions

1. A paired-contrast design for explanations. Three contexts are defined over the same state: description D = (x, a) (public state plus planned intervention), matched M = (x, a, h_i) (the current state's explanation), and donor S = (x, a, h_π(i)) (another state's explanation under a frozen mapping). Two contrasts, G_description = R_D − R_M and G_alignment = R_S − R_M, define predictive credit, and positive values favor the matched context.

2. Five checks that separate distinct observable behaviors. Prospective commitment, evidence-backed delivery, paired predictive value against description and simple priors, donor alignment, and sensitivity to known signals. These are designed to distinguish a claim being made explicit, repeated forecasts converging, accuracy improving, and uncertainty intervals widening — which the paper notes can occur independently.

3. Three prospective studies plus post-outcome controls. Controlled v5 (120 states, 900 pre-outcome units), the Tox21 anchor (12 endpoints, 72 states, 504 units), and OpenML v2.1 (24 tasks, 144 states, 2,304 units), followed by an assay calibration (1,440 calls), a Flash replay (432 calls), a component crossover (1,152 calls), a raw-note replay (864 calls), and a mechanism positive control (40 new states, 240 calls).

4. A separation of repeatability from predictive gain. The paper argues that reduced drift between repeated calls can coexist with unchanged or worse point error, and reports drift, point MAE, coverage, width, and interval score alongside each other rather than treating agreement as evidence of accuracy.

Main Findings

• The frozen prospective primary decisions did not confirm predictive credit. Controlled v5 returned inconclusive. Tox21 and OpenML both returned confirmation_no_go.

• Tox21's preregistered ROC AUC interval-score harm test was unmet. D−M = −.002628 with a 95 percent interval of [−.017392, +.010375]; S−M = −.002856 with [−.016750, +.011428]. The frozen harm gate required M−D and M−S to reach .005 with positive one-sided lower bounds.

• Structured cards sharply reduced repeat drift under Pro. Tox21 Pro drift fell from .01004 under description to .00357 with matched cards (64.5 percent reduction) and .00411 with seed-level donors (59.1 percent). Point MAE was .02056, .02041, and .02023 respectively, and interval scores were .08972, .09235, and .08949.

• The Flash replay reproduced the drift reduction but failed the equivalence rule. Matched and donor cards reduced drift by 60.6 and 59.2 percent, but matched point MAE rose from .01823 under description to .02020 (donor .02013). The frozen joint status was replication_not_supported because interval-score equivalence exceeded its ±.010 margin.

• Two Flash runs on the same stimuli gave different full-card drift magnitudes. The component crossover had a description baseline of .00427 (versus .00704 in the Flash replay) and full-card drift of .00319 — a reduction of .00108 with interval [−.00053, .00283]. All seven card contexts had point MAE above description's .01738 (range .01797 to .01873).

• Matched point-accuracy gains over description remained unconfirmed. In controlled v5, accuracy point MAE D−M was +.071 pp [−.102, +.252] and log-loss point MAE D−M was +.0363 [−.0061, +.0807]. The log-loss S−M contrast was +.0508 [+.0022, +.1076].

• Interval scores moved against matched cards in v5. Accuracy interval score D−M was −3.8436 [−7.9180, −.5468] and log-loss interval score D−M was −2.0878 [−4.2911, −.2750].

• OpenML seed-donor intervals spanned zero. Point MAE D−M was −.00741 [−.02804, +.00251] and S−M was −.00731 [−.03749, +.00654]. Matching effect signs were found in 27/36 Tox21 and 54/72 OpenML seed pairs; Tox21's median seed gap was .01362 ROC AUC against point MAE near .02.

• Full-card assignment widened OpenML intervals by about 21 percent. Mean width rose from .06179 to .07474, with a paired width change of .01294 and a task-stratified 95 percent interval of [.00141, .03306]. Coverage was 49.3 percent with full cards versus 51.4 percent with description, against a nominal 80 percent target. 78 of 144 assigned cards were empty scaffolds.

• Intervals were undercovered across all six OpenML contexts. Coverage ranged from 45.8 to 53.5 percent. On the post hoc |y| ≥ .01 subset it was 24.4 to 30.8 percent.

• Elicitation changed commitment rates but not calibration. Complete cards appeared in 0 of 60 ordinary proposals and 59 of 60 elicited proposals in v5. Under OpenML's common extraction path, completeness was 1 of 144 versus 20 of 144, a 13.2 percentage point difference with a one-sided 95 percent lower bound of 6.9 points — below the frozen .50 formation threshold.

• Direction accuracy largely followed a sign prior. Among v5's 111 explicit directions, 109 predicted improvement; accuracy was 65/111 (58.6 percent) while always predicting improvement scored 64/111 (57.7 percent). All 52 ordinary-prompt directions said improvement, reproducing that baseline's 59.6 percent. OpenML description-only accuracy was 63.19 percent against a constant-positive baseline of 62.5 percent.

• Direct-text Flash delivered all 144 notes without a detectable advantage. Description, matched, and donor point MAEs were .08257, .08311, and .08197; task-balanced scaled matched gains were −.0906 [−.2687, .0403] and .0064 [−.0964, .0898].

• Task scale concentrated OpenML error. Six brazilian_houses states accounted for 37.0 percent of description error and 41.1 percent of matched-full error. Overall description ensemble MAE was .08368 against .08341 for a zero forecast; omitting the task gave .05498 for description and .05614 for matched full against a .05910 zero baseline.

• A known-mechanism positive control improved accuracy by 2.60 percentage points. Point MAE was 3.795 with the true note, 6.391 with description, and 8.151 with the false note. The true note's gain over description had a 95 percent pair-bootstrap interval of [2.314, 2.880] and its advantage over the false note was 4.355 points [4.048, 4.654]. The executed quadratic intervention raised held-out accuracy by 11.44 points in nonlinear tasks and changed it by −1.06 points in paired linear tasks.

• The exact-signal calibration passed its gate. It produced 144/144 exact-signal wins and a point-value Spearman correlation of .99996; its frozen status was assay_sensitive.

Methodology in Plain English

The design is a paired, same-outcome comparison. For each state, researchers fix the public information and the planned intervention, then produce forecasts under three contexts that differ only in which explanation text, if any, is attached. Because the outcome, the forecaster, and the loss are held constant, any difference in loss is attributable to the explanation context.

Each forecast returns a point estimate and a central 80 percent interval, scored with a proper interval score that rewards narrow intervals and penalizes outcomes that fall outside them. Two independent calls per context measure repeat drift — the absolute difference between the two calls — which captures whether an explanation coordinates repeated outputs rather than improving them.

Three prospective studies supply the states. Controlled v5 crossed six study-authored fast machine-learning regimes with 60 spontaneous proposals and 60 elicited six-field prediction cards. The Tox21 anchor used 12 assay endpoints, three training-label fractions, and two seeds, with actions assigned cyclically so both seeds in a cell received the same intervention. OpenML v2.1 selected 12 classification tasks from OpenML-CC18 and 12 regression tasks from OpenML-CTR23 by a metadata-only hash lottery, with nested 25, 50, and 100 percent training-label budgets and two seeds, applying one of six changes to a LightGBM pipeline.

Commitment is measured by six prediction-card slots: a target direction, a quantitative target point or interval, a named intermediate observable, a benefit regime, a falsification criterion, and a counterfactual.

Inference: an intention-to-treat analysis keeps all assigned calls, including deterministic fallbacks for failures, and reports an integrity rate. Tox21 bootstraps endpoints, then budget cells, then seeds; OpenML stratifies by task kind and resamples tasks, with the 12 endpoints and 24 tasks as the external inference units. All prospective units froze before private outcomes were opened. The post-outcome controls (Flash replay, component crossover, raw-note replay, mechanism positive control, calibration) froze their own call plans separately, and the paper flags several arm contrasts as exploratory and unadjusted for multiplicity.

Models were accessed through tool-free Claude CLI sessions pointing at DeepSeek's Anthropic-compatible endpoint; source cards, forecasts, and calibration requested deepseek-v4-pro, and follow-up forecasts requested deepseek-v4-flash. Returned wrappers recorded usage without a resolved model ID.

Why This Matters

Impact on research. The paper reframes explanation evaluation for research agents around forecastable consequences rather than plausibility or post hoc narrative fit. It shows that a common summary statistic — repeated forecasts agreeing more closely — can improve while point accuracy does not, and that wider stated uncertainty can accompany worse coverage. It also documents an instance where the same stimuli produced different drift magnitudes across two runs of the same model family, and where a single task (brazilian_houses) supplied 37.0 percent of description error. Those are concrete reasons for benchmarks to report drift, point error, width, and coverage together, and to report task-level influence.

Real-world applications.

• Benchmarking autonomous research agents: scoring experimental rationales alongside achieved results, with the five checks as a reporting template rather than a single pass/fail number.
• Cost-aware experiment selection: the paper proposes that a utility rule can rank proposals using forecasters that pass paired predictive checks, so compute is allocated by demonstrated forecasting reliability.
• Scientific forecasting evaluation: extending paired-contrast measurement to predictions of empirical results in machine learning and adjacent fields, where the current signal is dominated by a constant sign prior.
• Calibration auditing: checking whether expressed uncertainty matches empirical coverage, since OpenML intervals covered 45.8 to 53.5 percent against an 80 percent target.

Industry relevance. Teams deploying research agents need

Authors’ abstract

Research agents explain planned experiments. We measure predictive credit with paired forecasts sharing an intervention, forecaster, and outcome while varying description, matched explanation, and donor context. Five checks track commitment, delivery, predictive gain, alignment, and known-signal uptake. Across 336 prospective states in controlled learning, 12 Tox21 endpoints, and 24 OpenML tasks, v5's frozen credit decision was inconclusive. Tox21's preregistered ROC AUC interval-score harm test was unmet ($D-M=-.0026$, 95 percent interval [$-.0174$, .0104]); OpenML's joint formation, point-equivalence, and repeatability rule was unmet. Matched point-accuracy gains over description remained unconfirmed, and Tox21/OpenML seed-donor intervals spanned zero. Under requested DeepSeek V4 Pro, matched and donor cards reduced secondary Tox21 drift by 64.5 and 59.1 percent. A DeepSeek V4 Flash replay raised matched point MAE from .01823 to .02020 and missed matched-donor interval-score equivalence. OpenML full-card assignment widened nominal 80 percent intervals by 21 percent, with 49.3 percent coverage versus 51.4 percent for description and content in 66/144 cards. Direct-text Flash delivered all 144 notes without detectable matched point-accuracy gain. A researcher-authored mechanism positive control lowered point MAE by 2.60 percentage points versus description. The protocol measures predictive credit for research-agent benchmarks and scientific forecasting; natural-explanation credit remained unconfirmed at the tested donor resolutions.