Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents

AI summary

Overview

Research area: Multi-agent systems and AI-for-science evaluation — specifically, methods for auditing the claims made by AI research agents.

Technical level: Advanced. The paper is a protocol/statistics paper that assumes familiarity with confidence bounds, preregistration, randomized paired trials, and equivalence testing.

Scope in one sentence: The paper introduces and empirically calibrates the Discovery Certification Protocol (DCP), a three-gate, preregistered audit that turns claims about AI research results into executable recovery tests and randomized feedback tests, and it reports two complete audits (SQLite optimization and virtual catalyst control) plus three diagnostic cases.

What This Paper Is About

When an AI research agent reports a successful result, a good score alone does not show that the agent's own experiments and reasoning produced it. The result might instead be reachable from the starting information, public material, or prior knowledge. DCP addresses this by asking three separate, checkable questions about one numerical outcome: was the result a useful improvement on sealed data (Gate 1), can a matched agent that receives the starting information and captured Web content but is denied the target run's research history reach the same score (Gate 2), and does truthful experimental feedback improve outcomes relative to a specified neutral policy (Gate 3)?

Key Contributions

1. An outcome-level audit with a shared recovery rule. DCP admits every valid alternative method to the same numerical outcome, using a registered validity predicate and a tolerance-based score threshold, so the rule is portable across programs, models, data products, and experimental recipes.

2. Two explicit decisions built from complementary evidence. "Core" combines qualified recovery witnesses, zero observed recoveries, and a finite-sample bound on recovery in one fresh registered episode. "Evidence" adds a randomized, checkpoint-conditional feedback effect after independent null calibration and a registered effect margin.

3. Calibration of the complete protocol across two domains and two models. SQLite-Web optimization (DeepSeek-v4-flash) and virtual catalyst control (DeepSeek-v4-pro) each ran all three gates, and three diagnostic cases (device calibration, multidimensional knapsack, low-sample affine) exercise the Core, recovered, and audit-incomplete decisions.

4. Portable evidence bundles and a deterministic, LLM-free verifier. The verifier reproduces decisions from frozen evidence records, separating evidence production from decision checking so independent readers can replay the same numerical decision.

Main Findings

• Both complete audits supported Core plus Evidence. SQLite-Web had a main score of 0.8855 against a baseline of 0 and a recovery line of 0.8805; virtual catalyst had a main score of 1.0000, a baseline of 0.5990, and a recovery line of 0.9500. Each audit recorded 0/96 recoveries, with an upper bound of 0.0468.

• Positive controls passed in all Core cases. SQLite-Web and catalyst each passed 45/45 positive controls, giving a recall lower bound of 0.8889, above the registered 0.8 minimum. Device calibration returned 0/80 recoveries with 60/60 positive controls and an upper bound of 0.0477.

• No-lineage challengers stayed below the recovery line. In SQLite-Web, challengers received every captured Web byte and reached at most 0.6734 against a recovery target of 0.8805. In catalyst control, challengers reached at most 0.8146 against 0.95.

• A developmental case triggered the Core veto. The multidimensional knapsack case had a recovery line of 0.9329; one matched episode returned legal artifacts scoring 0.9363 and 0.9356, both above the line, so the better one exceeded the target run and refuted Core.

• Feedback studies produced a large, null-calibrated effect. Both paired studies yielded 30/30 truthful recoveries and 0/30 neutral recoveries. The estimated binary policy effect was 1.0 with a conservative exact paired-binary 99% interval of [0.6379, 1.0]. Each separate 60-pair null study gave a contrast of 0 with a 99% interval of [-0.0950, 0.0950], inside the registered ±0.17 band.

• The Evidence margin was met under the registered arithmetic. The required total margin was 0.34 + 0.17 = 0.51, and the recorded calculation was 0.6379 − 0.17 = 0.4679 > 0.34. The verifier returned Evidence in both scopes. Separately, the neutral-recovery upper bound at 30 episodes was 0.1619, reported as a distinct quantity from the average feedback benefit.

• The decision space was covered by the case suite. Five cases exercised four outcomes: SQLite and catalyst support Evidence, device calibration supports Core, knapsack supplies qualified recovery, and the low-sample affine case remains audit incomplete.

• Costs and scale were reported. The catalyst and SQLite-Web audits used 435 and 507 recorded sessions and cost 56.40 and 61.17 USD respectively. Appendix details report the catalyst confirmatory execution as 419 recorded model sessions at 54.27 USD, the SQLite-Web Web-disabled Core audit as 130 recorded model sessions at 25.56 USD, and the SQLite-Web full recorded run as 507 model sessions at 61.17 USD.

Methodology in Plain English

DCP organizes an audit around five fixed objects: the background fixed before the run (K), task observations fixed independently of the run's actions (E0), the research history created during the target run (L*), the final output chosen by a preregistered selection rule (A*), and a machine check for recovering the claimed result (P). Registration happens before the target run and freezes the task, model, information boundary, tools, budgets, baseline, validity rules, and statistical plan.

Gate 1 checks that the target output is a valid artifact and that its mean sealed score beats the baseline by at least a registered minimum useful gain, using a lower confidence bound on the [0,1] score scale.

Gate 2 gives a fresh matched agent the same background, complete starting observations, model, tools, and all Web bytes the target run observed — but withholds the target run's research history. Any valid method that reaches the target score within a small tolerance counts as a qualified recovery witness and triggers a veto. If there are zero recoveries across n independent episodes, the protocol computes a fixed-sample upper bound on the probability that a fresh episode recovers; Core requires that bound to fall at or below a registered threshold, along with an adequate audit and passing positive controls.

Gate 3 (optional) starts fresh paired branches from a frozen checkpoint; one arm gets truthful feedback from its own actions, the other gets messages from a specified neutral policy that preserves timing, schema, and length but withholds the correct next action. The estimated effect is the expected utility difference conditional on the checkpoint, and it only counts as Evidence if an independent null study on known-answer tasks falls inside a registered equivalence band and the effect's lower bound exceeds the sum of a registered effect threshold and calibration margin.

The reported audits used a recording gateway for Web access, one symmetric evaluator for baseline, target, and controls, committed draws for randomized presentations, and a deterministic offline verifier that recomputes decisions from frozen bundles.

Why This Matters

Impact on research. The paper argues that a measured score does not by itself establish that an agent's experiments and reasoning produced a discovery. By separating utility validation, alternative-route recovery, and feedback effects into distinct decisions, DCP gives authors, reviewers, and auditors a shared evidence language that can be replayed offline rather than trusted on narrative alone.

Real-world applications (as exercised or described in the paper):

• Software and database optimization. The SQLite-Web audit reduced traffic-weighted SQLite virtual-machine work by 88.55% relative to a no-index baseline while preserving every query result.
• Experimental science and laboratory control. The virtual catalyst task involved five controls with eight legal levels each (8^5 = 32,768 possible recipes) and a 36-well one-factor plate expanded from a committed anchor recipe.
• Evaluation of agent benchmarking and search pipelines. DCP distinguishes adaptive research from a search-and-selection baseline, and treats a best-of-k procedure as a single k-candidate episode so the full search budget is counted.
• Auditing claims made against captured Web sources. Gate 2 discloses the exact model-visible Web bytes, and a claim about the contribution of Web access requires an additional Web-withholding intervention.

Industry relevance. The protocol's cost unit is the registered generation episode, with a reader's verification operating on saved evidence — the reported complete audits cost 61.17 and 56.40 USD. DCP's provenance requirements (sealed registration, complete ledgers, hashed records, independent countersigning) map onto practices organizations already use for compliance and reproducible benchmarking.

Future Directions

• Extending to new models, budgets, and domains. The paper states that later studies can register a new model or budget and compare recovery under that scope; only SQLite optimization and virtual catalyst control were exercised as complete three-gate audits.

• Prospective versus retrospective auditing. The paper describes retrospective use that reconstructs an outcome target and seeks a recovery witness, distinguishing an audit with pre-publication-equivalent model and information access from one using post-publication knowledge that measures present-day recoverability.

• Resolving the incomplete and uncertain cases. The low-sample affine case left the registered adequacy requirement unresolved despite an observed positive feedback contrast, and the protocol defines an inconclusive outcome for interval positions between thresholds — the paper does not report a follow-up design that resolves such cases.

• Broader adapters and harness reuse. The release provides the installable dcp-audit package, task-specific evidence producers, a reusable dcp-harness, example bundles, and figure-generation scripts; the paper does not report additional tasks beyond the five cases described.

Target Audience

Researchers and engineers working on AI research agents and automated discovery; evaluation and benchmarking practitioners who need to distinguish genuine agent-driven results from recoverable ones; statisticians and methodologists interested in preregistration, equivalence testing, and randomized feedback designs; and reviewers, auditors, or policy staff who need a replayable record to challenge a discovery claim. The protocol details in the appendices and the statistical allocations (0.005 each for main validation, candidate classification, positive-control adequacy, and neutral recovery; 0.01 each for challenger recovery, target feedback effect, and null calibration, totaling 0.05) will be most useful to readers with a quantitative background.

Authors’ abstract

AI research agents combine prior knowledge, public sources, and experimental feedback to produce useful results. The Discovery Certification Protocol (DCP) turns claims about these results into executable recovery and feedback tests. Gate 1 validates useful improvement on sealed evaluation. Gate 2 gives matched agents the registered starting information and observed Web content while withholding the target research history. Every valid method reaching the numerical target supplies a recovery witness and triggers the Core veto. DCP Core requires adequate controls, zero observed recoveries, and a finite-sample bound on recovery in one fresh registered episode. Optional Gate 3 measures the average effect of truthful feedback relative to a specified neutral policy from a shared checkpoint. DCP Evidence adds this effect after independent null calibration and a registered effect margin. Two controlled audits exercise the complete protocol in SQLite optimization and virtual catalyst control under different models. Each produced zero recoveries in 96 episodes, with an upper bound of 0.0468. Each paired study yielded 30 truthful recoveries and zero neutral recoveries, with passing 60-pair null studies. Additional cases exercise Core, recovered, and audit-incomplete decisions. A deterministic, LLM-free verifier reproduces the decisions from frozen evidence. DCP provides a common evidence language for useful outcomes, alternative routes, and feedback effects across AI research.