ClaimReceipt: Verifying Evidence Sufficiency and Coverage in Agent Evaluations

AI summary

Overview

Research area: Agent evaluation and verification — specifically, what evidence an agent benchmark must retain so that a reported claim can be recomputed, and what it must commit in advance so that missing runs become visible. The work sits at the intersection of evaluation methodology, construct validity, and audit receipts/provenance for multi-agent systems.

Technical level: Advanced. The paper defines a formal sufficiency condition over executions, a typed evidence schema, cryptographic commitments (Ed25519, X25519, HMAC-SHA256, OpenTimestamps), and a large frozen evaluation suite.

Scope (one sentence): The paper specifies and empirically tests ClaimReceipt v0.3, a claim-relative receipt specification and selective verifier that binds typed transaction evidence to a signed experiment manifest and returns Pass, Invalid, or Inconclusive per claim, evaluated on 1,392 historical buyer–seller records (CR-2) and a separate 30-transaction prospective epoch (CR-3).

What This Paper Is About

When an agent benchmark reports that a prompt, scaffold, guardrail, or model improved performance, two separate evidentiary questions arise. Sufficiency asks whether the reported number can be recomputed correctly from the evidence that was retained; coverage asks whether those retained records are the complete committed set rather than a favorable subset. The authors argue that generic logs, hash-linked transcripts, and final reward scalars answer neither question reliably, and they build ClaimReceipt as a constructive answer: a typed evidence schema, an experiment-level manifest, exact accounting semantics, and a verifier that is allowed to abstain rather than substitute a reported scalar.

Key Contributions

1. A formalization of claim-relative evidence sufficiency over executions. The paper defines a bounded claim class and provides a field-to-claim dependency matrix. The schema is claimed sufficient by construction for that class, not for arbitrary claims about intent or external truth.

2. A selective verifier operating at CR-2 and CR-3. It recomputes transaction and batch claims, preserves Inconclusive as a first-class output, and adds prospective manifest, ingress, encrypted-opening, receipt-chain, and conditional-coverage checks.

3. Evaluation on a previously audited corpus of 1,392 receipts across six epochs. The verifier agrees with 5/5 manual verdicts, exactly replays every declared transaction claim, and reports the complete historical 4 × 6 grid plus a separate prospective row rather than only labeled cells.

4. Schema and prospective testing. Field ablations, 11 frozen faults, and eight benign variations test schema semantics, while a separate 30-transaction prospective evaluation tests complete, missing-terminal, and missing-private-evidence cases with measured overhead.

Main Findings

• Manual audit reproduced (5/5). The verifier agrees on all five manually labeled cells. It marks E1 C2 Invalid because the unguarded arm used a room_attributes/price bundle and whole-bundle chooser while the guarded arm used a base-plus-add-ons schema and subset chooser; it marks E2 C2 Pass after those invariant fingerprints are unified; it returns Inconclusive for E4 C1 and E3 C3, and Pass for joint scripted accounting.

• No additional Invalid cell beyond E1 C2. The reported grid distinguishes contract abstention (I_C: no required contract) from evidential abstention (I_E: the contract exists but evidence does not cross its boundary). C4 passes with authorized evidence even when another criterion blocks the causal comparison.

• Constant-abstention ruled out by post-specification controls. Two synthetic decision-rule controls return Pass: C1 has a monotone +38.5 contrast with one-sided 95% lower bound 37.25, and C3 has a stable +28.5 contrast with central 95% interval [27.0, 30.1]. These controls do not enter the 1,392-record corpus or support a behavioral claim.

• Historical numbers reproduced rather than copied. In E4, mean seller profit is 25.8 under the compliance prompt, 57.5 under the standard prompt, and 33.8 under profit pressure; the strongest-minus-weakest profile contrast is +8.0 with one-sided 95% lower bootstrap bound −2.7 (central 95% interval [−12.0, 20.0]), and the ordered response is non-monotone. In E3, the profile-level guardrail effect is +37.6 with 95% interval [−34.2, 109.3] and mean within-cell generation SD 47.8, so the verifier abstains despite a positive point estimate.

• Exact replay at declared boundaries. All 600 deterministic records exactly match at policy, normalization, chooser, and metric stages. All 792 records in the LLM-associated epochs exactly match from the committed raw offer through choice and economic claims; their recorded LLM text is hashed and auditable but not regenerated. The 210 fixed-menu rows inside those epochs additionally replay from menu policy (policy 210/210, choice 792/792, metrics 792/792 in that tier).

• Generic evidence is not claim-sufficient. Outcome scalars, transcript + outcome, and generic signed receipts each determine 0/12 claims; an opened transaction receipt determines 6/12; ClaimReceipt v0.3 determines 12/12 within its declared scope.

• Non-redundancy under tested ablations. All 13 field-group removals make at least one claim underdetermined. The authors state this establishes non-redundancy only under the tested group ablations, not global field minimality over all 2^n subsets.

• Faults and benign controls. The verifier returns the frozen expected result on 11/11 semantic faults with 0/8 false positives. The held-out retry-shadowing fault returns C3 Invalid while the paired benign retained retry remains Pass. Benign variations that do not alter scientific verdicts include equivalent JSON key order, whitespace, integral numeric spelling, Unicode NFC, set order, idempotent redelivery, and transport-only timestamps.

• Prospective coverage behaves as predicted (CR-3). E7 committed 15 profiles under NONE and BOTH before inference; the signature-bound manifest time is 08:13:17.826Z, calendar attestations are obtained at 08:13:20.192Z, and the first of 30 preassigned ingress tickets is issued at 08:13:20.197Z. All receipts form one verified chain and reconcile to 24 accepted and six rejected terminals. The OpenTimestamps proof was upgraded to a Bitcoin block-header attestation at height 963833, though the host has no configured Bitcoin Core node.

• Omission is detectable. Removing the last terminal receipt leaves a valid 29-receipt prefix but returns INCONCLUSIVE_COVERAGE; C4 and descriptive licensing fall with it while C2 remains Pass. Withholding all private openings preserves coverage and protocol verification but makes all six economic and two treatment-fidelity claims Inconclusive — exactly matching a preregistered prediction frozen at SHA-256 72c554...4c0f. Opened evidence yields 6/6 determined economic claims and 2/2 determined fidelity claims.

• Negligible overhead. Receipt instrumentation adds 0.021% of model-inference time (428 μs including ticket versus 2,000.14 ms model inference per transaction) and 9.90 KB per transaction. Terminal write is 361 μs (0.018%), public verify 297 μs (0.015%), auditor verify 734 μs (0.037%, the 10-run median with range 724–828), and one cold auditor read is 1,040 μs (0.052%). Storage comprises a 2.64 KB public receipt, 6.21 KB encrypted envelope, and 1.05 KB ingress ticket; epoch verdict is 31.74 ms.

• The specification is not yet legible to an independent reader. Under the frozen single-response protocol the model exhausted its output budget before emitting final JSON, so no valid measurement was obtained and the run is not scored. A subsequent post-hoc, non-protocol-conformant per-item run yielded 10/11 parseable responses, of which 3/11 matched the verifier. The dominant error mode was temporal: the specification classifies faults by layer but never declares fault timing relative to commitment as an explicit attribute.

Methodology in Plain English

The authors start from a prior audit of an agent-to-agent commerce simulation that already separated four construct-validity questions: whether the seller instantiated the intended incentive (C1), whether treatment arms differed only in the intervention (C2), whether stochastic estimates were stable (C3), and whether outcome accounting was complete (C4). That earlier audit stopped a policy claim because the arms used different offer protocols and because incentive validity and generation stability did not pass.

ClaimReceipt is the constructive counterpart. Before any runs happen, a signed manifest commits the population, arm definitions, protocol fingerprints, replication structure, estimator, thresholds, and coverage policy. Each transaction receipt then links that manifest to an assignment, a private-profile commitment, an ordered trace, raw and normalized offers, the chooser decision, exact economic claims, and artifact digests. Private evidence is encrypted so an auditor — not the public — can open it; low-entropy values such as willingness-to-pay are committed with a keyed HMAC-SHA256 hash using a fresh secret 32-byte nonce rather than a bare hash, and canonical JSON follows RFC 8785 semantics.

Verification is deliberately staged. Replay has two tiers: for deterministic scripted sellers it starts at the committed policy artifact; for LLM sellers it starts at the recorded raw output. The authors explicitly do not claim that another model call reproduces the same text. Before the C1–C4 checks, the verifier canonicalizes and deduplicates idempotent deliveries, checks receipt IDs and chain links, opens profile commitments, resolves assignment coverage, and retains retry order; it then recomputes all transaction claims exactly. C1 and C3 average replications within profile cells before profile-level contrasts; generation rows never become independent buyers. A causal policy claim is licensed only when all four criteria pass and treatment fidelity is established, while a descriptive claim may remain licensed when C1 or C3 is inconclusive but C2 and C4 pass.

The specification was adversarially reviewed and frozen at v0.3 before the verifier was implemented (SHA-256 18d109...b81), with earlier v0.1 and v0.2 files and hashes remaining immutable. Two non-verdict observations informed amendments — an integrality audit of 9,744 historical amount cells and inspection of the historical match-quality scorer — and no C1–C4 verdict or fault result was observed before v0.3. The retrospective corpus of 1,392 terminal records is imported without its manual labels; E7 is a separate prospective epoch of 30 assignments committed before inference and is not added to the 1,392 count.

Why This Matters

Impact on research. The paper reframes verification as a claim-relative question rather than a cryptography question. A hash chain answers whether retained bytes changed; ClaimReceipt answers which retained distinctions a declared claim depends on. The authors present the field–claim matrix as the constructive dual of an identifiability failure: if removing a field group lets two admissible executions with different claim truth share one evidence projection, no later solver or metric can repair that loss. The paper also treats selective abstention as a result in its own right — a grid with many Inconclusive cells is intended behavior, on the argument that a gate which never stops an attractive result is not verification.

Real-world applications (as domains the paper engages):

• Agent benchmark reporting, where a claim that a prompt, scaffold, guardrail, or model improved performance needs to be recomputable from retained evidence.
• Multi-agent commerce and marketplace evaluation, where hidden buyer profiles and private willingness-to-pay are needed to audit welfare but may be policy-protected from the seller.
• Third-party or regulatory audit of evaluation claims, using the auditor-role separation, encrypted openings, and coverage-qualified verdicts the paper implements at CR-3.
• Model and evaluation disclosure artifacts such as datasheets and model cards, which the paper relates to machine-checkable contracts that stop missing evidence from silently becoming a positive conclusion.

Industry relevance. The measured cost is small — 0.021% of model-inference time and 9.9 KB per transaction — while the verifier distinguishes complete evidence, a missing terminal receipt, and unavailable private openings. The paper is also candid about what receipts cannot do: they do not create external coverage. A platform that prevents a transaction from reaching the witness remains outside observability, and every Pass therefore carries the qualifier "verified only within committed ingress." Independent key custody, authorization, retention, and deletion remain governance obligations; in the prototype, one operator still controls all local trust roles.

Future Directions

• Adopt the v0.4 fault-timing amendment. The paper proposes adding fault_timing ∈ {pre_commitment, post_commitment} to each taxonomy entry, with a preflight rule routing post-commitment cases to integrity checking before layer classification. The authors state the amendment but deliberately do not adopt it, because applying it after observing the probe would void the preregistration and re-testing on the same eleven items would evaluate a specification written to those items.

• Move from key-level to principal-level role separation. The trust model separates manifest authority, ingress authority, runner, and auditor by key and function rather than by organizational principal, and the OpenTimestamps proof is an external time anchor rather than an independent ingress witness. Independent ingress witnessing, key custody, and admission rules that reduce the open L4 layer are named as the next systems question.

• Test beyond the current scope. The evaluation draws on one configurable hotel testbed and one principal model family; E7 uses only Qwen2.5-Instruct 3B with k = 1, the 11 faults are designed rather than organic, and the abstraction analysis is scoped to the frozen claim class. The authors list a second domain, an independent implementation, and separately operated trust roles as untested.

• Study legibility as a measurable property of specifications. The legibility probe indicates the authors' own frozen specification is not yet unambiguous to an independent reader, but the result comes from one model under one protocol; the paper states it cannot separate specification ambiguity from model-specific reading failure and so bounds legibility from below rather than measuring it.

• Zero-knowledge metrics. Auditable welfare currently requires the evaluator's hidden profile, including sensitive low-entropy values; the paper lists zero-knowledge metrics as future work.

Target Audience

Researchers and engineers working on agent evaluation, benchmark methodology, and evaluation auditing; platform builders who need to justify performance claims about prompts, scaffolds, guardrails, or models; and practitioners of provenance, authenticated logging, or construct validity who want a concrete, machine-checkable contract for what an evaluation record licenses. The paper is also relevant to governance and compliance readers, because it is explicit about the privacy burden of auditable welfare and about the residual L4 risk that no receipt system can eliminate.

Authors’ abstract

Agent evaluations face two distinct evidentiary questions: whether a reported claim is recomputable from retained evidence (sufficiency), and whether the retained records cover the committed experiment set (coverage). Generic logs and hash-linked transcripts answer neither reliably. We introduce ClaimReceipt, a claim-relative receipt specification and selective verifier that binds typed transaction evidence to a signed experiment manifest and returns PASS, INVALID, or INCONCLUSIVE per claim. We freeze the specification before implementation (SHA-256 18d109...b81). On 1,392 historical buyer--seller records, a CR-2 verifier reproduces all five manually labeled audit verdicts, exactly replays 600 deterministic and 792 post-generation records, makes every one of 13 declared field groups non-redundant under tested ablations, and returns the expected result on 11/11 semantic faults with 0/8 false positives. We then run a separate prospective CR-3 epoch: 30 assignments are committed before inference, terminal receipts are signed and chained, and private evidence is encrypted for an auditor. Complete evidence yields coverage and accounting PASS; withholding one terminal receipt returns INCONCLUSIVE_COVERAGE, while withholding all private openings preserves coverage and protocol verification but makes economic claims inconclusive, exactly matching a preregistered prediction. Receipt instrumentation adds 0.021% of model-inference time and 9.9 KB per transaction. A specification-legibility probe indicates that our own frozen specification is not yet unambiguous to an independent reader. Claim verification therefore requires both claim-sufficient evidence and a committed universe against which omissions become visible.