From Tool State to Scientific Action: Auditing Decision Provenance in Biomedical Research Agents

AI summary

Overview

Research area: Reliability and auditability of AI research agents in biomedicine, specifically how silent numerical hardware faults propagate into the evidence an agent reads and cites, and into real experimental decisions such as which compounds to send to a cardiac-safety assay.

Technical level: Intermediate to Advanced. The core ideas are graspable without specialist background, but the paper leans on numerical analysis (floating-point error bounds, FP32/BF16/FP64), GPU fault classes, and agent execution traces.

Scope (one sentence): The paper introduces BioAgent-DF, a paired-intervention audit that saves "reference" and "candidate" tool values at the same computation site, restores or transplants them at their original coordinates on healthy hardware, and measures whether the agent's reads, citations, verdicts, and downstream scientific actions change — and which replacements undo those changes.

What This Paper Is About

Biomedical research agents call tools (literature search, molecular scoring models) and then decide what to read, what to cite, and what to test experimentally. If a tool silently returns slightly wrong numbers — for example, from faulty GPU arithmetic — the agent's final verdict may look identical while the evidence behind it changes completely, or a real decision such as which compounds to screen may flip. The paper asks two questions: do silently corrupted tool values change an agent's evidence choices and decisions, and how much of the tool's state must be restored to undo those changes?

Key Contributions

1. A paired tool-state intervention method (BioAgent-DF): Reference and candidate values are captured at a single computation site, then selectively restored (reference values substituted into the candidate state) or transplanted (candidate values substituted into the reference state) at their original coordinates, with the agent continuing on healthy hardware.
2. An audit formalization that separates evidence-path fidelity from verdict correctness: Endpoints are decomposed into retrieved documents, read documents, cited sentences, structured actions, and verdict, so that "same answer, different evidence" is measurable rather than invisible.
3. Empirical fault-injection results across two agent models and a real assay-allocation task: SciFact literature-agent experiments with Qwen3.5-4B and Gemma 3 4B IT, plus hERG cardiac-safety allocation over 2,009 held-out compounds run through Biomni A1's native workflow.
4. A recovery and certification procedure: Selective recomputation driven by arithmetic checksums and a decision-margin rule (R1+M, with ε = 0.009354), together with a certified-commitment proposition and algorithm governing when a decision may authorize action.

Main Findings

• Same verdict, different evidence: In the motivating example, Gemma's reference branch cites a sentence about ROS and autophagy (19922508/3) and its candidate branch cites one about ROS sensing microbe size (22696649/1), while both return the benchmark-correct SUPPORT label. Neither citation matches the annotated evidence, 28937856/5.
• Repeatable evidence-path changes across eight batches: Qwen3.5-4B shows repeatable citation changes in three capture–claim combinations and Gemma 3 4B IT shows repeatable read and citation changes in four. These seven combinations cover only claims 431 and 729. H1, H2, and full restoration agree, and verdicts do not change in these confirmed cases.
• Full restoration works, partial masks do not: Full restoration recovers the reference evidence sets in every confirmed case. Selected restoration and transplantation never reach their targets in the relevant groups (Qwen 0/9; Gemma 0/12, with three groups incomplete), and no partial-row mask both restored the reference path and reproduced the candidate path across the eight batches.
• Prompts change which endpoint moves: Under the baseline prompt, six Gemma groups on claim 431 kept reads and citations fixed while the candidate state turned an incorrect NEI into a correct SUPPORT; full restoration returned the incorrect reference verdict. With an explicit-example prompt, Gemma's read and citation changes were 15 across five combinations for claims 343, 431, and 729, but verdict differences fell from six to zero. In 36 Gemma groups complete under both prompts, the eight baseline citation differences (all claim 268) disappeared and nine appeared on other claims.
• Selection matters: In a Qwen cohort selected without requiring an upstream membership change (382 jointly completed, exposed groups out of 421), candidate–H1 discordance was 3, 5, and 2 groups for reads, citations, and verdicts, no higher than the H2–H1 discordance of 4, 8, and 2. Without selection on an upstream change, no excess over repeat-to-repeat variation appears.
• Content carries the decision: With titles and abstracts masked, all 243 masked attempts per model finished with uncited NEI under the baseline prompt, and verdicts changed in 57/81 Qwen and 28/40 Gemma comparisons against full-content runs.
• Arithmetic anomalies are distinguishable from ordinary precision effects: An exact-integer probe produced 405613.25 against an exact 405760. Three runs of the tensor-core matrix multiply-accumulate gave 38, 24, and 54 mismatching outputs versus zero for the paired FP32 fused multiply-add. In PBMC, disabling intermediate BF16 reduction cut Top-10 membership disagreements from 174 queries to zero, leaving two order disagreements — evidence of ordinary precision effects rather than hardware faults.
• hERG decisions changed but the shortlist size did not: Case 1 preserved Top-402 membership and changed triage in one of eight sources; Case 2 changed each endpoint in six of eight. All nine list replacements exchanged labeled blockers, so the list still held 351 labeled blockers among its 402 compounds. Compound K5612 (CHEMBL3804918, published hERG radioligand-displacement IC50 of 434 nM) left the list in four sources; labeled blockers K12252 and K1461 were assigned to advancement without testing; K8429 fell from 0.880 to 0.584.
• State restoration beats file replacement: After the agent had read faulty predictions, replacing the prediction file recovered none of the 39 changed decisions, whereas restoring the execution state saved before that read and continuing with recomputed scores recovered all 39 and preserved the other 63. No incorrect action was committed. Recovery before the agent read the predictions gave the healthy decision in all 102 requests.
• Checksum verification caught every faulty source: Stored-data integrity checks and finite/between-0-and-1 checks flagged none of the 16 faulty sources; row and column checksums with a rounding bound flagged all 16, with no false alarm in eight healthy verifier runs. R1 recovered every changed decision except one added assay for K13418, the compound closest to the triage threshold.
• The margin rule closed the gap: R1+M, which also recomputed all three ensemble members for the 68 of 2,009 compounds with a decision margin below ε = 0.009354, matched the healthy reference selection and triage in all 16 sources, and all 96 fresh native Biomni trajectories on R1+M inputs reproduced the healthy decisions.
• Persistent history matters for later choices: Across 24 groups resumed from saved prefixes under seven conditions, the candidate prefix and its repeat each showed citation discordance in 6/24 groups and tool-only replacement in 7/24, while healthy repeat, joint replacement of history and tool state, and fresh healthy replay each showed 2/24. Verdicts agreed throughout.
• Subsequent choices shift under ordinary rounding: In PBMC, actions or marker selection changed in 9/30 groups after BF16 rounding, versus 0/30 for repeats and zero-residual controls. NCI membership changed in 11/61 and 13/62; ESOL sampled membership changed in 12/24 in each environment, with order changes in 15/24 seeds.
• Speed: A streaming implementation of the arithmetic checks reduced total time for forward execution and verification by 42.4% relative to the original FP64 implementation, with identical flags and recovery outputs; a 0.47 s forward pass became 0.59 s, compared with 1.02 s for the original FP64 implementation and 0.88 s for double execution.
• Cost of recovery: Continuing from the restored state added about 29,000 tokens per request to the 44,000 already spent analyzing faulty predictions; recovery before reading needed one analysis of about 44,000 tokens.

Methodology in Plain English

The researchers first observed real silent data corruption on faulty GPUs used in distributed training, sorted into two classes: Case 1 (size-dependent, failing more on large-matrix paths than short probes) and Case 2 (expanded-trigger, failing on the short probes too). Rather than simulate errors, they captured actual reference and candidate numerical states at a single computation site and replayed them.

The audit works like a controlled swap. At the saved site, states share shape, coordinates, and representation, so a subset of coordinates can be swapped between reference and candidate. Restoration takes reference values into the candidate state; transplantation takes candidate values into the reference state. The remaining tool and agent steps then run on healthy hardware, at temperature zero, with matched inputs, model, document order, and budgets. Because the agent only sees tool outputs and not the ground-truth values, the researchers can check whether the agent's retrieved documents, read documents, cited sentences, actions, and verdict match the reference — and what fraction of the differing values had to be replaced to get there.

For the literature agent, they used SciFact with search, read, and finish actions and up to six document reads, 16 calls, and 384 output tokens per episode. They ran 27 Case 2 capture–claim combinations covering 11 claims, three technical repeats each, across seven conditions, then repeated everything in eight further batches with a fixed request seed (27 combinations × 8 batches × 7 conditions = 1,512 episodes per model). They added baseline and explicit-example prompt variants, content-masking controls, and unselected-claim controls.

For the biomedical action task, a three-member Chemprop directed message-passing ensemble predicted hERG blockade for 2,009 held-out compounds, and Biomni A1's native workflow using GPT-6.1 Sol selected a Top-402 shortlist and triaged compounds to patch-clamp testing or advancement without testing. They compared 16 saved BF16 passes against a healthy reference from the same GPU model and host, then tested recovery by coordinate restoration, by checksum-triggered recomputation (R1), and by a margin rule (R1+M).

Why This Matters

Impact on research: Standard evaluations score the final answer. This paper shows that an agent can produce the benchmark-correct verdict while citing sentences that do not support its own stated mechanism, or while sending different compounds to a safety assay at the same aggregate count. Auditing verdicts alone is therefore insufficient; provenance must record what was read and cited, and whether the numbers behind those reads were verified.

Real-world applications:

• Automated laboratory pipelines, where a numerical fault that changes a decision changes the experiment actually performed.
• Cardiac-safety screening, where a compound misassigned to "advance without testing" carries a liability that no aggregate error count would surface.
• Literature-grounded evidence synthesis, where citation fidelity matters independently of the claim label returned.
• Model-training infrastructure, where a single-bit arithmetic error can propagate through optimizer state into different top-ranked results while losses remain bitwise identical.

Industry relevance: The paper offers a concrete, cheap deployment pattern — checksum verification plus selective recomputation plus a certified-commitment gate that commits at most once per request and recomputes everything on trusted hardware if verification fails or the budget is exhausted — and reports that its streaming checks cost 0.59 s against a 0.47 s forward pass and 1.02 s for the original FP64 implementation.

Future Directions

• Beyond two claims and two models: The confirmed evidence-path changes span only claims 431 and 729 with two 4B models in Case 2 captures selected for upstream retrieval changes. Whether the effect generalizes to other claims, larger models, and unselected captures is open.
• Certification with valid per-score bounds: R1+M is explicitly described as an empirical rule, with ε set to twice the largest retrospective R1 residual. The paper's Proposition 1 and Corollary 1 give sufficient conditions for certified threshold and Top-K decisions, but a deployment with genuinely valid per-score error bounds — or a fallback to full trusted recomputation — remains to be built and validated.
• Recovering complete training order and state: When restoration was delayed by up to 32 updates, 30/32 partial restorations recovered membership but none recovered complete order or state, while all 16 full-state restorations recovered their endpoints. Cheaper partial recovery of optimizer state is unresolved.
• Certifying every value an agent sees: Certification fixes the agent's input only when every returned value is determined by the certified target or certified separately. Extending certification across all tool outputs an agent consumes, rather than the target decision alone, is left as an open engineering problem.

Target Audience

Researchers and engineers working on AI agents for scientific discovery, ML reliability and hardware-fault tolerance, and evaluation methodology for tool-using language models. It is also relevant to computational biologists and cheminformaticians who deploy automated screening or assay-allocation pipelines, and to reviewers and auditors who need to argue that an agent's conclusion was backed by the evidence it claims.

Authors’ abstract

A scientific agent can give the same verdict while reading and citing different evidence. BioAgent-DF tests this separation using paired reference and candidate values saved at the same scientific computation site. It restores or transplants those values at their original coordinates, then runs the agent on healthy hardware. Across eight SciFact batches, Qwen3.5-4B shows repeatable citation changes in three capture-claim combinations, and Gemma 3 4B IT shows repeatable read and citation changes in four. These seven combinations cover two claims. Full restoration recovers the reference evidence sets while verdicts remain unchanged. Verdict changes observed under the baseline prompt did not recur under the explicit-example prompt. Complementary tool and optimizer experiments distinguish recovery of numerical state, recovery of ranked lists, and restoration of an agent's visible history. These results show that an audit should track both the evidence an agent uses and its final answer. In hERG cardiac-safety assay allocation, silent numerical faults changed testing decisions while the fixed-budget shortlist kept the same number of compounds labeled as blockers. After the agent had used faulty values, restoring its earlier execution state and supplying certified scores recovered all 39 changed decisions, whereas replacing the prediction file recovered none; no incorrect action was committed. A streaming implementation of the arithmetic checks reduced the total time for forward execution and verification by 42.4% relative to the original FP64 implementation, with identical flags and recovery outputs.