Auditing Tool Traces in Agentic LLMs: Source Localization and Counterfactual Dependence Testing

AI summary

Overview

Research area: Evaluation and auditing of agentic LLM systems that ground predictions in external tool results, applied to a biomedical binding-prediction task (TCR–pMHC).

Technical level: Advanced. The paper assumes familiarity with AUROC, TF-IDF features, fine-tuning with LoRA, and counterfactual input-erasure analysis.

Scope: The paper introduces a two-stage audit — Per-Tool TF-IDF Decomposition (PT2D) for source localization and Token-Class Redaction for counterfactual dependence testing — and applies it to a Qwen3-4B model fine-tuned on five-tool traces for TCR–pMHC binding prediction.

What This Paper Is About

Agentic LLMs are increasingly given serialized outputs from external tools (databases, simulators, structure predictors) and asked to make scientific predictions from them. The problem is that these tool traces can carry hidden label shortcuts — identifiers, file paths, version strings, template artifacts — that correlate with the answer without embodying any real scientific evidence, so a model can look like it is reasoning from biology when it is actually exploiting a naming convention. The paper's goal is to build an audit that first locates which individual tool result carries such a shortcut, then tests whether the frozen model actually depends on it.

Key Contributions

1. A two-stage audit framework that separates localization from verification. Stage 1 (PT2D) localizes label-predictive text patterns to individual tool results; Stage 2 (Token-Class Redaction) counterfactually tests whether the frozen LLM behaviorally relies on the localized shortcut class. The paper argues these must be separate steps because a predictive text pattern found by a diagnostic classifier need not be used by the downstream LLM.

2. Per-tool decomposition as a remedy for the "dilution effect." PT2D audits each tool section independently rather than only the concatenated trace, on the argument that unrelated tool results can obscure a shortcut confined to one component. The paper supports this with both real tool-subset sweeps and label-independent synthetic additions.

3. A concrete case study that overturns an apparent success story. On TCR–pMHC binding, a tool-augmented Qwen3-4B model reaches 0.8136 test AUROC versus 0.4928 for a no-evidence baseline. The audit traces this gain to PDB identifiers in a single tool result and shows that redacting that identifier class collapses performance to 0.4972.

4. Token-class redaction methodology with a control. Redaction is applied globally across all prompt-bearing fields (to prevent duplicated values from surviving elsewhere), one class at a time at a frozen checkpoint, with a non-overlapping control class (peptide sequences) to distinguish targeted dependence from generic input perturbation.

Main Findings

• Full-trace screening produces a false negative. At the full-trace scope, PT2D AUROC stays near chance for all three feature sets: 0.4818 (word 1–2-grams), 0.4677 (character 2–5-grams), and 0.4876 (character 3–5-grams). Screening only the concatenated tool results would suggest little transferable leakage.

• Per-tool decomposition localizes the signal to Tool 3. Tool 3 (predict-thermompnn-ddg) reaches A₃ = 0.6288 and 0.6365 with character features, while Tool 5, which also serializes the PDB identifier, shows a weaker increase to 0.5259–0.5310; Tools 1, 2, and 4 remain near chance. For character 3–5-grams, the gap A₃ − A_full = 0.1489 quantifies what aggregation hides.

• The signal comes from identifier fragments, not biological language. Tool 3 word features stay near chance (A₃ = 0.4946), whereas character features reach 0.6365. Weight inspection points to the serialization template ThermoMPNN_inference_<pdb_id>.csv, with prefixes such as _21 favoring binders and _17 favoring non-binders. A two-digit PDB-prefix lookup learned on cluster 0 reaches 0.6967 AUROC on clusters 1–5.

• Dilution is monotonic and reproducible. Progressively appending Tools 1, 2, 4, and 5 to Tool 3 decreases the character 3–5-gram diagnostic AUROC from 0.6365 to 0.6049, 0.5284, 0.4956, and finally 0.4876. Label-independent synthetic additions reproduce the decay, with mean AUROC 0.5701 at K = 1, 0.5429 at K = 2, 0.5064 at K = 5, 0.4749 at K = 10, and 0.4900 at K = 20, reaching chance by K = 5.

• Redacting PDB identifiers destroys performance. The original model scores A_orig = 0.8136; with PDB IDs replaced by <PDB_ID>, AUROC falls to 0.4972, a drop of ΔA_PDB = 0.3163, eliminating nearly all above-chance ranking performance. Per-cluster AUROC falls from 0.5746, 0.8298, 0.8324, 0.8686, and 0.8322 (C1–C5) to 0.5126, 0.5241, 0.4818, 0.4963, and 0.4858.

• The control redaction leaves performance intact. Peptide redaction yields 0.8152 AUROC (ΔA_peptide = −0.0017), ruling out direct reuse of the literal epitope sequence and supporting a targeted rather than generic dependence.

• Format redaction shows broad sensitivity, not token-class attribution. Replacing step headers, trace banners, section headers, XML tags, and field names with <FORMAT> reduces AUROC to 0.5189 (ΔA_c = 0.2947); the authors explicitly frame this as a sensitivity check rather than a clean control because it removes multiple overlapping structural cues.

• Intervention sizes differ sharply by class. Across the 4,898 test prompts and their duplicated prompt-bearing fields, PDB-ID redaction applied 50,778 replacements, peptide redaction 63,674, and format redaction 3,582,603.

• The dependence is on the identifier class, not only on two digits. The authors state that short-prefix attribution comes only from PT2D weights, while the complete-ID intervention supplies behavioral evidence at the identifier-class level.

Methodology in Plain English

Setting up the audit. The authors take a labeled dataset where each example has an input, a binary label, and a "tool trace" — an ordered list of serialized results from five fixed tools (CDR3β biophysics, McPAS-TCR retrieval, ThermoMPNN ΔΔG predictions, a stability summary derived from ThermoMPNN, and a structure-based interface analysis). Training data comes from peptide cluster 0 and testing from peptide-disjoint clusters 1–5, a split designed so that test peptides differ from training peptides.

Stage 1: finding the suspect. For the full trace and for each tool section separately, they convert the text into TF-IDF features (word 1–2-grams, character 2–5-grams, character 3–5-grams) and fit a balanced logistic regression on cluster 0, then score clusters 1–5. Character n-grams are included specifically because they expose identifier fragments that word features would miss. This is a cheap diagnostic that requires no LLM retraining and no access to model parameters. Crucially, its evidence is only associational: it shows a text pattern predicts the label, not that the model uses it.

Stage 2: testing dependence. They trace the suspect evidence back to serialized fields and group values with the same generating semantics into "token classes" — PDB identifiers, peptide sequences, format scaffolding. They then replace every occurrence of one class at a time, across all prompt-bearing fields, with a placeholder, and re-run the exact same frozen checkpoint on the exact same examples with identical generation, parsing, and scoring. A large drop after targeted redaction plus stability under a control redaction is read as evidence of reliance.

Explaining the miss. To justify per-tool analysis, they run two dilution experiments: a real sweep that starts with Tool 3 and progressively appends the other tools, and a synthetic sweep that inserts label-independent text sections of matched length (mean 1,845.9 characters, drawn from a pool of 53,512 Tool 1, 2, and 4 sections) at K ∈ {0, 1, 2, 5, 10, 20}, averaged over five seeds.

Model and training. Qwen3-4B is fine-tuned with supervised fine-tuning for 10 epochs using rank-16 LoRA adapters (α = 32) on all linear layers, learning rate 10⁻⁵, global batch size 32; the redaction study evaluates the final update-4040 checkpoint. The model generates an integer a ∈ {1, …, 10} inside <answer>, mapped to s = (a − 1)/9 before computing AUROC.

Why This Matters

The paper shows that a benchmark gain in a scientific agentic pipeline can be entirely an artifact of how tool outputs were serialized. This matters because tool-augmented systems are increasingly presented as evidence that LLMs can reason over domain data, and because the failure mode is invisible to standard full-trace screening. The audit is cheap, requires no retraining, and produces an actionable finding: which tool, which field, which token class.

Real-world applications:

• Benchmark and leaderboard vetting. Running per-source audits before accepting tool-augmented gains as evidence of scientific reasoning or discovery, particularly for biology benchmarks built from deposited database records.
• Tool and serialization design. Removing or normalizing identifiers, file paths, and template artifacts from tool outputs before they enter a prompt, since the paper shows the shortcut entered through a filename and a duplicated complex_id field.
• Dataset curation and split validation. Detecting identifier-to-label correlations that survive semantic splits because their generating process does not follow scientific grouping.
• Model risk and compliance review. Providing a counterfactual artifact — performance before and after token-class redaction — that documents whether a deployed model depends on intended evidence or on incidental metadata.

Industry relevance: Any organization shipping LLM agents that call internal databases, simulators, or analysis tools faces the same risk, because serialization conventions are set by engineers rather than by the science. The audit's two stages map onto a practical QA workflow: a fast offline diagnostic over cached traces followed by a frozen-checkpoint ablation.

Future Directions

• Extending beyond a single benchmark and backbone. The authors note their experiments use one TCR–pMHC benchmark, one 4B backbone, and one tool suite, and that dilution behavior may vary with the length, format, and content of evidence in other settings.
• Making the audit preventive rather than post-hoc. The current method detects shortcuts after training; the paper leaves open how to screen tool traces or training data so that shortcuts never enter the pipeline.
• Moving from observable dependence to mechanism. The conclusions concern input–output behavior rather than the internal mechanisms producing that dependence, and the authors flag that redaction may also remove correlated signals or shift the input distribution.
• Aligning audit partitions with deployment protocols. PT2D is described as partition-agnostic, but its fitting and evaluation partitions must be aligned with the evaluation protocol of the target application, which needs to be worked out per domain.

Target Audience

Researchers and engineers building or evaluating tool-augmented and agentic LLM systems, especially in scientific domains where benchmarks are assembled from deposited database records. It is most useful to those who design tool interfaces and prompt serialization, to benchmark maintainers who need to distinguish genuine evidence use from metadata leakage, and to ML evaluation researchers interested in shortcut auditing and counterfactual input-erasure methods. Readers without background in AUROC-based evaluation, TF-IDF diagnostics, and fine-tuning will find the methodology sections demanding.

Authors’ abstract

Agentic LLMs increasingly support scientific prediction by grounding decisions in external evidence. In biology, they promise accurate, interpretable predictions for important problems by integrating complex experimental and computational evidence. However, serialized tool results can contain hidden label shortcuts, such as dataset identifiers or template artifacts, that create apparent success without biological reasoning. To address this risk, we introduce a two-stage audit. Stage 1, Per-Tool TF-IDF Decomposition (PT2D), localizes label-predictive patterns to individual tool results while avoiding signal dilution in the concatenated tool trace. Stage 2, Token-Class Redaction, counterfactually tests whether the frozen LLM depends on the localized shortcut. We apply this audit to TCR-pMHC binding, where tool augmentation substantially outperforms a no-evidence baseline. At face value, this suggests that the LLM has learned to use scientific evidence. The audit overturns this interpretation: PT2D localizes the strongest signal to identifiers in one tool result, and redacting them reduces performance to chance. These findings show how hidden label shortcuts can masquerade as scientific reasoning or discovery, motivating source-level audits before benchmark gains are trusted.