Who Verifies the Tools? How Warped Formatters Mislead Agents and Their Verifiers

AI summary

Overview

Research area: AI agent reliability, tool-use interfaces, and LLM-as-verifier evaluation (cs.AI), grounded in a production retrieval-augmented generation (RAG) system for a historical archive.

Technical level: Intermediate. The core idea is intuitive, but the paper assumes familiarity with agent tool-calling loops, RAG pipelines, and LLM-judge/verifier setups.

Scope (one sentence): The paper measures, through a paired experiment on 32 holdout cases across three model families, whether the text a tool prints changes what agents claim and what verifiers accept, independent of whether the tool's computation was correct.

What This Paper Is About

Agents and their verifiers only see the words a tool prints, not the code or database behind it. When a formatter omits a search cap, a result limit, an "estimated" label, or an identifier namespace, every sentence it prints can be true while the picture it conveys is wrong — the authors call this warped formatting. The paper asks a single controlled question: if you hold the tool's actual result fixed and change only the wording, do agents overclaim more and do verifiers accept more?

Key Contributions

1. A taxonomy of four formatter defect families derived from an audit of a live production research system: scope (claiming more coverage than was searched), completeness (presenting a truncated result list as the total), uncertainty (presenting a derived value as confirmed), and provenance (identifiers without a namespace).
2. A paired, hash-frozen evaluation in which the same real tool result is rendered in a warped version (Condition A) and a clear version (Condition B), so any downstream change is attributable to the wording. It spans 40 cases (10 per family), 8 tuning cases excluded, 32 holdout cases, three models, five replicates each, totaling 1,200 agent runs.
3. A two-part verifier measurement: a fixed-answer causal measure (one false answer per case shown under both wordings, 240 verdicts) and a frequency measure on answers agents actually wrote, plus a control crossing two questions (supported vs. correct) with two evidence levels (tool output alone vs. with verified facts).
4. A frozen, reusable audit prompt for general-purpose coding agents, demonstrated on the authors' own system (where it found both live defects and twelve uncatalogued ones) and on two open-source agentic systems, OpenClaw and Hermes, with a merged fix for OpenClaw.

Main Findings

• Clear wording sharply reduced agent overclaim for three of four families. Pooled overclaim under warped wording ranged from 62% to 100% depending on family; the clear wording cut scope to 8%, completeness to 0%, and uncertainty to 20%, while provenance stayed at 94%. Paired differences: scope +92 pp (95% CI +80 to +100), completeness +62 pp (+46 to +78), uncertainty +80 pp (+73 to +87), provenance +6 pp (+2 to +11).
• Verifier verdicts flipped completely. In the fixed-answer measure, verifiers accepted the false answer 30/30 times under warped wording and 0/30 under clear wording in each of scope, completeness, and uncertainty — 90/90 flips. Provenance moved only from 18/30 (60%) to 14/30 (47%).
• The control showed both evidence and task framing are required. On 46 warped-wording false answers (92 verdicts per cell), asking "is it supported?" gave 88/92 (96%) acceptance with output only and 56/92 (61%) with facts added; asking "is it correct?" gave 90/92 (98%) with output only and 0/92 (0%) with facts added.
• The frequency measure confirmed high acceptance on real agent answers: scope 229/240 (95%), completeness 136/148 (92%), uncertainty 220/240 (92%), provenance 120/240 (50%). The authors note these rates are not causal, since conditions select different answers.
• Provenance was the family wording could not repair. All but two of 17,807 document IDs were also valid scan IDs for unrelated items; agents presented document IDs as scan IDs even when tagged [doc:…], and only 7/120 clear-condition answers correctly stated that no scan ID was present. The authors conclude identifier namespaces need enforcement through types or validation rather than clearer formatting.
• "Estimate laundering" persisted under clear wording. The uncertainty tool line labels its guess ("ESTIMATED page 8 of 14 … proportional estimate, NOT a confirmed page") but also names the matching archive page (149) derived from the same guess; answers hedged the labeled number while stating the archive page as fact, leaving a 20% overclaim rate.
• Grading reliability was high. The two model graders agreed on 1,147/1,199 dually graded answers (96%); 53 answers went to human adjudication (52 disagreements plus one answer with a single valid grade). An AI research assistant blindly regraded all 64 pilot transcripts and matched consensus on all 64; a human matched 10/12 pilot transcripts and 35 of the 36 in-scope full-run answers from the three reported models.
• Robustness checks held. Strict document-level holdout left scope at 105/105 (100%) under A and 9/105 (9%) under B. A deterministic construction checker flagged 40/40 warped texts and 0/40 clear texts across the 80 frozen texts.
• The audit prompt generalized. Run on the authors' pre-fix source tree with Codex and Claude Code, it rediscovered both defects still live at that commit, did not flag the two already-fixed families, and surfaced twelve distinct additional instances, including a semantic_search formatter that silently returned three of five retrieved results. On OpenClaw it found attachments_fetch reporting a message outside its bounded recent-history window as not found, and the fix was merged. On Hermes it found search_files truncating matched lines at 500 characters without a marker; that fix remains open.
• Cost: total API spend across all pilots and the full run was $74.44.

Methodology in Plain English

The authors start from real incidents in a deployed research assistant for a digitized archive of 17,807 documents. Examples: keyword_search opened with "Found 10 document(s)" when 191 documents actually matched; date_range_search opened with "Found 20 document(s)" for 1920 when the archive held 4,772 matches, and 90 distinct years exceeded one fetch; find_in_document printed "Searched the FULL text" after inspecting at most the first 1,000,000 characters, in an archive where 14 documents had a transcript over the cap, 7 of which contained a term first appearing only beyond the limit (one at character position 1,000,072 in a 1,031,201-character transcript).

They then build a controlled comparison. Each case is one concrete tool call with a known correct answer, derived from reproduced database facts by deterministic rules. The same result is rendered two ways: Condition A with the warped wording (sometimes the exact text that had been in production) and Condition B with the clear wording that ships today, with the two texts hash-frozen before results so they cannot be revised. Only the tool output text changes; the system prompt and question stay identical across conditions.

For the agent side, three models — anthropic/claude-sonnet-5, openai/gpt-5.6-sol, and google/gemini-3.1-pro-preview — each produce five replicates per case per condition at temperature 1.0 with a 4,000-token output cap, giving 1,200 runs. Grading rules are frozen after pilots and before the final run: each case has a pre-written list of allowed and forbidden claims, with hedging accepted when a qualification covers the cited value. Two model graders from different vendors independently assess every answer; agreement is the grade, disagreement goes to an author.

For the verifier side, the fixed-answer measure swaps only the formatter wording under a fixed false answer, isolating causation. The frequency measure instead judges the agent's real answer, with no verifier judging its own vendor's output. A control then varies the question (supported vs. correct) and the evidence (tool output alone vs. plus verified facts) to test whether failures persist when verifiers have more to work with.

Finally, they write and freeze a short audit prompt for ordinary coding agents, asking the agent to inventory every formatter, compare what the output says against what the code did, and report findings without fixing anything.

Why This Matters

Impact on research. Existing groundedness and faithfulness methods — FEVER, FActScore, Ragas — grade claims against supplied source text. If the formatter already dropped a boundary or qualification, those methods cannot recover it. Benchmarks such as AgentBench and τ-bench treat visible tool output as part of the environment and never test whether it faithfully describes the tool's internal result; GroundEval treats the tool layer as trusted ground truth. This paper locates a failure mode upstream of all of them: at the formatter boundary, where nothing currently looks.

Real-world applications:

• Enterprise RAG and internal search assistants, where a silent result cap can turn "the top 20 of thousands" into a confident but false answer for an end user.
• Archival, legal, and compliance search, where a "not found" from a partially searched document can be read as evidence of absence.
• Agent observability and evaluation tooling, where verifier pipelines inherit exactly the same blind spot as the agents they police.
• Systematic review and data-extraction agents, where unlabeled estimates presented as confirmed values propagate into downstream records.

Industry relevance. The repairs are cheap and local: four disclosure rules (state what was actually searched, say when a list was cut, mark derived values as estimated, namespace every identifier). The audit prompt is a few paragraphs and runs on off-the-shelf coding agents; total experiment cost was $74.44. That the prompt produced an already-merged fix in OpenClaw suggests the check is practical for maintainers today, not just a research artifact.

Future Directions

• Isolate disclosure from instruction. Condition B bundles factual disclosures with behavioral instructions, so the measured agent-side effect belongs to the full text. Separating the two would show how much comes from telling the model a fact versus telling it what to do with that fact.
• Test other models and other failure families. The study covers three model families and four defect families in one production system, with cases selected adversarially from known failures. Whether additional families exist — the authors note one encountered during typesetting, where a renderer reported success but truncated a warning that its font library lacked a bold face — and how other models behave remain open.
• Enforce provenance structurally. Wording moved provenance by only +6 pp. Comparing type-level or validation-level enforcement against formatting fixes is the obvious next experiment.
• Evaluate the audit prompt properly. The prompt demonstration is anecdotal and written by people who already knew the defects. A blind, measured evaluation of what such sweeps find and miss would turn the demonstration into a method.

Target Audience

Practitioners who build or maintain agentic systems with tool layers — RAG engineers, agent framework maintainers, and evaluation/observability teams — along with researchers working on LLM-as-judge reliability and grounding. It is also useful to anyone who owns a production assistant whose answers are trusted by end users, because the first actionable step (run the appendix prompt on your own codebase) requires no ML expertise, only a coding agent and a codebase to point it at.

Authors’ abstract

Agents and verifiers often rely on the text a tool prints, and that text can misrepresent what the tool actually did, even when the tool's computation is correct. We saw this happen in a production RAG research system used by the Pilsudski Institute in New York, where tool output formatters misled an agent into giving wrong or incomplete information to users. Each tool did its job correctly, but its output omitted search limits, result caps, or identifier namespaces. We also recreated an unlabeled estimate caught in code review before release. We studied whether these formatters misled agents and verifiers and how much the repaired wording helps. We fed tool results to agents using two formatter versions: one with the key omissions and one with all missing details clearly stated. That difference alone changed what the agents claimed and what the verifiers accepted. Unsupported agent claims dropped substantially for three of the four failure types. In one test, verifiers went from accepting incorrect answers 90 out of 90 times to rejecting them 90 out of 90 times. A control showed the verifiers could catch the errors once they were given the missing facts and asked to judge correctness. We include the audit prompt we used with Codex and Claude Code in an exploratory demonstration, written with prior knowledge of our own system's defects. When pointed at an old version of our own system, it rediscovered the defects that were still live and surfaced twelve more we had not catalogued. When pointed at OpenClaw and Hermes, two of the best-known open-source agentic systems, it found warped formatters in both. We opened PRs to fix both, and the OpenClaw fix has already been merged. Practitioners can run the same prompt on their own systems today.