Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction

AI summary

Overview

Research area: Agentic document extraction (natural language processing / applied LLM systems), with an unusual physical-verification component drawn from causal-chamber measurement.

Technical level: Intermediate. The paper is readable without deep ML background, but it assumes familiarity with tool-calling agents, context windows, structured output constraints, and evaluation metrics such as precision/recall and Cohen's kappa.

Scope: The paper argues that fidelity scoring (does an extracted value match the source document?) is insufficient for agentic document extraction, and proposes per-tool dispatch instrumentation plus an independent physical oracle as a benchmark for 37 hand-curated datasheet claims.

What This Paper Is About

Standard evaluations of document-extraction agents ask only whether the extracted value matches the document. That question cannot distinguish a genuine extraction from a correct-looking answer produced without the agent ever reading the document, and it says nothing about whether the document's own claim is physically true. The authors build an agentic benchmark that logs every tool call and pairs a document-internal fidelity verdict with an independent physical reproducibility verdict from a causal chamber, using 25 hand-curated claims across three electronic components plus 12 more on a fourth component.

Key Contributions

1. Per-tool dispatch instrumentation that converts an agent's tool-call record into two instruments: a rule-based failure-attribution classifier and a silent-failure detector that flags fidelity-passing runs which bypass navigation or skip verification. Reported behaviour: zero false positives on 207 clean extractions and by-construction recall of all 50 planted faults.
2. A physical-grounding methodology in which each claim receives a fidelity verdict against the datasheet and a reproducibility verdict against independent chamber measurement. The authors state that physical grounding is necessarily partial and report a "verifiable envelope" of 2 of the 37 claims, with a taxonomy of why the rest are not physically gradable.
3. A cross-provider deployment study across three model families (Claude Sonnet 4.6, GPT-5.1, Qwen3.6-27B) reporting cost, latency, and portability consequences of running such an agent in production, distilled into a checklist of cheap dispatch-level signals.
4. Coverage of documents that outgrow a context window, where the authors report that agentic navigation stays bounded-cost because its cost tracks the pages a claim needs rather than the document's length.

Main Findings

• A silent zero-tool-call failure motivated the work. A model passed the fidelity check with zero tool calls, never opening the datasheet. A structured-output constraint (output_format set to a JSON schema) was realised by the open-weights backend as a guided-decoding token mask that blocked the token beginning a tool call; the model filled the answer schema anyway, with fabricated source text. Fidelity scoring counted this run a success.
• Frontier-model fidelity was perfect and stable. Claude Sonnet 4.6 and GPT-5.1 each passed all 25 claims in every repeat with no engine errors (fidelity 25.0 ± 0.0), under the two-pass freeze. Latency was 76 ± 5 s for Claude Sonnet 4.6 and 236 ± 78 s for GPT-5.1.
• One model showed large run-to-run instability. Qwen3.6-27B scored 19.0 ± 4.0 (passing 23, 19, and 15 claims across three repeats), with per-repeat engine errors of 1/6/10. Only 13 of 25 claims kept the same verdict across repeats, against 25 of 25 for the frontier models. With reasoning disabled, the same model passed 25/24/24 of the same claims with a single engine error. The authors attribute this to a documented vLLM/reasoning-mode interaction (a deployment-stack artefact, not a capability gap).
• The silent-failure detector recovers all planted faults. Two fixed rules read only which tools were called, never the extracted value: tool-bypass (no navigation call) and verification-skipped (navigation but no cross-check). Fidelity-only scoring flagged none of the 50 planted faults; the detector recovered 25/25 tool-bypass and 25/25 verification-skipped faults on Claude Sonnet 4.6, and under the same injection recovered 32/32 on GPT-5.1 and 21/21 on Qwen3.6-27B.
• Zero false positives, but the bound is weaker than it looks. The detector raised no flag on 207 clean fidelity-passing cells spanning three model families. Those cells come from only 74 claim-by-model groups (25 × 3, less one Qwen claim that never passed), so the authors compute a one-sided 95% false-positive bound of 4.0% per group or about 11% per claim, not the 1.4% that treating all 207 cells as independent would give.
• Recall is by construction. Because the planted faults withhold exactly the tools the rules check, the authors state that recall is structural and limited to rule-aligned faults, not silent failures in general.
• The rules miss wrong-document grounding. In two probe arms that left navigation, cross-check and read success intact but corrupted grounding (empty content from every call; every call served from a decoy datasheet), across 100 runs on Claude and GPT-5.1, 8 answered and none was right. Qwen3.6-27B was excluded because it errors on 35 of its own 50 probe runs under forced tool choice. Six of the 8 were faithful readings of the decoy; only 2 were memory answers.
• The attribution classifier is directional only. Validated against 30 traces stratified across models (10 per model), with 3 abstentions leaving 27 adjudicated cells, it agreed on 24 of 27 (89%, Cohen's κ = 0.61). All three disagreements were leniency, with no over-flag observed. Zero over-flags on 21 human-clean cells is consistent with a true over-flag rate up to about 13% (one-sided 95% bound), and sensitivity was 3 of 6.
• Qwen's flag count is dominated by unlabelable steps. The 41 non-ok slots reported for Qwen3.6-27B are counted per call across 7 of its 25 agentic cells: 21 are unclassified, 14 are engine_error, and only 6 are rubric labels proper (five verification-skipped, one condition-omission) in 6 cells. Neither frontier model carries a non-ok slot on any agentic cell.
• The physical oracle confirms almost nothing. With current chamber instrumentation only 2 of the 25 frozen claims reach a definitive pass and none a definitive fail; the other 23 are inconclusive. Over the combined corpus the envelope is 2 of 37. The taxonomy of the 23: five engagement-only, five absolute-accuracy, twelve dependent on a decisive condition the chamber cannot match, and one resolution-limited (approximately 0.080 hPa cross-sensor uncertainty against a ±0.06 hPa tolerance).
• The reproducibility verdict flips only at measurement uncertainty. Holding the real cross-sensor DPS310 measurement (945.285 ± 0.080 hPa) fixed and shrinking the datasheet's stated bound through it, fidelity passed throughout while the reproducibility verdict walked pass → inconclusive → fail exactly at the 0.080 hPa uncertainty band.
• The tool layer buys portability and observability, not accuracy. On this corpus baseline and agentic agreed on fidelity in 68 of 75 cells (25 claims × 3 models). The reproducibility verdict is engine-invariant by construction, differing on one Qwen cell only because that cell was lost to an engine error. Cost premium at list prices was 1.2× for Claude, 1.8× for GPT-5.1, and 3.9× for Qwen (which reflects a cheap page-image baseline rather than an expensive agent). The stated operational rule: default to single-pass, switch to agentic only when a document exceeds the context window or a provider's page cap. Here every datasheet fits one window (at most 65 pages), but on a 397-page PMIC datasheet agentic navigation stays bounded-cost.
• Six portability failures appeared on one inference gateway, two of them silent. The silent ones were the guided-decoding tool masking and a GPT-5.1 configuration that dropped its reasoning trace without error. The two GPT-5.1 failures were resolved through the provider-native Responses API; the other four are backend limitations.
• Fidelity independence was made structural. Before the two-pass freeze, 80% of Claude cells and 100% of GPT-5.1 cells called a chamber tool before submitting their extraction (92% in Qwen's reported run, which predates the freeze); under the freeze, none can, and a sample of Qwen two-pass traces showed 0% pre-submit chamber-tool calls.
• The off-corpus fourth component is a null result. On 12 claims from a 20-page A4988 stepper-driver datasheet, both frontier models passed 12 of 12 under exact-value matching, with no engine errors and no detector flags. The authors explicitly present this as a null result on an already-exposed document, not generalisation, and note that zero flags over 12 claims is a zero-event count, not a precision result.
• Benchmark and production detectors differ. The benchmark evaluates verification-skipped, a rule production does not ship; production ships a tool-read-failure rule and a source-grounding pass that the benchmark does not evaluate, so a clean flag sheet from the pilot carries no information about the grounding check. In live use none of the flags has fired: instrumentation covers 17 of 126 extraction jobs, of which 14 are large-PDF and therefore eligible to raise a flag, and the zero-tool-call run was found by a human reading a trace, not by a rule.
• A late methodology audit found the prompt leaked the graded answer. Two channels were closed and the full matrix re-run; every result in the results section is post-audit. The authors nevertheless note that most of the design predates the fix, and the detector rules are the exception, first authored four days after the fix.

Methodology in Plain English

The authors build a narrow, claim-level benchmark rather than an open-ended document-QA task. The corpus pairs manufacturer datasheets with a causal chamber that can independently measure a subset of the quantities those datasheets state. Each of 25 hand-curated quantitative claims (11 for the DPS310 barometer, 9 for the Si115x light sensor, 5 for the ACS70331 current sensor) is scored twice: fidelity — did the agent recover the value, units and stated operating conditions? — and reproducibility — does the chamber agree within a spec-appropriate tolerance? A chamber-side program computes the reproducibility verdict from the curated claim spec and the physical measurement, neither of which comes from the agent, so that verdict is identical across runs. Disagreements smaller than the combined uncertainty (cross-sensor measurement uncertainty and the claim's own spec tolerance) are treated as inconclusive rather than failures.

The agent runs in the authors' own loop of a few hundred lines, mostly instrumentation. It works in two phases under what the authors call a two-pass freeze: the extraction phase exposes only document-side tools (build_datasheet, get_section_text, search_text, extract_table_markdown, inspect_page) plus submit_extraction, which freezes the value; submitting reveals five chamber-side tools plus submit_chamber_outcome for the agent's own ungraded prediction. Every call is recorded with inputs, outputs and timing — the dispatch record.

The failure-attribution rubric is fixed in advance: tool-output errors are charged to the returning tool; tool-selection, condition-omission and verification-skipped errors are charged to reasoning. The silent-failure detector is a separate pair of predicates over tool presence in a single already-fidelity-passing extraction. Because the rules read only dispatch, not values, the authors test them by fault injection: for each of 25 claims they make two Claude Sonnet 4.6 runs with a fidelity-passing reference value pinned, one withholding navigation tools and one withholding cross-check tools, producing 50 runs that are silent failures by construction. They then check how often the rules fire on the 207 clean cells of ordinary runs.

Three models from independently trained families run the same harness logic under a uniform 30-turn budget and three repeats per claim per model; sampling is left at provider defaults, with only loop logic, reasoning effort (medium for Claude and GPT-5.1) and turn budget pinned. Transport differs by provider, so the authors describe the comparison as one of deployed stacks rather than weights alone. A single-pass baseline that sends the entire datasheet in one call runs end-to-end on all three models.

Why This Matters

For research, the paper's central argument is that document extraction needs an outcome axis (fidelity) and a process axis (dispatch). It shows that a fidelity-only benchmark silently scores a fluent, fabricated answer as a success, and it demonstrates a class of "corrupt success" that current scoring cannot see. It also makes a case for external oracles in document understanding that goes beyond LLM-as-judge validation, while being candid that the physical oracle here covers only 2 of 37 claims.

Real-world applications:

• Semiconductor and electronics engineering: automated parameter extraction from component datasheets, where the paper's 25 claims come from a barometer, a light sensor and a current sensor.
• Enterprise agent deployment on an inference gateway: the portability checklist targets provider-routed agents, and the authors' own service is described as an internal pilot live since early 2026 behind a production inference gateway, used on demand by engineers across multiple divisions on the company's own and competitors' datasheets (126 extraction jobs to date).
• Review routing rather than automatic release gating: the dispatch-level signals are positioned as warnings that route a suspect run to human review, not as release gates.
• Long-document pipelines: the cost argument applies to any workflow where a document exceeds a context window or a provider's page cap, such as the 397-page PMIC datasheet case.

Industry relevance: the paper is an operational report from a deployment, not only a benchmark paper. It reports cost multipliers at list prices (1.2×, 1.8×, 3.9×), latency, six concrete portability failures on one gateway, and a mid-study backend change in which a hosted backend regressed tool-calling with a 3.7 kB dereferenced extraction-result schema while each call still returned HTTP 200. Its practical recommendation is that provider portability is not settled by a working SDK call and often fails downstream and silently.

Future Directions

• Measure detection power against runs that use their tools and still answer wrongly. The authors state this is unmeasured: both rules are silent on an agent that calls its tools, reads them successfully, satisfies the predicate and still answers from the wrong content. They demonstrated the class exists but did not separate it from a clean run.
• Expand the verifiable envelope with better chamber instrumentation. Only 2 of 37 claims are physically gradable, and none of the fourth component's 12 can be staged at all. The taxonomy of inconclusive claims is described as a requirements list for instrumentation a production test laboratory already operates.
• Establish a human baseline. No comparison against an applications engineer reading the same datasheet is reported; the authors call the omission deliberate and say how hard these claims are for a competent human remains open.
• Calibrate the failure-attribution classifier on a larger, multi-annotator sample. The current validation rests on a single blind annotator over 27 adjudicated cells, with sensitivity of 3 of 6 and no demonstration that the classifier is calibrated.
• Close the gap between benchmark and production detectors. The benchmark evaluates verification-skipped, which production does not ship, while production ships a tool-read-failure rule and a source-grounding pass the benchmark does not evaluate; the authors state results do not transfer between them.

Target Audience

Engineers and researchers building or evaluating tool-using document agents, particularly those deploying behind an inference gateway across multiple model providers. It is also relevant to evaluation and reliability practitioners who need process-level diagnostics rather than single accuracy scores, to teams working on datasheet or specification extraction, and to researchers interested in external, non-LLM oracles for validating document claims. Readers looking for state-of-the-art extraction accuracy will not find it here; the paper's value is in instrumentation, failure diagnosis and a deliberately partial physical ground truth.

Authors’ abstract

One model passed our fidelity check without ever opening the datasheet. We found it while qualifying models for an internal extraction service: a structured-output constraint had silently disabled tool use, and the model answered anyway, with fabricated source text. Only the per-tool trace exposed it. Fidelity -- whether an extracted value matches the source -- is the standard measure for agentic document extraction, and it scores that run a success. We therefore log every tool call in an agentic benchmark of 25 hand-curated claims over three components, with 12 more on a fourth, 37 in all. From that dispatch record we build two instruments: a rule-based failure-attribution classifier, and a silent-failure detector whose two rules check only which tools were called, never the extracted value. The detector raises no flag on 207 clean fidelity-passing extractions across three model families, and recovers all 50 planted faults that withhold exactly the tools its rules check. The two results are not symmetric: the first bounds the false-positive rate, the second is recall by construction, and detection power against runs that call their tools and still answer wrongly is unmeasured. A second, independent oracle, a causal chamber that tests whether the datasheet's claims hold under physical measurement, is intentionally partial: it confirms only what the apparatus can exercise, a verifiable envelope of 2 of those 37 claims, and we give a taxonomy of why the rest are not physically gradable. Under a controlled perturbation, fidelity passes throughout while the chamber verdict flips exactly at the measurement uncertainty. Across three deployed model stacks (one destabilised by its serving stack, not by any capability gap) the tool layer buys portability and observability rather than accuracy, and earns its premium only once a document outgrows the context window.