Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models

AI summary

Overview

Research area: Natural language processing — evaluation of large language models on long-horizon, manual-governed procedural reasoning (benchmark design, retrieval-augmented generation, and agentic tool use).

Technical level: Intermediate. The paper is readable without deep domain expertise in medicine or law, but assumes familiarity with LLM evaluation concepts such as exact match, RAG, ReAct-style agents, and tool-calling harnesses.

Scope: The paper introduces TAM (Tasks over Application Manuals), a benchmark built from two real professional workflows — ICD-10-CM clinical coding and U.S. federal sentencing offense-level calculation — and shows that strong GPT-5 baselines fail badly at executing the underlying rules end to end.

What This Paper Is About

Existing LLM reasoning benchmarks (HotpotQA, MuSiQue, and similar) are short-horizon: they can usually be solved by retrieving and combining a handful of facts. Real professional work looks different. It requires following an authoritative manual of tens of thousands of rules through dozens of interdependent steps, where a single early mistake can silently invalidate everything that follows.

The paper's goal is to isolate and measure that missing capability — faithful execution of a long, cross-referenced written procedure — by building a benchmark where correctness is defined only by the exact final answer the manual produces.

Key Contributions

1. The TAM benchmark. A new evaluation suite built from two real-world domains — ICD-10-CM clinical coding and U.S. federal sentencing guideline calculation — where each instance requires navigating a large, structured manual to produce an exact output.

2. A formal definition of "manual-based reasoning." The authors frame each task as a decision trajectory over a state of accumulated context and partial output, where local plausibility does not imply global validity. This distinguishes TAM from retrieval benchmarks, constraint benchmarks, and interactive planning benchmarks.

3. Reference baselines on a state-of-the-art model. Four representative approaches (single-pass RAG, agentic RAG, ReAct-style tool use, and a LangChain Deep Agents harness) are evaluated on GPT-5 with identical case inputs and manual access.

4. Evidence of a capability gap and a taxonomy of failures. Exact match stays at or below 1% on ICD coding and tops out at 15.5% on sentencing. Manual review of 50 failed ReAct trajectories yields three recurring failure patterns, led by "global inconsistency."

Main Findings

• Exact match on ICD-10-CM coding is at or below 1% across every baseline. The best partial-credit numbers are far from solved: the agent harness reaches 59.9% precision and 50.1% primary-diagnosis accuracy but only 27.4% recall, while agentic RAG reaches 59% precision and 37% recall. No method reliably produces the complete, correctly sequenced code set.

• Exact match on federal sentencing tops out at 15.5%. ReAct-style tool use achieves this best score; single-pass RAG (7.5%), agentic RAG (11%), and the agent harness (14.5%) all trail it.

• Low mean absolute error can mask large real consequences. The agent harness achieves the best legal MAE at 2.34 offense levels, but under the USSG Sentencing Table a three-level shift can move the sentencing range by tens of months, or over five years in high-severity cases.

• Richer scaffolding helps only modestly. Moving from single-pass RAG to agentic RAG to layered tool use improves partial metrics, but exact-match gains on ICD are essentially nonexistent and legal gains are incremental. Retrieval quality is not the bottleneck.

• Global inconsistency is the dominant failure mode. In 38 of 50 reviewed trajectories (22 of 25 ICD, 16 of 25 sentencing), the first visible divergence was committing to the wrong branch — a wrong anchor term, code family, or guideline — and then reasoning coherently within it.

• Incomplete execution appears in 7 of 50 trajectories. The model starts a plausible procedure but stops before verifying global completeness, such as skipping a secondary-diagnosis sweep or never finishing the offense-level determination.

• Missing required elements appeared only in sentencing (5 of 25 cases). These failures involve dropping a mandatory reduction or a late-stage required constraint after entering an otherwise reasonable guideline path.

• One baseline hit a provider-side content-filtering wall. In the agent harness, 35 of 200 sentencing cases terminated early with no response when retrieved statutory text triggered content filtering — a deployment-relevant artifact not seen in other baselines.

Methodology in Plain English

The researchers picked two jobs where professionals genuinely work by following a manual, not by guessing:

• Clinical coding. Given a hospital discharge summary, produce the exact set of ICD-10-CM diagnosis codes. The manual is split into a 1,304-page Alphabetic Index (7,995 main terms), a 1,942-page Tabular List (34,502 leaf codes across 21 chapters), and 120 pages of coding guidelines. Coders loop: propose a term, verify specificity, apply chapter rules, check exclusions, then move on.

• Federal sentencing. Given a case-fact summary, compute the exact total offense level. This means finding the statute in Title-18, mapping it through USSG Appendix A to the right offense guideline, applying offense characteristics, repeating that per count, then aggregating under grouping, cross-reference, and later-adjustment rules.

Building the datasets. For ICD, they started from 254,377 MIMIC-IV admissions, restricted to 2017–2019 discharge notes, filtered out cases touching codes that changed across yearly ICD releases (using official addenda and changelogs), required a usable discharge note, and arrived at 38,332 eligible cases — then sampled 1,000 for evaluation. For sentencing, they pulled candidate dockets from CourtListener (2021–2025), required both a sentencing memorandum and a fact-bearing document, ran an LLM-based sufficiency audit (833 dockets down to 250), and finished with human verification, yielding 200 approved cases.

Testing four approaches. All four see the same case and the same manuals; only the access method changes. Single-pass RAG retrieves once and answers in one shot. Agentic RAG retrieves across multiple rounds. ReAct uses a narrow domain-specific browsing API — eight tools for ICD, seven for sentencing — so the model must decide what to look up next. The agent-harness baseline splits the job across specialized workers (Index search, Tabular verification, guideline review, final audit for ICD; statute identification, guideline mapping, core rules, later adjustments, final check for sentencing), each seeing only the local context it needs.

Scoring. The headline metric is exact match, because each task has one correct final answer. Secondary metrics (precision, recall, primary-diagnosis accuracy, MAE) quantify how much correct structure a model recovers even when the trajectory is ultimately invalid. The authors emphasize that exact match here measures whole-procedure success, not partial progress.

Why This Matters

Impact on research. The paper challenges a widely held inference: that strong scores on multi-hop QA imply reliable multi-step reasoning. TAM shows a specific, measurable gap that retrieval improvements alone will not close, and it reframes the problem as constrained search over partial solutions rather than evidence aggregation. It also gives the community a hard, human-validated target in domains where labeled data are scarce.

Real-world applications:

• Clinical documentation and billing. ICD coding sits directly upstream of reimbursement, audit risk, and population-health analytics. A 1% exact-match rate is unusable in production and defines a clear target for tool-augmented systems.

• Legal and regulatory compliance. Sentencing calculation is the paper's example, but the same structure — statute, guideline lookup, adjustments, grouping — recurs in tax computation, benefits eligibility, permit review, and insurance claims adjudication.

• Enterprise knowledge work. Any organization with a large internal policy manual (HR, finance, procurement, safety) has the same problem: the answer is well-defined and rule-governed, but it lives across thousands of cross-referenced pages.

• Agentic system design. The failure taxonomy — wrong branch entered early, procedure abandoned mid-way, mandatory late-step elements dropped — is directly actionable for anyone building orchestration layers, verification loops, or state checkpoints for long agent runs.

Industry relevance. The authors are from Manulife, an insurance and financial services firm, and both benchmark domains are drawn from regulated industries. The results argue that current RAG-plus-agent stacks are not yet a substitute for manual-following professionals, and that investment should go toward trajectory consistency and backtracking rather than better retrieval alone.

Future Directions

1. State tracking and explicit backtracking. If global inconsistency is the dominant failure mode, methods that maintain a verifiable state of accumulated commitments and constraints — and detect when a path has become invalid — are the most promising direction.

2. Verification and self-audit loops. The agent-harness results suggest decomposition helps slightly. Open question: can a dedicated auditor pass, or a symbolic checker over extracted rules, catch the "missing required element" failures, which appeared exclusively in the late-stage sentencing steps?

3. Symbolic or hybrid execution. Because the manuals are structured and rule-governed, compiling sections into executable constraints or a solver-backed representation may be more reliable than free-form reasoning over retrieved text.

4. Broadening the benchmark and closing the label-noise question. The authors note that coder disagreement caps the achievable ICD exact match, and that MIMIC-IV's public availability means prior training exposure cannot be ruled out (which would only make current results conservative). Extending TAM to more manuals, more models, and non-public data would sharpen the measurement.

Target Audience

This paper is most valuable to LLM evaluation researchers and benchmark designers, especially those working on agentic reasoning, tool use, and long-context retrieval. It is also directly useful to applied AI engineers building agents over enterprise or regulatory documents, and to domain practitioners — clinical coders, health-information managers, and legal professionals — who want to understand where automation currently stands. Product and engineering leaders in regulated industries will find the failure taxonomy and the exact-match results a useful reality check on deployment claims.

Authors’ abstract

Large language models (LLMs) have achieved strong performance on a wide range of natural language tasks, and recent benchmarks suggest that they are increasingly adept at multi-hop reasoning. However, these benchmarks are typically short-horizon, requiring only a small number of retrieval or inference steps, and provide limited evidence of reliability on real-world tasks that involve following manuals spanning hundreds of pages with complex, interdependent guidelines. In this paper, we introduce Tasks over Application Manuals (TAM), a benchmark for evaluating long-horizon procedural reasoning. We construct TAM by curating real-world tasks from two domains: ICD-10-CM clinical coding (mapping medical conditions to diagnostic codes) and U.S. federal sentencing (computing crime sentencing guideline outcomes, specifically offense levels), with human-validated labels. Each task requires following an authoritative manual with tens of thousands of rules and executing a sequence of interdependent steps across different sections to produce an exact answer. We evaluate general-purpose prompting approaches, including retrieval-augmented generation, ReAct-style prompting, and an agent-harness baseline on GPT-5, and find that the best exact-match performance remains extremely low: 1% on ICD-10-CM coding and 15.5% on sentencing tasks. These results show that current benchmarks may overestimate LLM reasoning ability and miss a key challenge: reliably following long, rule-based procedures. The complete TAM data and code are publicly available.