OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning

AI summary

Overview

Research area: AI evaluation benchmarks; enterprise document-grounded reasoning by LLM agents (retrieval-augmented and agentic systems).

Technical level: Intermediate. The paper is a benchmark and systems-evaluation paper rather than a new modeling technique; it assumes familiarity with agent frameworks, retrieval, and document parsing, but the results are readable without deep ML background.

Scope: The paper introduces OfficeQA Pro, a 133-question benchmark built on roughly a century of U.S. Treasury Bulletins, and measures how frontier LLMs and agent frameworks perform on end-to-end, verifiable, multi-document analytical reasoning.

What This Paper Is About

Existing frontier benchmarks such as Humanity's Last Exam and ARC-AGI-2 test abstract or academic reasoning, and closed-world benchmarks such as GDPval provide all needed context directly in the prompt (fewer than two artifacts on average), leaving untested the real enterprise challenge of retrieving and reconciling evidence from large, messy document collections. The authors argue that economically valuable tasks instead require what they call Grounded Reasoning: locating the right documents and sections, parsing them faithfully, and performing quantitative analysis to produce a verifiable answer. OfficeQA Pro is designed to proxy that real-world workflow with automatically gradable questions grounded in a century of U.S. Treasury Bulletins.

Key Contributions

1. A new verifiable enterprise benchmark. OfficeQA Pro consists of 133 questions over a corpus of U.S. Treasury Bulletins spanning nearly 100 years, comprising 89,000 pages and over 26 million numerical values, covering unstructured prose, complex nested tables, figures, and charts. A companion set of 113 easier questions (OfficeQA-Full = 133 Pro + 113 Easy) is released to support iteration and evaluation of non-frontier systems.

2. A creation and verification pipeline with automatic grading. Questions were authored with data annotation partners SuperAnnotate and Turing, reviewed with USAFacts for analyst relevance, filtered to remove questions answerable from parametric knowledge alone, and verified across multiple annotator rounds and two rounds of end-to-end quality assurance using AI agents to surface discrepancies. Each question has a single unambiguous answer, and 99% of answers are numerical, enabling deterministic exact-match evaluation with configurable allowable absolute relative error.

3. Controlled comparison of document parsing and agent design. The paper benchmarks three parsers — Docling, unstructured.io, and Databricks' ai_parse_document — and runs ablations on model selection, table representation, retrieval strategy, and test-time scaling.

4. An empirical failure-mode taxonomy. The paper documents remaining failure modes including temporal revision verification, parsing faithfulness, visual understanding, and analytical reasoning errors.

Main Findings

• Frontier models fail without the corpus. Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro Preview achieve less than 5% accuracy on OfficeQA Pro when relying on parametric knowledge, and less than 12% with additional access to the web. In the prompt-only setting, every model scored under 3% accuracy at 0.0% allowable absolute relative error, rising to 17–24% at 5% allowable error, with models refusing to answer 10–15% of questions.

• Web search alone is insufficient. Web search substantially improved performance, with GPT-5.4 reaching 11.3% accuracy compared to less than 1% in the prompt-only setting. However, Opus 4.6 failed to produce a final answer in approximately 80% of cases (Gemini 3.1 Pro Preview in 15%, GPT-5.4 in 2%), exhausting generation tokens. Models referenced FRASER Data API in roughly 70% of traces for Opus 4.6, 50% for GPT-5.4, and 5% for Gemini 3.1 Pro Preview, yet still applied data incorrectly.

• Frontier agents score 34.1% on average. Each agent framework (Claude Agent SDK, OpenAI Codex CLI, Gemini CLI) was run with its provider's latest flagship model and none surpassed 50% accuracy. On the full corpus with raw PDFs, Claude Opus 4.6 scored 48.1%, GPT-5.4 High scored 36.1%, and Gemini 3.1 Pro Preview scored 18.1%.

• Parsing quality is a major bottleneck. Providing agents with a structured document representation produced by Databricks' ai_parse_document yields a 16.1% average relative performance gain across agents. Agent correctness improved by 6.0–20.3 absolute percentage points (12.5–62.4% relative) over PDF baselines, and agents were 4–9x faster. In a separate LLM oracle-page experiment, parsed pages provided a relative gain of +50.2% on average at 0.0% allowable error: GPT-5.4 improved from 57.1% to 65.4% (+8.3pp), Claude Opus 4.6 from 36.1% to 57.1% (+21.0pp), and Gemini 3.1 Pro Preview from 52.6% to 56.4% (+3.8pp).

• Best agent configuration reaches 66.9%. With oracle Databricks-parsed pages, Claude Opus 4.6 achieved 66.92%, GPT-5.4 achieved 65.41%, and Gemini 3.1 Pro Preview achieved 46.62%. Compared to full-corpus PDF runs, oracle pages improved accuracy by 13–21 absolute percentage points, reduced average latency by approximately 76%, and cut costs by 74–88%. The GPT-5.4 agent was the most cost-efficient at $0.33 per sample, approximately 82% cheaper than its full-corpus PDF configuration.

• Raw PDF handling drives cost and latency. The Claude Opus 4.6 agent made approximately 11k tool calls, 65% using Claude Code's native Read tool, while Codex with GPT-5.4 operated primarily through Bash with roughly 90% of tool calls using OCR and PDF CLI tools (a tesseract + pdftoppm + Pillow pipeline). The GPT-5.4 agent was still approximately 2.4x faster than the Claude agent because it batched OCR extraction into shell one-liners.

• Parser choice changes accuracy by up to 22 points. Across the custom agent with GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro Preview, ai_parse_document achieved an average accuracy of 50.4% (average cost $5.29 per sample), compared to Docling at 38.4% ($8.18) and unstructured.io at 31.1%; full-corpus parsing cost $178 for ai_parse_document and $2,670 for unstructured.io.

• Model, table format, retrieval, and test-time scaling ablations. Across 10 frontier LLMs, Claude Opus 4.6 achieved the highest accuracy (57.1%). HTML table representations generally improved correctness by a slight margin over hierarchical Markdown. Combining file search with contextual-embedding vector search gave the best quality–cost tradeoff. Plurality voting produced modest but consistent gains, saturating for higher-performing models.

• Baseline agents suffer parsing-related failures. Agents using the original PDFs exhibited a 40–50% failure rate attributable to parsing errors such as misread numbers, corrupted text, and misaligned tables. GPT-5.4 performed approximately 15% worse on documents from the 1930s–1970s than on the more digitally native PDFs from the 1980s–2020s, while Opus 4.6 and Gemini 3.1 Pro Preview showed a smaller approximately 5% gap.

• Latency remains a challenge. Answering a question over the full PDF corpus required an average of 23.6 minutes across frontier agents, and 3.9 minutes even with state-of-the-art document parsing.

• Humans are outperformed by agents. Despite agent struggles, AI agents consistently outperformed human annotators in both speed and accuracy (Appendix C).

Methodology in Plain English

The authors built a benchmark rather than a model. They selected U.S. Treasury Bulletins — published monthly from 1939–1982 and quarterly thereafter, each 100–200 pages — as a corpus that mimics enterprise archives: mixed prose and tables, multi-level nested table hierarchies, changing units, heavy footnotes, statistics that get revised across issues, and a transition from scanned documents to digital-native PDFs in 1996. Embedded text layers were removed to ensure document interpretability was fairly assessed.

Questions were written by annotators who were given sample questions and told to ground every question in the corpus and to be unambiguous (for example, distinguishing fiscal from calendar year, or real from nominal dollars). Each question was then answered independently by a new annotator using the same PDF pages; disagreements went to a third annotator. Two rounds of end-to-end quality assurance followed, using AI agents to generate alternative answers so reviewers could distinguish agent failure, question ambiguity, or incorrect ground truth.

To ensure the benchmark tests retrieval rather than memorization, questions solvable by the frontier models available at construction time (Claude Opus 4.5 and GPT-5.1) were filtered out, leaving 246 questions. Questions both of the strongest agents at the time (powered by Claude Opus 4.5 and GPT-5.1 with Databricks parsed documents) answered correctly were labeled Easy and moved to OfficeQA-Full, leaving 133 in OfficeQA Pro.

Evaluation is exact match, with accuracy additionally reported under allowable absolute relative error thresholds (5.0%, 1.0%, 0.1%, 0.0%; 0.0% is the default). The metric normalizes punctuation, symbols, and abbreviations, and multi-part answers require overlap in both text and numerical components.

For experiments, the authors compared single-call LLM baselines (prompt only; web search; oracle PDF pages plus web search; oracle parsed pages plus web search) against agent frameworks from OpenAI, Anthropic, and Google, run non-interactively in isolated environments with a shared system prompt. A custom agent — up to 200 steps per question, a sliding window of the 30 most recent messages, with web search (DuckDuckGo), a stateful sandboxed Python REPL, and fs_search/fs_read file tools — was used for controlled ablations. Tool outputs were truncated at 25k characters, and runs allowed up to 30 retries.

Why This Matters

Impact on research. OfficeQA Pro reframes evaluation around Grounded Reasoning — parsing, retrieval, and computation combined — and shows that strong abstract-reasoning scores do not transfer to economically valuable document workflows. It also isolates document parsing as an "often underappreciated bottleneck": parser choice alone moved agent accuracy by up to 22 points, and parsing errors propagate into retrieval and downstream computation.

Real-world applications.

• Financial analysis over regulatory filings and statistical bulletins, including inflation adjustment, longitudinal reconciliation of revised figures, and regression on historical series.
• Enterprise search and retrieval across heterogeneous, multi-decade document archives with inconsistent layouts and changing reporting conventions.
• Automated compliance and audit workflows that require exact numeric answers and a single unambiguous ground truth rather than plausible prose.
• Internal analytics assistants that must select the right definition of a statistic (for example, fiscal versus calendar year, or correct CPI series) rather than a near-miss value.

Industry relevance. The findings speak directly to production deployment: document parsing choices materially affect accuracy, latency, and cost; agent costs ranged from $0.23 to $6.21 per question, and full-corpus PDF question answering averaged 23.6 minutes — both problematic for enterprise use. The paper also highlights a security-adjacent concern that agents cannot reliably tell when the answer is already public: models failed to reference the publicly available dataset on GitHub containing the exact questions, answers, and source documents. Conversely, agents already outperform human annotators in speed and accuracy, indicating where automation is immediately viable.

Future Directions

• Revision-aware search. Agents frequently converge on the first numerically plausible value rather than the most recently published figure; enforcing revision checks caused repeated search iterations, context saturation, and recursive retrieval loops. More exhaustive, revision-aware strategies are needed.
• Stronger visual reasoning. Figures were omitted from the parsed representations, and agents lacked the resolution to interpret dense financial charts even when given the image directly. Encoding strategies for visual figures are left to future work.
• Analytical reliability. Remaining errors include premature rounding, misapplied formulas (such as sample rather than population variance), and computing internally instead of delegating to scripts.
• Latency and generalization. Beyond correctness, latency is described as major and underexplored. The authors plan a held-out test set to measure agent generalization, and call for benchmarks spanning more domains and modalities (images, databases, documents) reflecting real production workflows.

Target Audience

Researchers and engineers building or evaluating agents for enterprise document workflows; benchmark designers interested in verifiable, automatically gradable tasks over large heterogeneous corpora; and practitioners choosing document parsers, retrieval strategies, and agent frameworks for retrieval-heavy financial or analytical applications. It is also relevant to teams studying retrieval-augmented generation, long-context reasoning, and test-time scaling, and to anyone assessing whether frontier model benchmark scores translate into economic value.

Authors’ abstract

We introduce OfficeQA Pro, a benchmark for evaluating AI agents on grounded, multi-document reasoning over a large and heterogeneous document corpus. The corpus consists of U.S. Treasury Bulletins spanning nearly 100 years, comprising 89,000 pages and over 26 million numerical values. OfficeQA Pro consists of 133 questions that require precise document parsing, retrieval, and analytical reasoning across both unstructured text and tabular data. Frontier LLMs including Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro Preview achieve less than 5% accuracy on OfficeQA Pro when relying on parametric knowledge, and less than 12% with additional access to the web. When provided directly with the document corpus, frontier agents still struggle on over half of questions, scoring 34.1% on average. We find that providing agents with a structured document representation produced by Databricks' ai_parse_document yields a 16.1% average relative performance gain across agents. We conduct additional ablations to study the effects of model selection, table representation, retrieval strategy, and test-time scaling on performance. Despite these improvements, significant headroom remains before agents can be considered reliable at enterprise-grade grounded reasoning.