Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows

AI summary

Overview

Research area: AI agent evaluation, specifically benchmarking frontier LLM agents on real-world enterprise finance and accounting (F&A) workflows centered on spreadsheets.

Technical level: Advanced (assumes familiarity with LLM agents, spreadsheet semantics, and benchmark design), though the core findings are accessible to practitioners.

Scope: Introduces Finch, a 172-workflow, 1,710-spreadsheet benchmark built from authentic enterprise artifacts spanning 2000–2025, and shows that the strongest frontier agents solve fewer than 40% of realistic F&A workflows.

What This Paper Is About

Existing benchmarks for AI agents on spreadsheets and data tasks rely on clean, isolated tables and single-step questions, which bear little resemblance to how finance and accounting professionals actually work. Real F&A work is messy, long-horizon, and knowledge-intensive: analysts navigate dozens of interlinked workbooks, PDFs, and images, reconstruct implicit business logic from formulas, and chain together data entry, retrieval, modeling, validation, and reporting. Finch is built to measure whether today's frontier agents can handle that reality, using workflows mined from genuine enterprise environments rather than synthetic tasks.

Key Contributions

1. FinWorkBench (Finch), a benchmark of 172 composite enterprise-grade F&A workflows containing 384 tasks, 1,710 spreadsheets (27 million cells), plus PDFs, images, Word documents, JSON, CSV, and Markdown — sourced from the Enron corpus (15,000 files, 500,000 emails), EUSES, financial institutions, the World Bank, Adidas, and the Canadian and British governments.

2. A novel workflow construction pipeline that derives tasks from three authentic channels: enterprise email threads where collaborators describe goals, LLM-assisted differencing of versioned spreadsheets that surfaces implicit analyst intent, and high-quality final deliverables from organizations and governments. This required over 700 hours of expert annotation and iterative quality control (40% of workflows revised at least once; 20+ went through three or more rounds).

3. A dual evaluation framework combining expert human evaluation (the gold standard, binary pass/fail based on whether the instruction was satisfactorily fulfilled) with a scalable multimodal LLM-as-judge pipeline that uses structured diffs, compact cell snapshots, and rendered screenshots to assess completeness, correctness, over-edit avoidance, and readability.

4. A systematic evaluation and error analysis of frontier systems — GPT 5.1 Pro, Claude Sonnet/Opus 4.5, Gemini 3 Pro, Grok 4, and Qwen 3 Max — revealing how real-world complexity compounds failure in ways single-factor benchmarks miss.

Main Findings

• Frontier agents fail most workflows. GPT 5.1 Pro, the strongest system tested, passes only 38.4% of workflows under human evaluation (41.9% under automated evaluation) while spending an average of 16.8 minutes per workflow. Claude Sonnet 4.5 passes 25.0%. No model exceeds 50%.

• Long-horizon composition is the dominant bottleneck. GPT 5.1 Pro drops from 48.6% on single-task workflows to 42.4% (two tasks), 33.3% (three tasks), and 5.6% (four or more tasks); aggregated, it falls from 44.3% on workflows with ≤2 tasks to 23.5% on those with >2 tasks. Claude Sonnet 4.5 drops from 30.3% to 11.8% under the same split. Errors accumulate across steps.

• Real workflows require heavy multi-step interaction. A 20-workflow case study using Claude Coworker (Opus 4.6) found a median of 16 tool calls per workflow (range 6–107). Excluding two web-search-intensive outliers, tasks still require 6–25 tool calls (median 14). Tool-call overhead correlates more with intrinsic business-logic depth than with task count.

• Task difficulty varies sharply by type. Data Entry/Import and Structuring/Formatting are consistently hardest, because Finch spreadsheets have irregular layouts, multi-level headers, merged cells, and nested subtotals. Translation — a task LLMs typically handle well — performs poorly, because financial tables lose critical structural cues (header hierarchies, row/column alignment) when translated.

• Multimodal artifacts compound difficulty. On the 20 workflows involving PDFs, Word documents, or images, GPT 5.1 Pro's pass rate drops to 35.0%, below its 38.4% overall rate.

• Correctness, not completeness, is the primary rubric bottleneck. Under automated rubric scoring, Claude Sonnet 4.5 achieves 55.8% completeness but only 36.0% correctness; Opus 4.5 reaches 69.8% completeness and 45.3% correctness. Models often do roughly the right thing but get the numbers or formulas wrong.

• Error analysis isolates five failure categories (Claude Sonnet 4.5): 35% formula reasoning errors (failing to reconstruct latent business logic encoded in formulas), 25% data retrieval errors (wrong cross-sheet or intra-table ranges), 25% code generation errors (invalid scripts or layout mismatches), 10% task misunderstanding (missing implicit context in artifacts), and 5% data rendering errors (wrong formatting, charts, or deliverable type — e.g., creating a new workbook instead of editing the original).

• Automated evaluation is a useful but imperfect proxy. The GPT-5-mini judge agrees with human labels on 82.1% of workflows for GPT 5.1 Pro and 90.2% for Claude Sonnet 4.5, with high recall (83.3%/88.4%) but weaker precision (73.6%/76.0%), suggesting automated scores overestimate accuracy by a few percentage points. Interestingly, the LLM judge sometimes catches errors humans miss — for example, formulas silently replaced with static values.

• API-based agents lag product-side agents. GPT 5.1 via the adapted SpreadsheetBench-style single-call API framework reaches 32.0%, versus 41.9% automated for GPT 5.1 Pro, because single-call setups cannot iteratively inspect, execute, and self-correct.

Methodology in Plain English

The researchers started from a philosophical stance: observe professional work in the wild before formalizing it. Rather than inventing tasks, they mined three authentic sources. First, they used GPT-5 to scan Enron email threads for messages that stated a business goal and referenced attached spreadsheets, then had experts turn those conversations into clean workflow instructions. Second, they collected families of versioned workbooks and used an LLM differencing procedure to infer what changes were made between versions — surfacing the implicit goals behind an analyst's edits — which experts then validated and refined. Third, they took high-quality final deliverables from investment firms, the World Bank, and national governments and reverse-engineered realistic instructions, inputs, and reference outputs from them.

For evaluation, they ran every workflow through product-side agents (ChatGPT 5.1 Pro, Claude Sonnet/Opus 4.5) and API-based models (GPT 5.1, Claude Sonnet 4.5, Gemini 3 Pro, Grok 4, Qwen 3 Max). API models were driven by an extended SpreadsheetBench framework with a semantic-rich tuple encoding that preserves each cell's address, value, type, and formula, plus multimodal handling for images and PDFs. Every workflow was judged both by human experts (binary pass/fail) and by an LLM-as-judge pipeline that compares structured diffs and rendered screenshots of the input, reference, and model output, scoring on four rubric dimensions. The authors then manually inspected failed trajectories to categorize the primary cause of each failure.

Why This Matters

Impact on research. Finch shifts agent benchmarking away from clean, single-table tasks toward authentic, messy, long-horizon enterprise work. It provides both a harder evaluation target and diagnostic structure — task types, business verticals, rubric dimensions, and error categories — that lets researchers pinpoint why agents fail rather than just how often. The finding that generic capabilities degrade sharply under combined messiness, heterogeneity, and domain knowledge suggests that current agent architectures are not simply "almost there."

Real-world applications:

• Financial modeling and valuation — calibrating DCF models, scenario and sensitivity analysis, and timing assumptions encoded in spreadsheet formulas.
• Budgeting and planning — consolidating department-level spreadsheets with inconsistent formats into unified plans and variance reports.
• Audit and reconciliation — cross-file validation, consistency checking, and accounts payable/receivable reconciliation.
• Regulatory and multilingual reporting — translating bilingual government reports while preserving structure, and generating data-driven summaries from World Bank or sovereign debt publications.

Industry relevance. Enterprise finance and accounting is a high-stakes, labor-intensive domain where spreadsheet manipulation is the daily substrate of work. The benchmark shows that current commercial agents — ChatGPT, Claude, and their peers — cannot yet be trusted with end-to-end F&A workflows, which matters for anyone considering deployment in controllership, FP&A, audit, or treasury functions. It also highlights that product-side agents' interactive affordances (iterative inspection, self-correction) provide real value over single-call API designs, informing how enterprise agent frameworks should be architected.

Future Directions

1. Agentic evaluation. The authors explicitly flag flexible, agent-driven evaluation as promising future work, since the current LLM-as-judge pipeline shows inter-rubric confusion and imperfect precision when scoring four dimensions in a single call.

2. Closing the product-vs-API gap. Designing API-based agent frameworks with iterative execution, tool feedback, and self-correction — rather than single-call code generation — is identified as an important direction for enterprise-grade F&A agents.

3. Improving formula-centric reasoning. Since formula reasoning errors account for the largest share of failures, future work could focus on encodings and training that force models to attend to formulas as latent business logic rather than treating displayed values as ground truth.

4. Robust multimodal and structural grounding. Handling PDFs, images, charts, pivot tables, merged cells, and multi-level headers without cascading off-by-one errors remains an open problem, particularly when spreadsheet logic is applied in batch across many irregular sheets.

Target Audience

Researchers and engineers building or evaluating LLM agents for spreadsheet, document, and enterprise automation; practitioners in finance, accounting, and FP&A assessing whether AI agents can be deployed on real work; benchmark designers interested in authentic, expert-annotated task construction; and product teams at AI companies seeking a rigorous diagnostic of where frontier agent capabilities break down under real-world complexity.

Authors’ abstract

We introduce FinWorkBench (a.k.a. Finch) for evaluating AI agents on real-world, enterprise-grade finance and accounting workflows that interleave data entry, structuring, formatting, web search, cross-file retrieval, calculation, modeling, validation, translation, visualization, and reporting. Finch is sourced from authentic enterprise workspaces from Enron (15,000 files and 500,000 emails) and other financial institutions, covering the period 2000--2025 and preserving the in-the-wild messiness of multimodal artifacts such as tables and charts across diverse domains including budgeting, trading, asset management, and operational management. We propose a workflow construction process that combines LLM-assisted mining of workflows from authentic enterprise environments with expert annotation: (1) LLM-assisted, expert-verified derivation of workflows from real-world email threads and spreadsheet version histories, and (2) meticulous annotation requiring over 700 hours of expert effort. This yields 172 composite workflows with 384 tasks, involving 1,710 spreadsheets with 27 million cells, along with PDFs and other artifacts, capturing the intrinsically messy, long-horizon, knowledge-intensive, and collaborative nature of real-world enterprise work. We conduct both human and automated evaluations of frontier AI systems, including GPT-5.1, Claude Sonnet 4.5, Claude Opus 4.5, Gemini 3 Pro, Grok 4, and Qwen 3 Max. Under human evaluation, GPT-5.1 Pro spends an average of 16.8 minutes per workflow yet passes only 38.4% of workflows. Comprehensive case studies further surface the challenges that real-world enterprise workflows pose for AI agents.