APEX-Accounting

AI summary

Overview

• Research area: Natural language processing, specifically agent evaluation and benchmark construction for real-world professional knowledge work (accounting and bookkeeping).
• Technical level: Advanced. The paper assumes familiarity with agent harnesses, tool-use architectures, LM-as-a-judge evaluation, and statistical correction procedures.
• Scope: The paper introduces and reports results from APEX-Accounting, a private benchmark of 160 expert-authored accounting tasks across 10 synthetic company worlds, evaluated across 9 frontier models using two agent harnesses.

What This Paper Is About

Frontier AI models can pass professional accounting certification exams, yet it is unclear whether they can perform the day-to-day, document-heavy work of accountants, such as reconciling accounts, posting journal entries, and carrying multi-step analyses through to a month-end close. APEX-Accounting, built by Mercor in partnership with Ramp, was created to measure that capability directly rather than through exam proxies. The goal is to determine how close frontier agents are to doing real accounting work reliably enough to be deployed unsupervised.

Key Contributions

1. A new expert-authored benchmark. APEX-Accounting contains 160 held-out tasks across 10 synthetically generated company worlds (16 tasks per world), created by 42 accounting experts, with a public dev set of 1 world and 10 tasks including prompts, rubrics, and task files.
2. A realistic world and file environment. Each world is a self-contained company frozen at a specific month-end close, follows U.S. GAAP accrual-basis accounting, and contains spreadsheets, PDFs, docx files, mail, and accounting-software tooling. Worlds drew on an average of 73.1 unique required input files.
3. A rigorous grading pipeline. Every task has a golden response and a rubric of binary, outcome-based criteria (13.7 criteria per task on average). The judge, DeepSeek-v4-Flash, was validated against 1,687 majority-vote expert labels from 120 trajectories, reaching 97.1% accuracy (F1 = 0.970).
4. A failure taxonomy and diagnostic analysis. The authors built a two-level taxonomy adapted to accounting work and used it to annotate low-scoring trajectories from the three best-performing models (n = 74), showing that reasoning failures dominate.

Main Findings

• Leaderboard leader: Claude-Fable-5 (Max) leads with 56.4% Mean Criteria@3, ahead of Muse-Spark-1.1 (xHigh) at 52.6%, then GPT-5.6-Sol at 51.5% and Claude-Opus-4.8 at 48.0%. Qwen3.5-397B-Fp8 is lowest at 24.4%.
• Consistency is the real gap: No model scores more than 2.6% Pass^8 (GPT-5.6-Sol (Max+Pro)). The highest Pass@8 is 21.5% (Muse-Spark-1.1 (xHigh)), narrowly ahead of Claude-Fable-5 at 20.1%.
• Scale of evaluation: 9 frontier models each executed each task 8 times, producing 11,520 trajectories under a cap of 500 steps and 5 million tokens per task.
• Category difficulty: Schedules & Accruals is the hardest category for every model (trailing their highest-scoring category by 7 to 21 percentage points). Claude-Fable-5 leads three of four categories; Muse-Spark-1.1 edges ahead on Variance Analysis (57.2% vs 56.8%).
• Task statistics by category: Reconciliation (61 tasks, 14.61 mean criteria, 7.49 mean files), Data Entry (26 tasks, 16.35, 8.04), Variance Analysis (28 tasks, 14.54, 7.86), Schedules & Accruals (45 tasks, 10.29, 7.00). Overall: 160 tasks, 13.66 mean criteria, 7.51 mean files.
• Simpson's paradox in token budgets: Raising the per-task dollar budget from $1 to $50 raises scores, but within a fixed budget harness, tasks where a model spends more tokens score lower. After Benjamini-Hochberg correction, every statistically significant within-cap spend-score association is negative.
• Budget effects vary sharply by model: Claude-Fable-5 goes from 11.8% at $1 (scoring exactly zero on 38.8% of its tasks) to 55.2% at $50, a gain of +43.4 pp. Muse-Spark-1.1 leads at the tightest cap (47.2%) but gains only +4.7 pp across the range. At $50, Claude-Fable-5 spends $32.36 per run on average against Muse-Spark-1.1's $5.25, yet their scores are within 4 percentage points.
• Budgets are rarely binding at the top: Models use just 64.7% of the maximum budget available at $50, and the cap constrains only 0% to 16.9% of runs across the five models tested.
• Harnesses matter less than models: Averaged over the eight models run under both harnesses, the Ramp-minus-Loop shift is +1.2 pp, or +0.3 pp excluding one outlier. Grok-4.5's +7.5 pp gain (40.8 to 48.3) is the only shift surviving multiple-testing correction. Claude-Fable-5 and Muse-Spark-1.1 both score lower under the Ramp Harness (-0.8 and -2.8 pp).
• Subagent delegation is a model trait, not a lever: Only the Ramp Harness allows delegation. Claude-Fable-5 spawns 4.9 subagents per task in 89% of trajectories, while Gemini-3.1-Pro and Qwen3.5-397B spawn 0.2-0.4 per task in 12-13%. Delegation propensity does not align with harness gains.
• Reasoning failures dominate: Across the 74 annotated failures, 79% of Claude-Fable-5's failures are reasoning failures, 75% for GPT-5.6-Sol, and 59% for Muse-Spark-1.1. Information gathering and instruction following combined account for 17%, 21%, and 36% respectively. Not a single annotated failure involves tool use.
• Specific error patterns: Non-numeric reasoning failures (22 of 74) and data handling errors (16 of 74) jointly account for just over half of all annotated failures. Common patterns include substituting the wrong basis or authorization logic downstream and forgetting information already derived (7 cases across all three profiles).
• Headroom remains large: 93 of the 160 tasks (58%) are not fully solved by any model in any run. Muse-Spark-1.1 shows the largest single-pair headroom, beating Qwen3.5-397B on 32 tasks, and wins more tasks than it loses against every other model despite trailing Claude-Fable-5 on mean score.
• Significance of rankings: Of 36 pairwise Mean Criteria@3 comparisons, 34 remain significant after Benjamini-Hochberg correction at a 5% false discovery rate; two adjacent-rank pairs are statistically indistinguishable.

Methodology in Plain English

The researchers did not use real company books. Instead, they had accounting experts design 10 fictional but internally consistent companies, each frozen at a month-end close, with their own entity type, chart of accounts, revenue model, and prior-period balances. World construction happened in four stages: a cross-world scoping pass, a detailed per-world specification (including deliberate traps and a trap register of seeded contradictions), a reference document used to derive a per-world style guide, and validation of every file against that spec. Documents are novel and screened against public sources so they cannot be memorized. On average, 14 experts worked on each world.

Each task is tied to one world and consists of a prompt, required input files, a golden expert response, and a rubric of binary Pass/Fail criteria. Prompts are written the way an accountant would actually ask, without explaining methods a competent professional would already know. Models must locate their own evidence in the filesystem.

Models are run inside agent harnesses (the standard Loop Harness for the leaderboard, plus a Ramp Harness with retry-awareness, tool allowlists, tool validation, unavailable-tool recovery, a read/write execution policy, and subagent delegation), with access to 91 operational tools and, for the Ramp Harness, 11 meta-tools. Outputs are graded criterion by criterion by an LM judge that sees the prompt, the criterion, and the model's final answer, but not the trajectory. Fairness was checked by having three experts independently annotate 1,687 criteria, which reached 92.8% raw agreement and a Fleiss' kappa of 0.857.

Why This Matters

The paper shifts the question from whether models can pass accounting exams to whether they can complete accounting work. Its failure analysis suggests the bottleneck is not tooling or retrieval but multi-step judgment and discipline: models find the right documents and often compute the right intermediate numbers, then apply the wrong basis or drop a correct result before the final answer. That has direct implications for where effort should go.

• Accounting and bookkeeping firms considering agent deployment can read the Pass^8 results (no model above 2.6%) as evidence that unsupervised close work is not yet viable.
• Financial operations teams at companies evaluating cost-per-task can use the budget ablation showing that score gains flatten well before the $50 cap is reached.
• Benchmark and evaluation researchers can reuse the world-based design, the outcome-based binary rubric requirements, and the judge-validation protocol against 1,687 expert labels.
• Model developers can target the dominant failure modes: non-numeric reasoning and data handling errors that account for 51.4% of annotated failures.

Industry relevance is clear from the paper's framing: accounting generates approximately $700 billion in revenue globally, and the US alone employed close to 1.6 million accountants in 2024. The authors also note that GPT-4 with 10-shot prompting already averaged 85.1% across the CPA, CMA, CIA, and Enrolled Agent exams in 2023, which makes the gap between exam performance and on-the-job performance the central practical question.

Future Directions

• Extend coverage beyond the close cycle. The benchmark explicitly excludes tax, audit, consolidation, multi-entity and multi-currency work, and external reporting.
• Add human-in-the-loop tasks. Tasks requiring a clarifying question were excluded, removing a skill real staff accountants exercise constantly.
• Increase statistical power. With only 160 tasks across 10 worlds, two adjacent-rank pairs remain statistically indistinguishable after correction.
• Address the task-filtering confound. Tasks were selected using low scores from three frontier models (Claude-Opus-4.8, GPT-5.5, and Gemini-3.1-Pro Preview), which raises the risk that models in those families have depressed scores; the authors argue the effect does not determine the overall ranking.
• Investigate why delegation and harness design do not lift scores. The results suggest progress must come from the models themselves, such as accounting-specific training that instills discipline in carrying results through and refusing to post unsupported entries.

Target Audience

This paper is most useful to AI evaluation researchers and benchmark designers working on agentic tasks, to model developers who need concrete failure taxonomies for professional reasoning, and to accounting, audit, and finance operations leaders deciding whether current agents are ready for production bookkeeping work. It also serves policy and workforce analysts tracking how automation capability is advancing in a large, highly regulated profession.

Authors’ abstract

We introduce APEX-Accounting, a benchmark built by Mercor in partnership with Ramp, to assess whether frontier models can do the real work of accountants. Tasks include reconciling accounts, accruing expenses, posting transactions, and producing reports. The private eval set comprises 160 tasks, split across 10 worlds. Each world contains an accounting system, as well as spreadsheets, PDFs, and other files. Every task was authored and solved by experts in accounting and bookkeeping, who also wrote grading rubrics. Across nine frontier models, Claude-Fable-5 (Max) leads with 56.4% Mean Criteria@3, ahead of Muse-Spark-1.1 (xHigh) at 52.6%. No model scores more than 2.6% Pass^8 (GPT-5.6-Sol (Max+Pro)) and the highest Pass@8 is 21.5% (Muse-Spark-1.1 (xHigh)). We experiment with increasing the token budget from $1 to $50 and observe an instance of Simpson's paradox: scores increase as the token budget increases but within a given budget-constrained harness, scores are lower on tasks where the model spends more tokens. As APEX-Accounting is a closed benchmark, leaderboard evals can be run for any frontier model on request.