The Pulse
SecondState Releases FAB, a 50-Task Test for Finance Agents
SecondState’s Finance Agents Benchmark gives AI agents 50 due-diligence tasks on a synthetic distributor’s 160-file data room. The best of four models met every criterion on 60% of answers; none passed more than 24 tasks in all three trials
AI.info Team · Written with AI · fact-checked ·
In SecondState’s new benchmark of AI agents doing financial due diligence, the best model produced an answer that met every grading criterion 60% of the time. The company has published FAB, the Finance Agents Benchmark, as an open-source project: 50 tasks set in the data room of a fictional industrial distributor, a harness for running and grading agents, and the 600 answers behind its first results.
SecondState makes an AI workspace for finance teams, aimed at accounting, due-diligence, audit, advisory and corporate-finance work. FAB tests the kind of job a buyer’s deal team hands to an analyst, and scores it strictly: a task passes only when every one of its grading criteria passes.
One synthetic company, 160 documents
Every task runs against the same company, Meridian Industrial Supply LLC, a synthetic US industrial distributor with revenue of $120 million in FY2024 and $144 million in FY2025. Its data room holds two years of books and evidence through February 15, 2026, spread across 160 files: SAP-shaped CSV exports, spreadsheets, PDFs, Word documents, a PowerPoint deck and emails.
The records were not written by hand. According to the benchmark’s walkthrough, they were generated from a common double-entry ledger and a company specification, and nineteen planted facts test reconciliation, financial judgment and the recognition of insufficient evidence. The answer key comes from those generated records and never enters the agent’s workspace.
The 50 tasks split into 15 easy, 20 medium and 15 hard requests, with 231 grading criteria in total. Easy ones ask for reported EBITDA or a reconciliation of FY2025 net revenue between SAP and the management accounts. Hard ones ask the agent to assess each EBITDA adjustment, bridge enterprise value to equity value, look for year-end window dressing, or write a red-flag memo on the five most important issues for the deal team.
A sealed sandbox and an LLM judge
Each trial starts in a fresh sandbox with the data room mounted read-only and no network access. The agent can list, search and read the room’s file formats, run Python or DuckDB, and must write its answer to a single file. By default it gets 200 model turns and 500 tool calls per task. Grading criteria and private generation data are never mounted in the sandbox.
A judge model, GPT-6 Luna at maximum reasoning, scores each criterion separately against the task and the answer. The walkthrough notes that some criteria were clarified after the agent runs while the agent instructions stayed unchanged, and that the harness system prompt has changed since the published runs.
DeepSeek leads on tasks, GPT-6 Sol on criteria
SecondState ran four models three times on all 50 tasks, for 600 answers. DeepSeek V4.1 Flash had the highest task pass rate, 60.0%, passing 38 tasks at least once and 23 in all three trials. GPT-6 Sol passed 58.7% of tasks but had the highest criterion pass rate, 83.4%, and the most tasks passed in all three trials, 24. GPT-6 Luna scored 50.7% on tasks and 79.5% on criteria; GLM 5.3 Flash scored 47.3% and 76.2%.
The gap between the two measures follows from the all-or-nothing rule: a model can clear most individual checks and still fail the task. The difficulty split in the results report separates the two leaders. DeepSeek passed 42 of its 45 easy answers against 35 for Sol, while on hard tasks Sol passed 21 of 45 and DeepSeek 13.
The report also estimates the cost of a completed run, excluding judging and treating all input as uncached: about $0.06 for GPT-6 Luna, $0.14 for GLM 5.3 Flash, $0.88 for DeepSeek V4.1 Flash and $0.94 for GPT-6 Sol. SecondState labels these figures estimates, not invoices.
Limits the company states itself
All 50 tasks share one company, and two of them, tasks 041 and 049, are scored using saved regrades; the results report preserves an earlier concern about unclear assumptions in those two tasks. SecondState wrote in the benchmark’s walkthrough:
“These scores describe one synthetic company and do not establish general superiority across financial work.”
SecondState, FAB walkthrough on GitHub
The task set, documents and question index are on Hugging Face, and a companion dataset holds all 600 runs with their traces, answers, rubrics and final grades. The code is MIT-licensed and the data, tasks, rubrics and results are released under CC BY 4.0. Running the harness requires Python 3.11 or later, uv, Docker or Podman, and model API keys. The dataset card asks users to keep the benchmark out of model training data and to cite the dataset and code revisions when reporting results.