The Pulse
Surge AI’s Tax Benchmark Gives Its Top Agent a 10.8% Pass Rate
Surge AI’s DAYJOB: Tax benchmark tests AI agents on 50 professional tax assignments and 1,789 files. Claude Sonnet 5.5 leads the company’s leaderboard, passing 10.8% of attempts under its scoring rules.
AI.info Team · Written with AI · fact-checked ·
Claude Sonnet 5.5 passed 10.8% of attempts on Surge AI’s new tax-work benchmark, the highest score among the models tested. The company released DAYJOB: Tax on October 27, with 50 assignments built around the incomplete instructions, conflicting records and judgment calls that tax professionals face. Even the leading model met Surge’s pass standard in roughly one attempt out of nine.
What Surge counted as a pass
Surge evaluated agents on 50 tax environments containing 1,789 files. Each assignment starts with a client request and supporting records; the agent must produce a tax return, workpapers, a technical review or client advice. The company says the tasks span eight areas, including individual and family tax, small businesses, partnerships, international tax, corporate transactions, and tax disputes.
The company reports pass@1 at a 95% rubric threshold: an attempt passes when its work satisfies at least 95% of the criteria for that task. Those criteria cover calculations, supporting evidence, consequential issues and the finished deliverable. That makes the score a measure of strong, end-to-end completion under Surge’s rubric—not a tally of whether an agent can answer an isolated tax question.
The records are varied as well as numerous. Surge lists 1,487 PDFs, 117 spreadsheets and 38 Word documents among the files, with a median of 30 files per assignment and a maximum of 123. Some tasks require agents to reconcile returns and payroll records; others include correspondence, contracts, photographs or dispatch logs.
A missed deadline, hidden in a home sale
One example follows a client, Anna, who moved out of her condo in August 2024 and asked whether selling in October 2026 or January 2027 would be better for taxes. Surge says the agent correctly found that either proposed date met the residence requirement for a capital-gains exclusion of up to $250,000, but missed a later deadline that could matter if the sale were delayed. The company reports that Astra missed the deadline in all five attempts, while Opus identified it in all five.
The example illustrates the benchmark’s emphasis on issues outside the client’s stated question. Another assignment involves a company that believes it operates only in Alabama, while field dispatch records show crews working in five states—a clue that could matter to its tax obligations. In both cases, the test asks whether the agent can connect details scattered through records to a decision the client may not know to raise.
Three Claude models lead the table
Claude Sonnet 5.5 tops the leaderboard at 10.8%, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. The board also lists Muse Wonder 1.0 at 6% and Grok 4.7 at 5%; several evaluated configurations score below 1%, while Kimi K3, Mistral Large 4 and Qwen 3.8 Max each score zero.
Edwin Chen wrote on LinkedIn: “knowing tax rules isn't enough. great tax agents should give you advice too, and have the professional judgment and experience to handle everything you didn't know to ask.”
“knowing tax rules isn't enough. great tax agents should give you advice too, and have the professional judgment and experience to handle everything you didn't know to ask.”
Edwin Chen, LinkedIn
The score does not measure every use of tax AI
DAYJOB: Tax focuses on complex assignments that require agents to investigate records and carry decisions through to a finished deliverable. Its low pass rates show how often the tested systems fell short of Surge’s demanding threshold on these tasks; they do not establish how well or poorly the same models perform on narrower, supervised tax work. The distinction matters for taxpayers and advisers weighing assistance on a discrete calculation against delegating an entire filing or advisory assignment.
Surge says the full leaderboard is available on its benchmark page and that access to the full benchmark can be requested. The reported result is specific: on this set of 50 assignments, the leading model cleared the stated rubric on 10.8% of its first attempts.