The Pulse
BizBench Finds Rule Violations Behind Agents’ Success Rates
BizBench tests AI agents on 50 simulated business tasks based on payment workflows, checking both task outcomes and compliance during execution. In the benchmark, Qwen3.8-27B reached its goal on 42% of runs but met all success criteria on 3
AI.info Team · Written with AI ·
“For Qwen3.8-27B, goal attainment reaches 42.0%, but strict success drops to 32.7%, demonstrating how outcome-based evaluation can mask invalid execution.”
Enhan Zhao et al., “BizBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business”
The BizBench benchmark, published as an arXiv preprint on October 14, measures not just whether an AI agent finishes a business task, but whether it follows the rules while doing so. Its authors argue that a task can appear complete even after an agent has crossed a financial or authorization boundary that should count as failure.
The study evaluates 50 multi-step tasks based on real-world payment workflows. Its simulated environment contains 789 knowledge documents and 601 business capabilities, requiring agents to find applicable guidance, gather customer information, and manage actions whose consequences carry into later steps.
Success means more than reaching the goal
BizBench scores each run against three conditions: whether the requested outcome was achieved, whether the process followed applicable requirements, and whether the agent avoided prohibited side effects. Its deterministic evaluator checks business records and execution histories, rather than asking an AI judge to assess the run. Different action sequences can pass if each meets the criteria.
That distinction matters in the authors’ examples. An agent might make a payment that a tool accepts, yet leave an account below a required cash reserve. A later refund can restore the final balance, but it does not erase the earlier breach. The benchmark also checks whether an operation received specific approval before execution, and whether the approval still covers the terms actually used.
Qwen’s 9.3-point gap
Qwen3.8-27B achieved its stated goal in 42% of runs, but its strict success rate—requiring all three conditions—was 32.7%. The paper records 14 runs in which the agent reached the goal but failed a process or protection check; eight involved attempted execution before approval. Goal completion alone would have counted those outcomes too generously.
The top strict success rates in the reported comparison were 64% for Claude Opus 5 and 61.3% for GPT-6 Sol. The paper says Claude Opus 5 was tested once on each of the 50 tasks, while most other models, including GPT-6 Sol and Qwen3.8-27B, were tested three times per task. Every model in the table had intermediate failures, though the frequency varied widely.
For DeepSeek-V4.1-Flash, five of 150 runs reached the requested goal but failed the full criteria. Four temporarily fell below a required cash-reserve floor before later restoring the balance; another attempted an operation before approval and received a rejection. The findings show why a closing-state check can miss problems recorded in the action history.
Information gathering adds cost
Agents also spent substantially more effort searching than the researchers’ reference executions. DeepSeek-V4.1-Flash averaged 191.3 tool calls per run, compared with 111.8 in the reference traces. Its extra calls were concentrated in database queries and searches or reads of knowledge documents, rather than in business operations themselves.
Providing the relevant documents in advance raised that model’s strict success rate from 43.3% to 50%. Giving it all customer-known information at the outset raised the rate to 70%, but left 30% of tasks unsolved. The authors say the result points to difficulties eliciting customer constraints and preserving them through a workflow, as well as the challenge of finding the right policy.
A simulated benchmark, not a deployment audit
BizBench’s tasks draw on workflows from a leading payments and financial technology company, but the paper does not name that company as the source of the workflows. The authors say they rewrote sensitive business details while preserving decision logic, then built an executable simulation with synthetic records and expert-reviewed checks. The benchmark therefore tests agent behavior in a constructed environment; it does not measure rule violations in live customer operations.
That scope still makes its central result concrete: a task-completion score can conceal a run that violated an approval requirement or briefly spent money reserved for another obligation. In BizBench, only runs that satisfy the goal, process, and protection checks count as strict successes.