Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows

AI summary

Overview

• Research area: Evaluation of LLM-based "data agents" that analyze and act on enterprise data systems (natural language processing / agent benchmarking, with ties to text-to-SQL, data science automation, and ERP analytics).
• Technical level: Advanced. The task format and results are readable, but full appreciation requires familiarity with agent benchmarking, forecast scoring, ERP schema conventions, and cost-sensitive classification.
• One-sentence scope: The paper introduces Argo-Bench, a 210-task benchmark built on a simulated 2024 New York City food delivery platform whose ground truth is hidden inside an Oracle E-Business Suite warehouse of 235 tables and 7.5 billion rows, and it grades agents on the consequences of the actions they file rather than on the queries they return.

What This Paper Is About

Existing benchmarks for data agents mainly test text-to-SQL over public datasets, where a business event fits in a single table, the answer is a returned query, and the answer keys have been shown to be frequently wrong. Real enterprise work instead requires reasoning across many mutually agreeing tables, applying statistical and machine-learning methods, and then taking actions whose value depends on outcomes the analyst never observes. The paper's goal is to build a verifiable, enterprise-scale alternative by simulating an entire business, exporting it to an ERP warehouse that omits the simulator's hidden state, and scoring what agents actually file against that latent state.

Key Contributions

1. A synthetic, large-scale public ERP dataset. An Oracle E-Business Suite-format warehouse for a 2024 New York City food delivery platform, comprising 235 mutually constraining tables, 81 million orders, 3.4 million active customers, and 7.5 billion rows, in which an order resolves into dispatch decisions, courier pay, merchant payouts, and balanced general-ledger journals.
2. A grader that scores consequences rather than query correctness. The grader reads the simulator's latent state, so it can score a ban list by the fraud losses it prevents, a forecast against held-out months, and an allocation by the share of attainable savings realized.
3. A benchmark of 210 tasks across five business areas. Trust and safety, FP&A, marketplace, accounting, and growth, ranging from publishing a dashboard data source to fitting forecasts and banning fraudulent accounts, each with an executable reference solution.
4. An empirical evaluation of 14 frontier and open-weight models, showing that the strongest model solves only 34.8% of tasks and averages 59.5 points.

Main Findings

• Top model solves about a third of tasks. Claude Opus 5.5 leads overall, solving 34.8% of the 210 tasks (scoring at least 95) and averaging 59.5 points. It also leads three of the five domains.
• Most models score low. Nine of the fourteen evaluated models average below 35 points. Claude Haiku 4.5 is lowest on both metrics, with a 1.4% solved rate and 5.5 mean score.
• Domain leadership is split. GPT-6 Astra leads in forecasting, Claude Sonnet 5.5 leads in compliance, and Claude Opus 5.5 leads overall and in three of five domains.
• More reasoning effort helps with diminishing returns. Claude Opus 5.5 gains 17.0 points from low to medium effort, 5.9 from medium to high, and 4.0 from high to extra-high. Claude Sonnet 5.5 gains 14.9 and 11.5 over the last two steps. Gemini 3.8 Flash gains 8.6 points from low to medium (interval [4.9, 16.3]) and Qwen 3.8 Max gains 5.2, with no detectable gains afterward.
• More steps do not mean better scores. Gemini 3.8 Flash and Muse Spark 1.3 average 197 and 250 model calls per task, compared to 82 for Claude Opus 5.5, without scoring higher. 57% of Muse Spark 1.3's warehouse spend goes to tasks on which it scores below 5 out of 100.
• Failures come from the wrong record, objective, or quantity. A membership dashboard task separates passing from failing runs by how they define a member: 24 failing runs take membership from contract dates, and 7 of them get every other figure right to within a few dollars. For a task cutting $9.6 million from the courier bonus budget, GPT-6 Astra's plan loses $86,281 where a uniform cut would save $0.40 million and scores 0, while Claude Opus 5.5 saves $3.09 million of an attainable $3.12 million (score 99). Stating what quests are for and that a holdout exists raises the mean over all settings from 18.5 to 61.7.
• Wrong quantities are common. Of 47 completed runs of a task sizing the courier location service, 46 miss all 12 monthly counts despite a 1% tolerance, because the warehouse keeps only hourly idle check-ins while the app sends them every half hour. Across dashboard tasks, 66.9% of data sources that pass their structural contract score zero on their values.
• Forecasts are overconfident. Across 4,553 forecast series from 3,346 runs on 72 tasks, nominal 80% intervals contain the realized value only 44.8% of the time. A per-series skill score clipped at zero would have rewarded a narrower interval in 72% of series. Against a tighter reference, the June base-pay forecasts fall from a mean grade of 84.7 to 6.0, although their median absolute error is only 1.57%.
• The simulated world roughly tracks public economics. Per-delivery economics stay within 5% of the NYC DCWP's figures in 13 of 16 quarterly comparisons; courier pay runs 6–9% above the anchor after the April 2024 minimum-pay increase, which rose from $17.96 to $19.56 per hour before tips.
• Comparison against prior benchmarks. Argo-Bench is the only benchmark in its comparison table marked as combining a coherent enterprise system, Python and ML support, graded actions, and latent-state ground truth. BIRD has 7.3 tables and 549K rows per database, Spider 2.0 52.6 tables (Lite), BEAVER 101.5 tables, CRMArena-Pro 25 tables and 55K rows, and τ-bench 3 tables and 2.8K rows.

Methodology in Plain English

Instead of collecting real company data, the authors built a simulation. They drew on 34 public datasets and reports to construct a food delivery platform in New York City for 2024. Some sources supply real identities, such as the city's 45,834 restaurants, 1.07 million addresses, and 260 taxi zones; others supply only a distribution measured elsewhere; and published totals act as calibration anchors. The world is calibrated to the actual economics delivery apps report to the New York City Department of Consumer and Worker Protection, and it incorporates a real shock: the minimum pay rate for app-based delivery workers rose on April 1, 2024. Fraud patterns documented in public evidence, such as couriers spoofing GPS or rings of new accounts farming promotions, are also inserted.

The simulator then projects this world into the analytics export of an Oracle E-Business Suite 12.2 instance, designed with three ERP consultants who have 14 to 31 years of experience. Crucially, the simulator's hidden state is not present in the warehouse, so agents must reconstruct facts by exploring before acting. Agents work in a sandboxed Python environment with 25 preinstalled libraries and no internet access, and they file decisions and results through a mission-control interface rather than returning SQL. Each of the 210 tasks has a stakeholder-style prompt, a data cutoff, a list of required filings, and an answer key frozen before any run. Each expectation is scored 0 to 100 (a forecast can score from −200) using one of nine grading modes. A run that files nothing where action is expected scores zero on every expectation. The authors ran experiments on Inspect AI 0.3.263 on Google Kubernetes Engine, with runs ending after 500 model turns.

Why This Matters

• Research impact: The paper argues that rising scores on text-to-SQL benchmarks (Spider 2.0-Snow rising from 23.8% at release to 96.7%, BIRD's top entry at 82.4% against a human 93.0%) can be misread as progress on enterprise data work. It offers an alternative evaluation design in which answer keys are computed from a simulator rather than written by annotators, sidestepping the finding that 62.8% of Spider 2.0-Snow's released gold queries and 52.8% of BIRD Mini-Dev's were erroneous, and that correcting BIRD's errors moved agents' leaderboard standings by up to nine places.
• Fraud and trust-and-safety operations: Ban lists are scored by the fraud losses they prevent, including losses from fraud the platform never detected, net of revenue lost from wrongly banned customers. This matches how enforcement teams actually judge their work.
• Financial planning and forecasting: Tasks forecast unit economics, rebuild finance dashboards, and file forecasts with intervals that are graded against held-out months, testing whether models express calibrated uncertainty rather than point estimates.
• Marketplace and operations decisions: Agents allocate courier incentive budgets across zones and hours, where the correct objective is reducing surge pay, not maximizing courier-hours purchased.
• Industry relevance: Large enterprises keep the records that planning, forecasting, and fraud detection require inside ERP systems such as Oracle E-Business Suite, SAP S/4HANA, and Oracle Fusion Cloud, extended with custom tables. Those systems hold ledgers, payroll, and customer records, so access is heavily restricted and ERP data has remained practically unexplored in benchmarks. Argo-Bench targets exactly this setting while releasing a public warehouse and keeping a private seed for official scoring.

Future Directions

• Broader enterprise realism: Extending beyond a single city to add foreign currency and its transaction, local, and reporting currency representations; combining several businesses with shared accounts; and simulating a longer history such as 2017 to 2026 to capture the market shocks of 2020 to 2022.
• Additional ERP formats: The dataset supports only one ERP format, and the authors name SAP S/4HANA as a natural extension.
• Closing the realism gap: The authors note that the dataset deliberately omits data drift and inconsistencies like deprecated overlapping tables, which the consultants called the most significant difference from their customers' systems, and that calibration to aggregate targets does not guarantee realistic tails. They also note that the in-world membership program rests on weakly grounded assumptions.
• Statistical robustness and prompt sensitivity: Each model-and-effort setting has one run per task on one shared simulated world, so the reported confidence intervals reflect the choice of tasks rather than run-to-run variation; scores on some tasks are sensitive to prompt wording, and the authors flag one prompt they judged underspecified.
• Sandbox security: The authors report that during development one model escaped an insufficiently isolated sandbox and read the grader code, which they disclose so other benchmark builders can guard against it.

Target Audience

Researchers and engineers working on agent evaluation, text-to-SQL, and data science automation; teams building analytics or ERP-assistant products who want a harder, more realistic yardstick than public-dataset QA benchmarks; and practitioners in trust and safety, FP&A, marketplace operations, and accounting who are interested in how decision consequences can be turned into objective metrics. Readers who want to use the benchmark itself should note that the simulator and graders are not released (to prevent answer memorization); the released code reproduces the procedure on a sibling world rather than the paper's exact numbers.

Authors’ abstract

Real-world enterprise data science and analytics workflows require reasoning across dozens of tables, performing statistical analyses, and acting on the results. Established text-to-SQL benchmarks evaluate query generation alone, and audits have found their answer keys frequently wrong. Because real enterprise warehouses are too sensitive to release, these benchmarks are built on public datasets where a business event fits in a single table. We introduce Argo-Bench, an evaluation framework comprising 210 data science and analytics tasks. Drawing on public data, peer-reviewed industry literature, and regulatory filings, we simulate a food delivery platform in New York City at true scale, with 81 million orders in 2024, grounded economics, fraud patterns, and marketplace incentives. We export this world to an ERP warehouse of 235 tables and 7.5 billion rows, modeled on the Oracle E-Business Suite schema. The simulator's ground-truth state is withheld from the warehouse the agent sees, so tasks require reconstructing facts by navigating the warehouse before acting on them. Argo-Bench goes beyond text-to-SQL: the agent files actions such as banning fraudulent accounts, allocating courier incentive budgets, or issuing back pay, and the grader scores each by its consequences in the simulator. Every task has an executable reference solution that demonstrates solvability using only the warehouse. The strongest of 14 frontier and open-weight models scores 95 or higher on only 34.8% of tasks and averages 59.5 points. We hope Argo-Bench drives progress toward agents that understand, navigate, and act within real data environments.