# FDE-Gym: The Last Mile of AI Delivery

AI.info

> Most agent benchmarks hand the model a well-specified problem — an issue report, a specification, or a ticket — and ask whether it can be solved. The authors argue that real customer delivery looks…

**arXiv**: 2610.22350

**Published**: 2026-10-15

**Authors**: Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo, Ying Zhao, Youmin Chen, Xiaojian Liao, Congming Gao, Tianyang Jiang, Yixiao Chen, Qing Wang, Zhiguo Yu, Xiaqing Li, Yuhao Zhang, Jiawen Sun, Renzhe Xu, De Mi, Tiancheng Wu, Xianya Fu, Xuanlu Zhao, Bo Feng, Meixuan Ren, Guanming Lin, Ziheng Zhou, Xinhua Wang

## AI summary

## Overview

**Research area:** AI agent evaluation and enterprise AI deployment (benchmark design for agentic systems).

**Technical level:** Intermediate. The scoring formulas are simple, but the paper assumes familiarity with agent benchmarks, normalized metrics, and LLM evaluation protocols.

**Scope:** FDE-Gym is a benchmark of 95 end-to-end customer delivery tasks, built from 20 real projects across 18 sectors, that scores agents on measurable improvement to a customer's own business metric rather than on task completion, after penalizing the cost of contacting stakeholders.

## What This Paper Is About

Most agent benchmarks hand the model a well-specified problem — an issue report, a specification, or a ticket — and ask whether it can be solved. The authors argue that real customer delivery looks different: a forward-deployed engineer (FDE) has to work out whether the request even targets the right problem, discover constraints nobody stated, and operate inside live customer systems. FDE-Gym is built to measure that last mile, scoring the business improvement a delivered system produces relative to a do-nothing baseline and an expert-authored reference solution, with a deduction for every question asked of a customer stakeholder.

## Key Contributions

1. **A delivery-oriented benchmark.** 95 tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects carried out in China, Singapore, and the United States since 2024. Each task pairs a simulated customer environment and scripted stakeholders with a production-like system the agent must modify.
2. **A four-level task instruction ladder.** The same workspace, stakeholders, and grader are reused while the instruction degrades from L0 (underlying problem, rules, and scoring all stated) to L3 (the sponsor's original request, which may target the wrong problem). Because only the instruction changes, score differences isolate the effect of withheld information.
3. **An outcome-based, cost-aware scoring method.** The FDE-Gym score is the normalized business uplift on held-out production data, adjusted to zero for operational incidents, minus a stakeholder contact cost charged per message, per trial run, and per day of waiting. A task counts as *resolved* only at a score of at least 0.8 with no incident.
4. **Anti-gaming construction and an open platform.** Four admission anchors (reference solution at least 0.95, do-nothing baseline at 0, literal compliance at most 0.2, shortcuts at most 0.8), graders tested against more than 270 shortcut and bypass strategies, over 10,000 expert hours invested, plus an online gym at fde-gym.com where humans are graded by the same criteria.

## Main Findings

- **Even the best model is far from the reference.** At L3, Claude Opus 5.5 leads with a mean score of 0.56 (95% interval 0.50 to 0.62) and resolves 14 of 95 tasks; Claude Sonnet 5.5 reaches 0.49 and resolves 15, GPT-6 Astra 0.38 with 9 resolved, and GPT-6.1 Sol 0.37 with 7. Five of the thirteen configurations resolve none. Choosing the best configuration per task still yields only 0.70 with 30 tasks resolved, and on 16 tasks no configuration reaches 0.5.
- **Discovering unstated constraints is the main bottleneck.** With the rules provided (L0), the three leading models score 0.73–0.82 and resolve 11–13 of the 20 original tasks. Requiring them to find the constraints (L1) drops them to 0.39–0.62 and 1–7 resolved. The paired L0-to-L1 decline is 0.28 (interval 0.17 to 0.39), while the L1-to-L3 decline is only 0.04 (−0.03 to 0.14). Making the request vague or misdirected costs far less than withholding the constraints.
- **Getting the facts does not mean delivering the outcome.** At L3 the leading models ask 10.0–12.3 questions per task and obtain 72–85% of the key facts held by stakeholders, yet deliver only 0.60–0.70 of the reference uplift. Across the 73 task–model cases where one of the three obtained every key fact, uplift averages 0.66 and reaches at least 0.8 in 27 cases (37%). For Claude Opus 5.5 alone, all key facts were obtained on 23 tasks but uplift reached 0.8 on only 10.
- **Stakeholder contact cost erodes otherwise resolved tasks.** GPT-6 Astra's delivered systems reach 0.8 on 36 of 95 tasks, but only 9 stay resolved after the cost deduction; for Claude Opus 5.5 the figures are 40 and 14. The cost consumes 0.14 of Opus's 0.70 uplift and 0.22 of Astra's 0.60. Agents are not told that questions cost anything, and they ask as if questions were free. Mistral Small 4 sits at the other extreme, asking 0.7 questions per task and obtaining 6% of key facts.
- **Weak models leave customers worse off than doing nothing.** Claude Haiku 4.5 scores below the do-nothing baseline on 79 of 95 tasks and GPT-6 Luna on 63, compared with 5 for Claude Opus 5.5 and 7 for Sonnet 5.5. Before any contact cost is deducted, Mistral Small 4's systems fall below baseline on 51 tasks and Claude Haiku 4.5 (thinking) on 55, versus 1–4 for the leading models. On 23 tasks, Mistral Small 4's delivered system caused more harm than the reference solution produced in benefit. Extended thinking did not help: Claude Haiku 4.5 scored −0.33 without it and −0.35 with it.
- **No model covers all scenarios.** Claude Opus 5.5 is best on 11 of 20 scenarios, Sonnet 5.5 on four, GPT-6 Astra on two, and GLM-5.3 Flash, Gemini 3.8 Flash, and GPT-6.1 Sol on one each. On five scenarios no model averages at least 0.5; the hardest, news_incident and revenue_dispute, top out at 0.18 and 0.27. On revenue_dispute the three leading models score 0.12, 0.27, and −0.44. Within a scenario, the median score range for a leading model is 0.34–0.44, and it exceeds 0.5 in 6–8 of the 20 scenarios — the same scenario with different key facts produces different results.
- **Spending more effort does not buy a better score.** GPT-6.1 Sol spends USD 0.24 and 25 turns per run to score 0.37, while Gemini 3.8 Flash spends USD 1.54 and 121 turns to score 0.19. Qwen 3.8 Flash runs longest at 169 turns and 110 minutes, submits on only 11 of its 152 task–instruction combinations, and scores 0.07. The three leading models submit on every combination, in 26–60 turns.
- **Component benchmarks poorly predict delivery.** Models that solve up to 88% of Terminal-Bench 2.1 tasks, 90% of Vibe Code Bench, and 61% of Finance Agent score 0.19–0.56 on FDE-Gym. The closest FDE-Gym analogue to a success rate is the resolved share: Claude Opus 5.5 resolves 14 of 95 tasks (15%).

## Methodology in Plain English

The authors started from real consulting-style delivery projects and had industry experts rewrite each one as a self-contained task: a customer, a sponsor who assigns the work, a copy of the customer's production system, supporting documents and historical data, and a staff directory. For each of the 20 scenarios, experts also wrote two to seven variants that change the key facts — which rule is outdated, who holds critical information, whether the sponsor's request actually addresses the underlying problem — so memorized solutions do not carry over. No single problem pattern appears in more than one-third of the set.

Stakeholders are scripted rather than simulated by a live model, because the authors cite evidence that swapping the user-simulation model can shift measured success by several percentage points. Over 130 stakeholders across the scenarios have fixed knowledge, fixed misconceptions, limited patience, supporting-evidence requirements, dependencies on earlier conversations, temporary unavailability, and contact restrictions. A small model simply maps each question to a topic.

The scoring works in three steps. First, the delivered system is deployed unmodified into production and evaluated on held-out data against the unchanged system and an expert reference, giving a normalized uplift where 0 is no improvement and 1 matches the reference. Second, an incident adjustment zeroes out any positive gain if a predefined operational incident occurs while preserving losses. Third, the sum of charges for messages, trial runs, and days of waiting is subtracted. The reported average floors the adjusted uplift at −1 so a single catastrophic task cannot dominate.

Before a task is admitted, it must pass four anchors: the reference scores at least 0.95, the do-nothing baseline scores 0, a literal solution that does exactly what the instruction says scores at most 0.2, and shortcut strategies score at most 0.8. The one exception is a task where the sponsor's request is both correct and complete; there the literal solution coincides with the reference, so distrusting the sponsor is never universally rewarded. Every configuration ran with the same minimal five-tool framework, three runs per task–instruction combination, capped at 200 turns, 150 minutes, or USD 8 of model usage.

## Why This Matters

**Impact on research.** The paper argues that progress on coding, terminal, and research benchmarks is a weak proxy for whether an AI system will help a real customer. By scoring on a business metric with a do-nothing baseline that deliveries can fall below, it turns "an agent built something" into "an agent made the customer measurably better off," and it separates three distinct failure modes — misread problem, undiscovered constraint, poor implementation — that most benchmarks blur together. The finding that withholding constraints costs 0.28 while misdirecting the request costs only 0.04 is a concrete reframing of where agent research should focus.

**Real-world applications:**

- **Enterprise AI procurement.** A buyer comparing vendors can use the resolved-task share (14 of 95 for the best model) as a sanity check against vendor claims built on component benchmarks.
- **Agent product development.** The literal-compliance anchor at most 0.2 shows that building exactly what the spec asks can pass every functional check and still move no business metric, which argues for requirement-challenging behavior in deployed agents.
- **Consulting and systems-integration practice.** The scripted-stakeholder design, with patience limits and contact restrictions, is a template for training and assessing forward-deployed engineering teams.
- **Domain-specific deployment in regulated sectors.** Scenarios cover banking anti-scam queues, prior authorization, hospital triage, power-grid settlement, and oil and gas downtime claims — areas where a wrong automated action is costlier than no action.

**Industry relevance.** The finding that weak models score below the do-nothing baseline on 60–83% of tasks reframes cheap models as a net-negative purchase rather than a budget option: "a weak agent is not a cheaper version of a strong one." The contact-cost mechanism also names a real deployment problem — agents that interrogate customers as if questions were free.

## Future Directions

- **A human baseline.** The online platform is collecting results from industry experts on the L3 tasks under recorded conditions; these will be reported in a future version and will establish how far humans get from only the sponsor's request.
- **Teaching constraint discovery.** The sharp L0-to-L1 drop, and the fact that even at L0 no model exceeds 0.82, suggests separate work is needed on surfacing unstated rules and on turning recovered facts into working systems.
- **Cost-aware and completion-aware agent behavior.** Agents are not told that contact is charged and generally ask as if it were free; the gap between systems reaching 0.8 on 36 tasks and only 9 staying resolved points to pricing awareness and knowing when work is finished as open capabilities.
- **Held-out material for reuse.** The 75 variants and all reference solutions are withheld, leaving room for later evaluation rounds and for testing whether gains on the published tasks transfer to altered key facts.

## Target Audience

AI researchers and benchmark designers working on agents and evaluation; ML engineers and product teams building enterprise agents or forward-deployed engineering tools; and technical decision-makers in enterprises who need to judge what an agent benchmark actually predicts about delivered business value. Readers interested in the human side of AI deployment — requirements elicitation and stakeholder interaction — will also find the scenario design instructive.

## Authors’ abstract

Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery…

[The full abstract on arXiv](https://arxiv.org/abs/2610.22350)

[Read the original paper](https://arxiv.org/abs/2610.22350)

---

Canonical: https://ai.info/paper/fde-gym-the-last-mile-of-ai-delivery-2610.22350
