C-World: A Computer Use Agent Environment Creator

AI summary

Overview

Research area: LLM-based autonomous agents, tool use, environment/benchmark construction, and agent training data generation.

Technical level: Intermediate. Familiarity with LLM agents, function/tool calling, and the Model Context Protocol (MCP) helps, but the paper is readable without deep background.

Scope: The paper presents C-World, a system that lets users generate complete agent environments—action space, tasks, environment dynamics, and reward signals—either against live APIs or entirely in simulation, and uses it both to evaluate nine frontier LLMs and to train smaller models with very little data.

What This Paper Is About

Agents powered by large language models still underperform humans on long-horizon, multi-step tasks with interacting constraints, and the main reason is that no large, diverse, realistic environment exists for them to learn in. Building such environments by hand is prohibitively expensive, and existing benchmarks are narrow, static, and mostly non-interactive. C-World addresses this by turning environment construction itself into an automated, component-based system that can be extended to new tools, tasks, and failure modes—including tools and domains that do not exist in the real world.

Key Contributions

1. A four-component formalization of agent environments. The authors define an environment as an Action Space, a Task Distribution, a Transition Function, and a Reward Signal, and implement each as a concrete, composable module rather than a fixed benchmark artifact.

2. A large, validated, format-unified tool library. 5,571 executable tools drawn from 276 MCP servers covering 204 production-grade applications, filtered through a three-stage validation pipeline (authenticated availability, successful invocation, usable responses) and indexed for natural-language retrieval so agents can discover tools on demand.

3. A task synthesis engine with "wild constraints." Tasks are generated through a check-then-revise loop that samples coherent but cross-application tool bundles, then iteratively produces long-horizon instructions with compositional constraints, plus fuzzy rewriting that hides tool names so tasks cannot be solved by keyword matching.

4. A World Engine that simulates tool execution without live APIs. By encoding category-level cards for tool families (email, calendar, code hosting, etc.), the engine predicts realistic responses—including failures—and achieves a Spearman ranking correlation of ρ = 0.883 with real execution.

5. A state controller and multi-signal reward protocol. The controller injects tool-level, state-level, and constraint-level disruptions mid-execution to test recovery; rewards combine deterministic checks (schema compliance, order, information diversity) with an LLM-as-judge ensemble scored against human rankings.

Main Findings

• Planning is uniformly strong; execution is the bottleneck. All nine evaluated models scored 7.7–8.6 on goal decomposition, but success rates and answer completeness diverged sharply, indicating the Actor role—not the Planner—limits performance.

• Constraint following, not tool invocation, is the dominant failure mode. Models generally call the right tools; they fail to satisfy ordering, format, and tradeoff constraints embedded in the task.

• More tool calls do not mean better results. gpt-4o-mini made the most calls (51.71 on average) yet produced the lowest-quality output, falling into a "looping" pattern. Gemini-3-pro-preview made a similar number of calls (47.86) and converted them into far higher completeness.

• Robustness and quality are partly decoupled. DeepSeek-v3.2 achieved the best recovery rate (90.6%) and flexibility (72.4%) despite not topping the overall scoreboard, while gpt-5.2 had the strongest grounding (3.80) but collapsed into early abandonment and hallucinated interfaces.

• A strong start does not predict long-horizon success. Under the Planner–Actor framework, gpt-5.2 dropped five ranks from its Turn-1 standing, while gemini-3-pro-preview and glm-4.6v climbed four and five positions respectively. Sustained reflection matters more than a good opening move.

• Synthesized environments preserve model rankings. Executing the same 50 tasks through the World Engine yielded ρ = 0.883 correlation with live-API rankings, supporting simulation as a cheap substitute for evaluation and data generation.

• C-World is an extremely data-efficient training engine. Fine-tuning Qwen2.5-7B and Qwen3-8B on only 1,170 curated trajectories beat baselines trained on 119k (Toucan) and 11.3k (ToolACE) samples on BFCL and MCP-Universe, with the gains clearly attributable to fine-tuning given sub-20% zero-shot baselines.

• The LLM-as-judge protocol nearly matches human agreement. DeepSeek-v3.2 reached ρ = 0.759 and GPT-5.1 reached ρ = 0.733 against human rankings, close to the human–human ceiling of ρ = 0.773, and the cross-family majority vote neutralized single-vendor bias.

Methodology in Plain English

The researchers treat an agent environment as four separable pieces and build each one programmatically.

For the toolset, they crawled the Smithery MCP registry for tools people actually use in daily work—documents, code, search, collaboration—and then tested every tool under dedicated virtual accounts to confirm it could be listed, invoked, and would return usable output. The surviving 5,571 tools were embedded as searchable documents so agents can query for relevant tools at runtime rather than being handed a fixed list.

For tasks, they start by randomly picking one to three seed tools, use those to retrieve a broader set of related tools, then group by server and round-robin sample so no single app dominates. An LLM drafts a task that plausibly needs those tools, and a loop repeatedly critiques the draft on tool coverage and constraint quality, revising until both are satisfactory. Tool names are rewritten into intent-level phrasing to prevent trivial keyword solutions.

For environment dynamics, they place a lightweight Python middleware between the agent and the tool backend. It forwards calls normally but can inject realistic failures—timeouts, rate limits, corrupted results, session expiry—and even alter task constraints mid-run to see whether the agent replans.

For evaluation, they combine deterministic checks computed straight from execution logs (schema validity, ordering against a dependency graph, diversity of sources reached) with an LLM-judge ensemble scoring completeness, grounding, format, and tradeoff quality.

Agents themselves run as a Planner–Actor pair using the same LLM: the Planner decomposes the task into a sub-goal graph upfront and monitors progress during execution, while the Actor follows a ReAct loop of reasoning, retrieving tools, invoking them, and observing results.

The World Engine replaces live execution by grouping tools into functional categories and hand-authoring a "card" per category describing typical response shapes and failure modes, then generating responses conditioned on the card, the tool schema, and the session log for consistency.

Why This Matters

Impact on research. C-World reframes environment construction as a generative, extensible problem rather than a one-off benchmark engineering effort. It also provides direct evidence that simulated tool execution can substitute for live APIs in both evaluation and training, which is a meaningful methodological claim for the agent-research community. The persistent finding that constraint following—not tool selection—is the bottleneck redirects attention toward a specific, tractable research target.

Real-world applications:

• Enterprise agent deployment and pre-deployment testing. Companies can build environments around their own MCP servers and internal workflows to evaluate agent readiness before granting production access.
• Agent training data generation at low cost. Teams without budget or credentials for thousands of live API calls can use the World Engine to produce trajectories for fine-tuning.
• Stress-testing robustness. The state controller's targeted failure injection supports systematic resilience testing—recovery rates, replanning behavior—rather than only happy-path evaluation.
• Domain prototyping for not-yet-existent tools. Synthesized environments allow testing agents against hypothetical services (medical databases, legal corpora, custom enterprise APIs) before those services are built.

Industry relevance. The data-efficiency result is directly commercially relevant: matching or exceeding baselines trained on ~100x more data means agent fine-tuning pipelines can be far cheaper. The tool validation and credential-provisioning methodology also offers a practical template for anyone trying to make a large API surface reliably testable.

Future Directions

• Expanding scenario coverage. The evaluation set is only 50 synthesized scenarios, which cannot cover the combinatorial space of tool, server, and constraint interactions. Scaling this without sacrificing diversity is an open engineering problem.

• Deepening the World Engine. Current simulation works at the category level with hand-authored cards. Extending it to finer-grained, automatically learned response models—and validating fidelity beyond ranking correlation—remains open.

• Characterizing sub-10B models. The authors note their behavioral findings may not transfer to smaller models; a dedicated study of the planning–execution gap in that regime is needed.

• Closing the execution gap itself. Knowing that models plan well but fail at constraint following raises the question of what training signal or architectural change would fix it—an obvious next target given C-World's ability to generate targeted, constraint-dense trajectories.

Target Audience

Researchers and engineers building or evaluating tool-using LLM agents will get the most from this paper, particularly those working on agent benchmarks, MCP-based infrastructure, or synthetic training-data pipelines. It is also useful for ML practitioners who need a practical framework for testing agents against large, heterogeneous API surfaces, and for teams deciding whether simulated execution is trustworthy enough to replace live-API evaluation. Readers interested in LLM-as-judge reliability will find the human-alignment analysis (Section 5.2) independently valuable.

Authors’ abstract

To close the gap between LLM-based agents and humans in planning and reasoning, agents need large-scale, diverse environments for continuous learning -- yet building such environments is itself prohibitively expensive. We present C-World, an environment creation system that enables users to build agent environments on demand. We define a complete agent environment through four components: an Action Space of 5,571 format-unified tools across 204 common applications, a Task Distribution engine that synthesizes long-horizon workflows with wild constraints, a Transition Function implemented as a state controller that injects realistic failures and perturbations, and a Reward Signal combining verifiable metrics with LLM-based judgment. C-World operates in two modes: a realistic mode grounded in live API execution, and a synthesized mode powered by the World Engine, which approximates tool behavior without live service access, enabling scalable environment creation -- including environments for domains and tools that do not yet exist in the real world. Evaluation of nine state-of-the-art LLMs reveals that planning ability is uniformly strong but execution remains the bottleneck, and that constraint following -- not tool invocation -- is the dominant failure mode. The World Engine achieves Spearman $ρ= 0.883$ ranking correlation with real execution, and fine-tuning on just 1,170 C-World trajectories outperforms baselines trained on 119k samples, demonstrating C-World's dual value as a rigorous evaluation environment and a scalable data engine. Our code and data are available at https://ziqiao-git.github.io/C-World/