CoworkEnv: Synthesizing Training Environments for Long-Horizon Cowork Agents

AI summary

Overview

• Research area: AI agent training — synthetic environment construction for tool-using "coworker" agents, combining supervised fine-tuning (SFT) and reinforcement learning (RL).
• Technical level: Advanced. The paper assumes familiarity with agent harnesses, tool-calling trajectories, LoRA, GRPO-family RL objectives, and grading-based evaluation.
• Scope: A capability-guided pipeline that turns target coworker skills into executable, automatically validated task environments, plus evidence that the resulting demonstrations and rollouts improve compact 8B and 4B models on the Claw-Eval and ClawMark benchmarks.

What This Paper Is About

AI coworkers must coordinate tools across multiple turns while a shared workspace changes underneath them, and training that behavior requires tasks where the request, the underlying service records, the available tools and the scoring code all agree with each other. CoworkEnv addresses the construction bottleneck by having developers specify which capabilities they want, then generating multi-service workflow tasks complete with fixtures, tool interfaces, completion checks and prohibited actions. The same task definitions are then reused twice: teacher executions become curated SFT demonstrations, and student interactions become rollouts with rewards for RL.

Key Contributions

1. Capability-guided synthesis of executable coworker environments. Target capabilities are translated into a coverage plan, and each task is represented as a tuple of request, initial state, tool interfaces and grading checks — instantiated across shared services so that records, distractors, permitted actions and scoring all derive from one specification.
2. Validation and feedback repair built into generation. Programmatic checks verify parseability, service registration, that completion weights sum to one, that required and prohibited tool sets do not conflict, and that answer identifiers are not leaked into the request; a separate grading-consistency gate tests the grader itself against synthetic tool-call and audit records, with failures returned to the generator for regeneration.
3. A dual-use training resource. Teacher executions through a Claude Code adapter in Harbor are curated into SFT demonstrations, while the retained environments serve as RL tasks where the grader computes rewards from the student's own tool calls and service audit records.
4. Empirical training utility and RL method comparison. PocketWorker models are trained and evaluated on Claw-Eval and ClawMark, with a data-scale ablation and a comparison of GRPO, Dr.GRPO, GDPO and GiGPO as reward-processing, advantage-estimation and credit-assignment variants; the authors also commit to releasing datasets and the construction pipeline, plus a "Lessons" write-up.

Main Findings

• Large SFT gain on Claw-Eval. For Qwen3-VL-8B-Instruct, the SFT checkpoint raises the Overall score from 46.6 to 64.1 (a 17.5-point gain), with General rising from 46.7 to 60.9 (+14.2) and Multi-turn from 45.9 to 77.8 (+31.9).
• Reliability improves too. Combined Pass@3 / Pass^3 rises from 28.6 / 21.6 for the base model to 50.8 / 37.2 for the SFT model; Multi-turn alone reaches 78.9 / 50.0 and General 44.1 / 34.2 (threshold 0.75 per trial, three trials).
• The 4B model also benefits. Qwen3-VL-4B-Instruct improves from 28.7 to 36.6 on General (+7.9), 56.1 to 66.0 on Multi-turn (+9.9) and 33.9 to 42.2 Overall (+8.3), though its Pass@3/Pass^3 rates on General move slightly down (−0.6 each).
• Transfer to a second benchmark. With an SFT-only extension to ClawMark (100 tasks, 13 professional domains), PocketWorker-8B scores 19.4 versus 11.8 for the base model, a 7.6-point gain.
• RL after supervised warm-up helps most on multi-turn work. GRPO reaches Overall 56.6 (+10.0) and Multi-turn 72.5 (+26.7); GDPO reaches Overall 57.2 (+10.6) and Multi-turn 74.9 (+29.0); GiGPO reaches the highest Overall (60.7, +14.1) and General (59.7, +13.0) but a lower Multi-turn 64.8; Dr.GRPO is lowest of the four at Overall 54.3 (+7.7).
• Multimodal tasks barely move under RL. Starting from a base of 20.7 on 101 Multimodal tasks, the four methods land between 21.0 and 22.1; the "All tasks" (n=300) figures range from 43.1 (Dr.GRPO) to 47.3 (GiGPO) against a base of 37.9.
• Different RL configurations suit different capabilities. GRPO outperforms GDPO on Multimodal, GDPO edges ahead on Multi-turn, and GiGPO's step-level advantages favor General tasks — the paper reads this as task-dependent rather than uniformly superior configurations.
• More SFT data keeps helping. Training on 20%, 50% and 100% of the data yields Overall scores of 50.8, 55.6 and 64.1 (General 50.1 / 53.4 / 60.9; Multi-turn 53.6 / 64.9 / 77.8), with no saturation observed at the full fraction.
• Training rewards rise under optimization. Reward curves for all four methods increase over roughly three epochs on the constructed tasks, which the authors present as preliminary evidence that the environments support policy optimization; they explicitly caution that policy-loss magnitudes use different normalizations and are not comparable across methods.
• Compact models can approach larger systems on multi-turn tasks. The 8B model's 77.8 on Multi-turn falls within the range of the 14 published Claw-Eval reference models, while the authors note that claiming to beat a specific reference model would require aligned evaluation settings.

Methodology in Plain English

The process starts with a coverage plan written by the developers, which names the services to combine (for example CRM, helpdesk and email), an intended difficulty, a fixed reference date, and a scenario describing selection conditions, distractors and which actions are allowed. An LLM receives this plan plus service catalogs and example tasks and writes four things: the user's request, the starting service records, structured completion checks, and a list of prohibited actions.

Because records are spread across services, tasks require actual investigation. In the escalation example, an agent must find enterprise tickets that have been open more than 48 hours and then pull the account manager's address from CRM to send mail — while a newer high-priority ticket sits in the data as a distractor that looks urgent but fails the age test. For safety tasks, a Generator–Discriminator split separates drafting from scoring: the Generator writes the request and records, and a Discriminator that sees the records plus a safety note (but not the full request) specifies the checks and forbidden tools. One documented payment-change task rewards opening the inbox, reading the suspicious request, reading the legitimate vendor message, listing financial history and inspecting transactions, while calling finance_submit_report or gmail_send_message zeroes the safety factor in the score r = safety × (0.8·completion + 0.2·robustness).

Generated specifications are not trusted blindly. Rule-based checks catch malformed or inconsistent tasks, and a grading-consistency test feeds synthetic tool-call and audit records to the grader: a record that satisfies the completion conditions must score full completion and safety, an empty record must score zero completion, and adding a prohibited call must zero the safety score. Failures go back to the generator as feedback and the whole specification is regenerated.

A teacher model (via a Claude Code adapter in Harbor) then attempts each accepted task, producing sequences of actions and observations that are scored by the grader. Curation is task-aware: tasks the teacher solves consistently keep only one demonstration per normalized task prefix, very low-completion tasks are set aside, and the middle band supplies multiple retained demonstrations. A second conversion stage drops sequences that are too short, contain too many tool calls, or exceed a character budget, and truncates long tool observations.

For SFT, those messages are fed to the student's chat template with LoRA (rank 32, scaling 64, dropout 0.05, sequence length 4,096), supervising assistant responses and tool calls while treating user messages and observations as context. For RL, the student interacts with the same environment online and the grader scores its own calls and audit records; invalid or missing rewards abort the sample and incomplete rollout groups are retried before advantage estimation.

Why This Matters

The paper treats environment construction and verification — not just data filtering — as the binding constraint on coworker-agent training, and it shows a single specification can feed both imitation and interaction-based learning. That is a practical template for labs that cannot hand-build hundreds of consistent multi-service sandboxes.

Real-world applications:

• Customer support and IT operations: multi-service triage where account history, ticket state and internal procedure documents must be reconciled before acting.
• Security and compliance review: investigation tasks where the correct behavior is gathering evidence and withholding a destructive action such as changing vendor banking details or sending mail.
• Enterprise back-office workflows: escalation routing, ticket prioritization and record updates that span CRM, helpdesk and mail systems.
• Training compact on-premises workers: an 8B model specialized cheaply enough for settings where frontier API calls per task are impractical.

Industry relevance is direct: the paper reports that advanced long-horizon agentic SFT and RL are hard for academic groups because of limited GPU access, infrastructure gaps and high API costs, and it targets exactly that gap with small models, LoRA, and reusable task packages. The finding that RL gains concentrate on Multi-turn tasks while Multimodal tasks stay flat also gives practitioners a concrete signal about where policy optimization pays off and where it does not.

Future Directions

• Broaden what graders can check. The authors state that execution scores only measure what the graders encode, so errors outside those checks — including whether the agent semantically recognized fraud or explicitly refused — remain unmeasured.
• Rethink the teacher-success filter. Filters can favor redundant demonstrations and discard useful recovery behavior; the paper points to work on iterative skill evolution suggesting that feedback containing failures can produce useful skill updates, and flags that regenerating original teacher trajectories may require access to a proprietary model service.
• Separate reward granularity from advantage estimation. The current comparisons change multiple factors at once (Dr.GRPO varies both advantage and loss normalization; GDPO varies reward processing and batch size; GiGPO is evaluated without a matched ω = 0 ablation), so cleaner single-factor studies are an open need.
• Close the training–inference mismatch across harnesses. The Lessons material names harness effects on training outcomes and mismatch between training and inference harnesses as unresolved issues, alongside credit assignment and RL optimization.

Target Audience

Researchers and engineers building agentic post-training pipelines — particularly groups with modest compute who want to fine-tune compact (4B–8B) models for tool-using, multi-turn work. It is also useful for benchmark designers interested in how grading checks, distractor records and safety gates are validated, and for practitioners who need to reason about which RL configuration to try first for a given task category. Readers without background in agent harnesses, LoRA and policy-gradient methods will find the appendices necessary rather than optional.

Authors’ abstract

AI coworkers must coordinate tools, communicate across turns, and act on changing workspace state while respecting workflow constraints. Constructing training experience for these capabilities requires coherent requests, service records, tool interfaces, and grading checks. We present CoworkEnv, a capability-guided framework for synthesizing executable training environments for AI coworkers. Workflow requirements guide task construction, and programmatic validation with feedback repair checks generated specifications and grading behavior. Teacher executions yield curated demonstrations for supervised fine-tuning (SFT), while student interactions provide rollouts and rewards for reinforcement learning (RL). Using these resources, we train compact PocketWorker models. For Qwen3-VL-8B-Instruct, SFT raises the mean score from 46.6 to 64.1 on Claw-Eval, including a 31.9-point gain on Multi-turn tasks. An SFT-only extension to ClawMark demonstrates applicability across coworker benchmarks. We further compare RL training dynamics across configurations that vary reward processing and credit assignment. Improving training rewards provide preliminary evidence that the constructed environments support policy optimization.