WorkRover: Scaling Long-Horizon Cowork Agent Training Via Recursive Environment Evolution

AI summary

Overview

Research area: Reinforcement learning for long-horizon "cowork" LLM agents that operate across files, tools, and applications, with a focus on automated synthesis and iterative repair of training environments.

Technical level: Advanced (assumes familiarity with reinforcement learning, GRPO, credit assignment, agent environments, and verifier-based evaluation).

Scope: The paper introduces a framework — WorkRover — that combines harness-driven environment synthesis, a rollout-driven repair method called Factorial Compatibility Replay (FCR), and a turn-level credit-assignment method called Verifier-to-Turn Credit Assignment (VTCA), then post-trains Qwen3.5-35B-A3B and evaluates the result on five external cowork-agent benchmarks.

What This Paper Is About

Training long-horizon professional agents with reinforcement learning requires environments that are executable, internally consistent, and reliably scored. The authors argue that simply generating more synthesized environments is not enough: work contexts, tasks, and verifiers frequently fall out of alignment after initial construction, so a task may look unsolvable or a wrong answer may be rewarded. WorkRover addresses this by repeatedly using rollout feedback to diagnose and repair environments (recursive environment evolution) while also giving the policy finer-grained per-turn learning signals derived from verified workspace progress.

Key Contributions

1. Harness-driven construction of semantically aligned environments. A versioned Harness jointly constructs work contexts, executable tasks, and evidence-grounded verifiers, explicitly tracking their dependencies and validating task feasibility (via independent reference executions from a clean workspace) and verifier consistency (rejecting negative cases while accepting valid alternatives).

2. Recursive environment evolution via Factorial Compatibility Replay (FCR). FCR freezes one proposed repair per eligible component (work context, task, harness, verifier), enumerates combinations (for d ≤ 4 eligible components, 2^d combinations, at most 16), expands each into a dependency-closed update set, replays them under matched conditions, and selects the smallest validated repair that preserves task requirements and evaluation constraints.

3. Training from verifier-grounded progress via Verifier-to-Turn Credit Assignment (VTCA). Frozen deterministic checks measure workspace progress and setbacks between turns; positive increments are distributed with at least half staying on the producing turn and the remainder shared among prerequisite turns whose tool outputs or artifact versions were directly used, with a local trace cost for tool errors or unproductive repetition. No intermediate judge calls are used.

4. A post-trained model and controlled evaluations. Qwen3.5-35B-A3B is post-trained to produce WorkRover-35B-A3B, with SFT followed by VTCA-GRPO, plus ablations isolating repair selection, joint-repair necessity, and construction components.

Main Findings

• Benchmark gains over the backbone: WorkRover-35B-A3B exceeds Qwen3.5-35B-A3B on all five external benchmarks: JobBench +26.27 percentage points (10.62 → 36.89), OfficeQA Pro +29.32 points (19.55 → 48.87), AutomationBench +22.67 points (6.00 → 28.67), WorkBuddy Bench (Office) +13.63 points (60.50 → 74.13), and GDPval v2 +413.25 Elo (804.75 → 1218.00).

• Best among listed 27–35B open-weight models: WorkRover has the highest reported score on every benchmark within that group; Occamy-1.0 is second on each. The largest margin over Occamy-1.0 is on WorkBuddy Bench (Office): 74.13% versus 66.65% (7.48 points), followed by JobBench 36.89% versus 34.27% (2.62 points), AutomationBench 1.07 points, and OfficeQA Pro 0.77 points.

• Remaining gaps against larger and proprietary models: JobBench 36.89% is below GPT-5.5 (38.30%) and Kimi K3 (54.27%); GDPval 1218.00 Elo is below all listed open-weight models above 400B (1303.96–1572.42 Elo); WorkBuddy Bench (Office) 74.13% is below GPT-5.5 (86.05%) and Claude Opus 4.7 (92.51%). WorkRover exceeds GLM-5.2 and MiniMax-M3 on both AutomationBench and OfficeQA Pro, and Kimi K3 on WorkBuddy Bench (Office) (74.13% versus 72.18%).

• FCR improves repair under matched budgets: On naturally occurring defects, FCR raises Safe Resolution Rate (SRR) from 60.7% with full regeneration to 70.3%, reduces changes to task intent from 14.3% to 1.3%, and uses fewer model tokens (relative cost 1.00 versus 1.34).

• Repair selection matters separately from proposal generation: With proposals fixed, FCR raises SRR by 9.0 percentage points on natural defects and 10.4 points on injected defects versus Apply-all, and reduces cross-task regression from 7.0% to 0.7%, at a higher relative model-token cost (1.00 versus 0.59). In the frozen-proposal maintenance comparison, FCR reaches 73.0% natural SRR and 81.2% controlled SRR with 0.7% task regression, compared with Apply-all (64.0%, 70.8%, 7.0%), Greedy-sequential (58.0%, 63.1%, 2.7%), and Local-only (42.0%, 45.4%, 3.2%).

• Joint repairs are often necessary: With proposals fixed, 48–74% of safely resolved cases require joint edits, depending on the component pair, meaning no smaller subset of the selected edits passes the same repair checks.

• Stability across generations: Across three repair generations, SRR stays at 70.3–72.4% and sealed-audit pass rate at 93.8–94.6%. Task success rate also increases over five repair iterations.

• FCR improves initial construction: With the candidate set and repair budget caps fixed, synthesis pass rate reaches 74.8%, compared with 49.8% before repair and 68.8% with full regeneration.

• VTCA beats episode-only GRPO: With SFT checkpoint, task and seed schedules, and rollout budget fixed, VTCA outperforms episode-only GRPO (E-GRPO) on all five benchmarks. The largest percentage-point gains are on AutomationBench (+5.7) and OfficeQA Pro (+5.4), with GDPval v2 improving by 43 Elo. Both RL methods improve over SFT across all five benchmarks.

• Construction ablations show the value of dependency-aware assembly: Full WorkRover achieves 74.8% task yield, 95.1% reference execution, and 58.7% success rate. Removing relation-aware expansion drops these to 52.6%, 77.9%, and 41.8%; removing validation-guided rewriting gives 49.8%, 80.6%, and 44.5%; removing file–skill grounding gives 60.9%, 84.2%, and 47.4%; removing job–skill grounding gives 67.2%, 91.7%, and 53.1%; and a semantic-only baseline gives 48.7%, 73.6%, and 38.6%.

• Rewriting control on identical task intents: On the same 10,000 initial task intents with the same frozen agent, adding rewriting raises admission from 49.8% to 74.8% and end-to-end success from 22.2% to 43.9%. Reference execution rises from 80.0% to 92.0% and verifier validity from 70.0% to 87.0%. Unconstructed, rejected, and budget-exhausted intents remain in the denominator.

Methodology in Plain English

The approach has three parts.

Building environments. Files, skills/plugins, and mock or installed applications are collected and profiled, then assembled by a "Harness" into a task package made of three pieces: the work context (files, capabilities, apps), the task (goals, deliverables, constraints), and the verifier (the checker). Construction agents adapt their actions to workspace state and execution feedback rather than following a fixed workflow. An independent Demonstrator Agent runs each task from a clean starter workspace to confirm it is feasible and to produce a reference state. The verifier is derived from an evaluation contract fixed before the reference state is inspected, so that scoring criteria are bound to evidence in files, app states, or state differences. Verifiers are then calibrated to reject the starting state and rule-violating cases while accepting the reference state and independently obtained valid alternatives, and are frozen for a one-time admission test.

Repairing environments. When rollouts reveal a mismatch, diagnostics first separate environment defects from agent mistakes (agent mistakes alone do not trigger environment change). One repair proposal is frozen per eligible component, and FCR tests individual and joint combinations through dependency-closed rebuilds and replays under matched policies, budgets, and seeds. Resolution must be confirmed by independent checks that preserve required outcomes, information boundaries, and other affected tasks — never by agent success or scores from the proposed verifier. Among passing combinations, FCR picks the one whose full update set changes the fewest component versions, with ties broken by fewer rebuilt dependencies and then lower cost. Fresh rollouts with new seeds confirm the selection.

Training. The policy is initialized with supervised fine-tuning on verified demonstrations, then trained with RL on the same task and initial state. Rewards combine verifier outcome scores and trace scores; any critical, integrity, protected-state, or safety failure zeroes the reward. VTCA adds per-turn shaping to the group-standardized trajectory advantage: frozen deterministic checks score supported criteria before and after each turn, at least half of each positive increment stays on the producing turn, and the remainder is split among prerequisite turns whose tool outputs or exact artifact versions were directly used. Negative increments stay fully local, and unfinished rollouts can retain progress credit while violations disable it for every turn. Clipped GRPO averages token contributions within each turn and turns within each trajectory, so long turns do not dominate. Setting the two shaping weights to zero recovers the episode-only baseline.

Why This Matters

Impact on research. The paper reframes environment construction as an ongoing, feedback-driven process rather than a one-shot generation step, and provides concrete mechanisms (dependency-closed repair selection, frozen-verifier checks, workspace-grounded turn credit) that other long-horizon RL pipelines can adopt. It also contributes controlled evidence that joint edits are frequently necessary and that topic similarity alone is insufficient for assembling executable workspaces.

Real-world applications:

• Training agents for occupational and office workflows such as assembling professional deliverables across documents and applications.
• Document-grounded question answering over enterprise material, where verifiers must distinguish correct answers from plausible but unsupported ones.
• Cross-application automation, where an agent must carry state between tools and recover from intermediate errors.
• Automated maintenance of existing agent-task suites, where defects surface only after deployment and repairs must avoid changing original task intent.

Industry relevance. The results indicate that a 27–35B open-weight model, post-trained this way, can outperform its backbone by large margins and lead its size class on all five evaluated benchmarks — relevant for teams that need capable cowork agents without relying on proprietary models. The paper also reports a relative token cost for repair strategies, which matters for the economics of continuously maintaining large environment collections, and it explicitly distinguishes infrastructure-invalid episodes from policy failures, a practical requirement for production evaluation pipelines.

Future Directions

• Verifier coverage and dependency quality. The paper states that the reliability of VTCA's shaping signal depends on checker coverage and on how complete the dependency records are; improving both is an open problem.
• Global minimality of repairs. FCR's bounded enumeration over at most 2^d combinations does not establish a global minimum over all possible repairs, leaving room for more exhaustive or principled search.
• Closing the gap to larger and proprietary models. WorkRover still trails models such as Kimi K3 and DeepSeek-V4-Pro on JobBench and remains below all listed open-weight models above 400B on GDPval, and below GPT-5.5 and Claude Opus 4.7 on WorkBuddy Bench (Office).
• Scaling evolution and training data. The framework was exercised with 30,115 SFT trajectories, 20,000 synthesized task–verifier pairs narrowed to 3,200 by consistency filtering, three repair generations, and five repair iterations; how far repair quality and gains hold under much longer evolution cycles is not reported.

Target Audience

Researchers and engineers working on reinforcement learning for LLM agents, automated environment and benchmark construction, verifier design, and long-horizon credit assignment. It is most useful to readers already comfortable with RL training loops and agent tooling; those new to the area would need background in GRPO-style policy optimization, agent trajectories, and evaluation harness design to follow the method sections.

Authors’ abstract

Scaling interactive environments is important for training long-horizon agents through reinforcement learning. These environments often require further testing and refinement after initial generation, and correcting defects in a task may also require updating its work context and verifier. However, existing construction pipelines offer limited support for such iterative refinement while keeping these components aligned. We introduce WorkRover, a framework for training long-horizon cowork agents through recursive environment evolution. Its harness-driven synthesis enables construction agents to jointly build and validate semantically aligned work contexts, executable tasks, and evidence-grounded verifiers, adapting their actions to workspace state and execution feedback rather than following a fixed construction workflow. To support environment evolution, we introduce Factorial Compatibility Replay (FCR), which uses rollout feedback to evaluate combinations of component revisions through dependency-closed replay and selects the smallest validated repair among tested candidates while preserving task requirements and evaluation constraints. To provide finer-grained training signals, we propose Verifier-to-Turn Credit Assignment (VTCA), which augments trajectory-level GRPO advantages with signals from deterministic workspace-progress checks and assigns a bounded share of positive-progress credit to prerequisite turns using recorded tool-output and artifact dependencies. Using WorkRover, we post-train Qwen3.5-35B-A3B to obtain WorkRover-35B-A3B and evaluate it on several cowork-agent benchmarks. Its reported scores exceed those of Qwen3.5-35B-A3B by 26.27 percentage points on JobBench, 29.32 percentage points on OfficeQA Pro, and 413.25 Elo points on GDPval v2. These results suggest that WorkRover offers a promising foundation for scalable training of long-horizon cowork agents.