EnvHarness: Awakening Static Worlds for Agent Learning

AI summary

Overview

Research area: Large language model (LLM) agents, agent learning signals, automated environment generation, and reinforcement learning for agents.

Technical level: Advanced. The paper formalizes environments as tuples, defines interface-level transformations, and evaluates skill-based learning and online RL pipelines.

Scope: EnvHarness is a programmable layer of plug-in components that wraps a static environment and reshapes its initial states, interaction rules, and task composition, with EnvRigger automating the component design from observed policy rollouts. Experiments cover five benchmarks in four domains (ALFWorld, WebArena, SWE-bench Verified, OfficeQA, SpreadsheetBench), plus reinforcement learning on ALFWorld and WebShop.

What This Paper Is About

Agents learn by interacting with environments, but those environments are hand-built and static: they behave identically no matter which agent uses them, so they never target that agent's specific weaknesses and stop teaching anything once the agent can solve the existing tasks. Existing automated environment generation does not solve this either, because the pipelines are domain-specific and their LLM-generated verifiers are expensive or unreliable. The paper's goal is to customize the environment side of the agent-environment loop without rebuilding environments from scratch and without touching the original human-built tasks or verifiers.

Key Contributions

1. EnvHarness, a programmable layer that wraps a static environment and customizes it through the environment's own reset/step interface, without modifying the underlying logic. It is instantiated as three plug-in component types: Stage (changes the initial state), Contract (rewrites actions, observations, and transition dynamics), and Chain (connects multiple base environments into an extended episode). Every reshaped environment retains the original environment's verifier.

2. EnvRigger, an automated procedure for task-policy-conditioned environment customization. It treats the target policy as a black box and runs four stages — Observe, Diagnose, Write, and Validate — where the last two form a write-and-validate loop that generates a candidate EnvHarness component, tests it on fresh policy rollouts, and revises it on failure.

3. Evidence across five benchmarks in four domains that EnvHarness outperforms both original environments and domain-specific environment generation pipelines, improving held-out performance by up to 9.0 points with 9.8% fewer execution steps.

4. Demonstrations that EnvHarness serves as an optimization signal beyond skill extraction, including online RL training (up to 6.5 points of improvement in Table 4) and environment scaling where EnvHarness keeps improving to 300 environments while real and generated environments flatten out.

Main Findings

• Skill-based learning gains are consistent. Skills extracted from EnvHarness-customized environments outperform skills from original environments on every benchmark, with improvements of +2.9 (in-distribution), +9.0 (out-of-distribution), and +5.9 (average) on ALFWorld; +1.9, +2.2, +6.2, +2.3, and +3.1 across WebArena's Reddit, Shopping, Shop Admin, GitLab, and average; +2.70 success rate on SWE-bench Verified; +1.80 EM and +1.96 F1 on OfficeQA; and +3.27 Pass@1 and +1.01 mean score on SpreadsheetBench.

• Static environments can actively hurt. On SpreadsheetBench, skills from unmodified environments (46.44 Pass@1, 61.32 mean score for no skills; 45.88 and 61.47 for original environments) fall below the no-skill baseline, and on SWE-bench Verified they lengthen execution trajectories from 53.58 to 55.01 average steps.

• EnvHarness beats specialized generators where they apply. On ALFWorld, EnvHarness (68.3 average) surpasses GenEnv (62.6 average) by 5.7 points on average and by 8.5 points on out-of-distribution instances (70.4 versus 61.9). On SWE-bench Verified, it outperforms SWE-smith by 2.46 points in success rate (52.58 versus 50.12) while requiring 5.11 fewer execution steps per episode (49.61 versus 54.72).

• Efficiency follows from targeted repairs. On SWE-bench Verified, EnvHarness skills reduce average steps per episode from 53.58 to 49.61, whereas skills from unmodified environments raise it to 55.01. The paper attributes this to Contracts and Stages that disrupt repetitive action loops and filter verbose observations.

• EnvHarness works as an RL training signal. Using Qwen3-8B-base optimized with GRPO, training on EnvHarness environments beats training on original environments on three of four metrics: ALFWorld in-distribution success rate 87.9 versus 81.4, ALFWorld average 88.4 versus 85.5, WebShop score 79.2 versus 75.6, and WebShop success rate 67.4 versus 66.0. The only regression is ALFWorld out-of-distribution success rate, 88.8 versus 89.6, which the paper calls slight and negligible.

• Chaining improves long-horizon efficiency. Chain-only skills cut average steps from 53.58 to 41.96, with a standalone success rate of 49.63 that is marginally below the 49.88 original-environment baseline. Combining Stage/Contract skills with Chain skills produces the best results: 54.30 success rate and 43.12 average steps.

• EnvHarness scales where other environment sources flatten. On SWE-bench Verified, under an identical budget where each batch of 50 environments yields one skill bank (alternating 2 and 3 skills per bank, totaling 15 skills at 300 environments), EnvHarness climbs from 47.67 to 54.79, a 7.12-point gain, and is still rising at 300 environments. The same budget yields 52.13 on original environments and 50.37 on generated ones.

• Gains hold across model backbones. Testing four policies on SWE-bench Verified — Gemini 3.1 Flash-Lite, Qwen3.6 27B, Gemini 3.5 Flash, and Claude Sonnet 4.6 — EnvHarness skills beat real-environment skills on all four by 2.7 to 3.7 absolute points, even though skill-free success rates span 30.7 to 67.2. The two weakest policies gain most: +9.3 and +11.1 points for EnvHarness and +6.1 and +7.4 for unmodified environments, versus under 5.5 points for the two strongest.

• EnvRigger accepts explicit user constraints. Given a natural-language weakness ("the policy submits a patch without running the failing test, so the fix stays unverified"), EnvRigger generated a Contract that rejects submissions unless tests were run, and the resulting trajectories yielded a generalizable "Verification-Driven Development Loop" skill.

Methodology in Plain English

The researchers draw an analogy with the agent harness: just as an agent harness adds tools, memory, and execution loops around a frozen model without changing its weights, EnvHarness adds plug-in components around a frozen environment without changing its logic. Formally, a component is a transformation that takes an environment and returns a reshaped environment, altering only the initial state, the exposed action and observation spaces, and the transition mechanics, while leaving the original verifier intact so scoring still works.

Three component types cover the customization modes: Stage replays a short sequence of actions on the freshly reset state (for example, hiding a mug in a drawer so the agent must search, or cleaning the mug in advance so the task is shorter); Contract rewrites the action space, transition dynamics, and observations (for example, truncating room descriptions to two sentences, blocking a cleaning action until the mug is held, or removing teleport navigation); and Chain joins a second environment with a composition rule so the agent must satisfy both, such as appending "heat a potato and put it on the countertop" to the mug task. Because all components share a standard interface, they stack, and the stacking order matters since the transformations do not commute.

To pick and parameterize components automatically, EnvRigger runs the target policy on a base task, observes both successful and failed trajectories, and produces a textual diagnosis of systemic flaws such as repetitive action loops, failure to parse long observations, or misread tool constraints. It then writes candidate components matching the diagnosis — making the task harder when the policy already succeeds at 100%, or scaffolding it when the policy struggles — and validates them by re-running the policy in the wrapped environment on fresh rollouts. Candidates are accepted, rejected as unsolvable or non-challenging, or sent back for revision until accepted or until the revision budget runs out. Across the main experiments, EnvRigger and the policy use the same backbone: Gemini-3.1-Flash-Lite for ALFWorld and WebArena and Gemini-3.5-Flash elsewhere, so gains do not come from distilling a stronger model. Skills are then extracted from trajectories in the customized environments following ReasoningBank and evaluated on strictly disjoint held-out episodes. The Chain component is excluded from the automated pipeline because EnvRigger cannot easily observe the internal states of joined environments, so chaining is analyzed separately.

Why This Matters

Impact on research. The paper reframes environment customization as an interface-level, domain-agnostic operation that preserves human-built verifiers, in contrast to domain-specific generation pipelines that require over-generation and heavy filtering. It also positions environment reshaping as part of self-evolving agents, where the world the agent learns from changes, rather than only the agent's prompts, skills, memory, or weights.

Real-world applications.

• Software engineering agents: verifying fixes, running test suites before submitting patches, breaking repetitive debugging loops, and shortening execution trajectories on repository-level tasks.
• Office automation: improving spreadsheet manipulation and document question answering, where the paper reports gains on SpreadsheetBench (+3.27 Pass@1) and OfficeQA (+1.80 EM).
• Web agents: web navigation across Reddit, shopping, shop administration, and GitLab-style sites, including environments with no existing generation baseline.
• Embodied and household task agents: text-based embodied control in ALFWorld, where the largest reported gain appears (+9.0 out-of-distribution).

Industry relevance. The work comes from Google Cloud AI Research with academic collaborators at UNC Chapel Hill, Google Cloud, and Washington University in St. Louis, and the code and project site are public (github.com/google-research/envharness and www.envharness.com). For teams training agents, EnvHarness offers a way to reuse existing benchmarks and their trusted verifiers instead of building new simulation stacks, and the reported step reductions (9.8% fewer steps, 53.58 to 49.61 on SWE-bench Verified) map directly onto inference and rollout cost.

Future Directions

• Automating Chain generation. The paper excludes Chain from the EnvRigger pipeline because observing internal states of joined environments is difficult, which leaves automated long-horizon composition as an open problem.
• Scaling the co-evolution further. Environment scaling was measured up to 300 environments with EnvHarness still improving; whether the trend continues, saturates, or requires new component types at larger budgets is not established.
• Broadening the component vocabulary. The paper presents three components chosen to cover three fundamental modes and states that more are expected to follow, leaving the design space of additional interface-level transformations unexplored.
• Extending domains and backbone coverage. Evaluation covers five benchmarks in four domains and four policy backbones; generalization to other agentic settings, verifier types, and constraints — including the quantitative target constraints the paper only places in an appendix — remains to be tested.

Target Audience

Researchers and engineers working on LLM agents, agentic reinforcement learning, and environment or benchmark construction, particularly those interested in training-signal design rather than model architecture. It is also relevant to practitioners who already own working environments with reliable verifiers and want to squeeze more learning signal out of them without rebuilding simulation stacks. Readers without background in reinforcement learning, environment interfaces, or agent training loops will find the formalism in Section 2 heavy going.

Authors’ abstract

LLM agents learn by interacting with environments, yet these environments are hand-built and static: blind to an agent's weaknesses, and quickly left behind as it improves. While recent environment generation methods attempt to address this, they require domain-specific pipelines, rely on expensive or unreliable verifiers, and still produce static environments. To alleviate the engineering burden of rebuilding environments from scratch, we propose Environment Harness (EnvHarness), a programmable layer of plug-in components that wraps a static environment to reshape its behavior without modifying the underlying logic. Operating through standard interfaces, EnvHarness applies across diverse domains while ensuring every reshaped environment retains its original verifier. To automate this process, we introduce EnvRigger, which treats the target policy as a black box, observing its execution trajectories to synthesize EnvHarness components targeting diagnosed flaws, and validating them via fresh rollouts. Across five benchmarks in four domains, EnvHarness outperforms both original environments and domain-specific environment generation pipelines, achieving up to a 9.0-point improvement on held-out instances with 9.8% fewer execution steps. Furthermore, EnvHarness provides a superior optimization signal for reinforcement learning, enabling continuous, targeted co-evolution of the policy and its environment.