EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning

AI summary

Overview

• Research area: Agentic reinforcement learning for large language model (LLM) agents, long-horizon tool use, and learned world models.
• Technical level: Advanced — the paper formalizes tool-interactive tasks as a POMDP and builds on GRPO with a role-wise advantage estimator and a multi-stage test-time procedure.
• Scope: The paper introduces EnvACE, an agentic RL method that removes external environment interaction from training by having a single shared policy alternate between acting and rehearsing the environment responses its actions would induce.

What This Paper Is About

Training LLM agents for long-horizon tool use normally requires either real or synthesized executable environments — which are costly to build and verify — or external simulators whose responses are hard to ground. EnvACE instead lets the policy itself play both roles: it issues a tool call, then generates the environment response that call would induce, and conditions its next decision on that self-generated response. The goal is to internalize the action-to-response relationship into the policy's own parameters, producing an "agent world model" that supports both better training and additional test-time computation.

Key Contributions

1. World rehearsal: A training scheme in which the policy takes on the role of the environment, generating the response induced by an agent action, so environment dynamics are internalized without querying an external environment.
2. EnvACE: An agentic RL method that interleaves acting with world rehearsal in self-unfolded trajectories, using a single shared policy for both roles and optimizing them jointly end-to-end with task-success rewards.
3. Role-wise GRPO: An optimization scheme that computes a separate reward baseline and advantage for each role (Act and Rehearse) while outputs from both roles jointly update the shared policy parameters.
4. Test-time scaling via private rehearsal: After training, the policy performs N private rehearsal attempts (in parallel or sequential mode), summarizes them into a rehearsal memory, and uses that memory to guide a single committed execution in the external environment.

Main Findings

• Overall benchmark performance: EnvACE achieves an Overall score of 32.91%, outperforming all environment-scaling baselines with complete results on the three benchmarks, surpassing EnvScaler-8B by 0.99% and AWM-14B by 0.37%.
• BFCL V4: EnvACE reaches 46.04%, outperforming Qwen3-8B by 2.00% and AWM-8B by 1.75%, while remaining within 1.03% of EnvScaler-8B.
• τ²-Bench: EnvACE obtains the second-highest average of 36.7%, exceeding EnvScaler-8B by 3.8%, AWM-8B by 5.5%, and AWM-14B by 6.0%.
• VitaBench: EnvACE achieves 16.0%, the best average among all 7B–8B methods, outperforming EnvScaler-8B by 0.2% and ScaleEnv-8B by 1.0%.
• FinMCP-Bench: EnvACE achieves the best TF1 score of 46.78%, outperforming EnvScaler-8B by 3.10% and AWM-8B by 4.28%, and obtains the highest tool precision of 54.04%.
• World rehearsal beats standard GRPO: In a controlled comparison at the 8B scale on τ²-Bench, EnvACE improves the average score from 31.2% to 36.7%, a gain of 5.5%.
• Parameter sharing matters: Against a Per-role Policy variant with separate policies for acting and rehearsal, sharing parameters improves the τ²-Bench average from 35.5% to 36.7%, a gain of 1.2%.
• Training dynamics: The offline evaluation score rises from 30.0% at step 50 to 36.7% at step 470, with fluctuations at intermediate checkpoints that recover later in training.
• Scaling with model capacity: Moving from 1.7B to 8B improves the BFCL V4 average from 31.81% to 46.04% (a gain of 14.23%) and the τ²-Bench average from 15.3% to 36.7% (a gain of 21.4%), with EnvACE outperforming standard GRPO at both scales.
• Test-time scaling (N = 2): Parallel rehearsal with EnvACE achieves the best Overall score of 40.9%, improving the Non-TTS result of 36.7% by 4.2%; the τ²-Bench average rises from 31.4% to 38.0% and BFCL Multi-Turn from 41.9% to 43.9%. Sequential rehearsal with EnvACE reaches an Overall of 38.5%.
• The rehearsal policy matters: Using EnvACE for rehearsal consistently beats using the base model; the base model yields only marginal improvement in parallel mode and underperforms Non-TTS in sequential mode, indicating gains come from internalized environment knowledge rather than extra inference compute alone.
• Rehearsal budget: Increasing N from 1 to 2 improves performance across all evaluated configurations; at N = 3 performance decreases relative to N = 2 but stays above base-model rehearsal in both modes, which the authors suggest may stem from longer inputs approaching or exceeding the effective context range.
• Qualitative behavior: In the case study, EnvACE anticipates a potential tool-call failure and repairs invalid parameters before execution, and in a flight-reservation scenario predicts that a write operation would fail and replaces it with a safe read-only query, whereas EnvScaler8B and Vanilla agents execute the invalid write and fail.

Methodology in Plain English

The researchers start from the standard picture of a tool-using agent as a partially observable Markov decision process: the policy picks an action, the environment returns an observation, and the history grows. Normally that observation comes from outside — a real environment or a separate simulator — which means environment modeling lives outside the acting policy.

EnvACE moves that job inside the policy. A single shared policy is given two role tags. In the Act role it produces a tool call; in the Rehearse role it produces the environment response that the tool call would elicit. The rehearsed response is appended to the history exactly as if a real environment had returned it, and the policy then acts again on that self-generated observation. This alternation continues until the trajectory terminates, so training rollouts unfold without any external environment.

For optimization, each instruction gets a group of rollouts, each with a trajectory-level reward from a verifiable outcome evaluator or a checklist-based LLM judge. Every policy output in a rollout inherits that rollout's reward, but advantages are computed relative to a reward baseline calculated separately for each role — averaging over all outputs generated under that role across the group. The shared policy is then updated with the clipped GRPO objective, so the rehearsal behavior and the acting behavior shape the same parameters.

At test time, the trained policy rehearses before committing. It runs N private attempts starting from the same task context. In parallel mode the attempts are independent; in sequential mode each attempt sees the previous rehearsal trajectories plus their assessments and revision suggestions. Each attempt is self-evaluated into feedback, and all attempts are summarized into a compact rehearsal memory that conditions one real execution. The rehearsals are private and do not change the external environment.

Implementation specifics reported: the main experiments use Qwen3-8B trained on the dataset introduced by CM2, for 470 training steps with a learning rate of 1×10⁻⁶, batch size 16, and four rollouts per prompt; KL coefficient 1×10⁻⁴ and entropy coefficient 0.0; 64 instances sampled per training step; maximum input and response lengths of 12,000 and 8,000 tokens; up to 30 interaction turns per trajectory; and Qwen3-30B-A3B as the LLM judge. Non-TTS results are reported as Avg@4 averaged over four independent runs, while TTS results come from a single run due to computational cost. In the test-time-scaling table, both acting and rehearsal roles use sampling temperature 1.0 and top-p 1.0; other roles use temperature 0.01. Outside that table, the acting role uses temperature 0.6 and top-p 0.95. Training uses the verl framework on 16 NVIDIA H20 GPUs. Baselines include Qwen3 at 1.7B, 4B, and 8B, Qwen3 trained with standard GRPO, Simulator-8B, TOUCAN-7B, EnvScaler-8B, AWM-8B/14B, and ScaleEnv-8B.

Why This Matters

• Impact on research: The paper reframes environment modeling from an external dependency — real environments, synthesized executables, or separate simulators — into a role of the policy itself, and shows that the resulting internalized world model can also be spent at inference time. This offers an alternative axis for scaling agent training that does not require building or verifying more environments.
• Function-calling and API assistants: Evaluated on BFCL-v4 across single-turn, multi-turn, and agentic tasks, and on FinMCP-Bench through the Model Context Protocol, the method targets the everyday problem of an assistant choosing and parameterizing tool calls correctly.
• Customer-service and transactional agents: τ²-Bench covers Retail, Telecom, and Airline environments, where actions depend on state observed in prior turns — precisely the setting where anticipating an environment response before acting matters.
• Everyday service workflows: VitaBench includes food delivery, in-store consumption, online travel, and cross-domain scenarios, describing consumer-facing tasks that combine dialogue and tool use.
• Safer action selection: The reported case study shows EnvACE avoiding a forbidden write operation by rehearsing it internally first, which is directly relevant to agents that can mutate real state.
• Industry relevance: The work is a collaboration spanning Tencent, Zhejiang University, the National University of Singapore, Sun Yat-sen University, Central South University, The Chinese University of Hong Kong, and Shanghai Jiao Tong University, with code released at a public repository, and its training setup (Qwen3-8B, verl, 16 NVIDIA H20 GPUs) reflects a practical industrial-scale recipe.

Future Directions

1. Larger backbones: The authors state that, due to computational constraints, EnvACE was evaluated only up to the 8B scale, leaving effectiveness on larger models for future investigation.
2. Broader agentic settings: The current evaluation focuses primarily on tool-interactive tasks; extending world rehearsal to a wider range of agentic settings remains an open direction.
3. Rehearsal budget and context limits: Performance drops from N = 2 to N = 3 and the authors attribute this to rehearsal trajectories lengthening the input toward or beyond the effective context range, so managing rehearsal length is an unresolved practical question.
4. Grounding the internalized world model: The paper positions world rehearsal as an alternative to grounding against real environments; how the self-generated responses stay accurate without any external verification is not resolved here.

Target Audience

Researchers and engineers working on LLM agent training, reinforcement learning with verifiable rewards, and tool-use agents will benefit most, since the paper assumes familiarity with POMDPs, GRPO, and multi-turn agent rollouts. Practitioners building production agents that call APIs or modify state — particularly in customer service, travel, retail, telecom, and finance — will find the test-time rehearsal mechanism and the failure-anticipation case studies directly applicable. Readers looking for an accessible introduction to agentic RL may find the technical level high, but the core idea of a policy that plays both actor and environment is conveyed clearly through the paper's overview figure.

Authors’ abstract

Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is publicly available at https://github.com/Within-yao/EnvACE.