EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents

AI summary

Overview

Research area: Reinforcement learning for LLM-based agents that interact with users over multiple turns (user-centric agentic RL).

Technical level: Intermediate — the paper assumes familiarity with RL basics (PPO, GAE, rewards, rollouts) and with LLM agent benchmarks, but its central idea is explained conceptually and is accessible to a motivated non-specialist.

Scope: The paper introduces EAU, a meta-agent that diagnoses and rewrites the user simulator, environment, and training configuration of an ongoing RL run for multi-turn LLM agents, and reports gains on three benchmarks (ColBench, UserGym, τ²-bench) with two Qwen backbones.

What This Paper Is About

When an LLM agent is trained with reinforcement learning to hold multi-turn conversations, the learning signal comes from three things at once: the agent, a simulated user, and an environment with tools and verifiers. The paper argues that all three of these are usually frozen during training, so a stochastic or inconsistent user simulator, a broken tool, or a wrong verifier produces rewards that are noisy or outright misleading, and no amount of reward reshaping downstream can fix that. EAU's goal is to move part of the fix upstream: an LLM-based "meta-agent" watches the training run, diagnoses what is broken or uninformative, and edits the simulator, environment, and RL settings while training continues.

Key Contributions

1. Problem identification: The paper names fixed user simulators and fixed environments as a key, previously under-addressed bottleneck in interactive-agent RL, distinct from the more common focus on reshaping the reward signal.

2. The EAU meta-agent framework: A meta-agent that monitors rollout transcripts, tool traces, rewards, and aggregate training statistics, diagnoses whether failures come from the policy, the user simulator, the environment, or the training configuration, and proposes targeted modifications. Validated changes are hot-swapped into the live run, and lessons persist across runs.

3. An editable symbolic user simulator: Explicit rules that deterministically map parsed interaction states to user responses, exposed as tunable "knobs" the meta-agent can rewrite, and mixed with an LLM simulator through an ε-symbolic strategy (u_sym with probability ε, u_LLM with probability 1−ε).

4. Intra-run as well as inter-run intervention: Unlike prior paradigms that only allow changes between runs, EAU can modify the symbolic-user rules, add or revise environmental rewards, and adjust RL hyperparameters mid-run, and can decide when a restart is genuinely necessary.

Main Findings

• Overall benchmark improvement: Across three interactive-agent benchmarks and two backbones, EAU improves the overall score over the strongest baseline (InfoPO) by 14.7–15.0%. On Qwen3-4B the overall score goes from 0.545 to 0.627 (+15.0%); on Qwen2.5-7B-Instruct from 0.544 to 0.615 (+14.7%).

• A 4B model beating closed-source prompting: With the Qwen3-4B backbone, EAU's overall score (0.627) exceeds the evaluated closed-source models, including Gemini-3-Flash (0.606) and GPT-4.1 (0.568).

• ColBench gains are largest on the strictest metric: On Qwen3-4B, Pass improves from 0.553 to 0.685 (+23.9%) and Succ. from 0.439 to 0.600 (+36.7%), raising the ColBench average from 0.496 to 0.643 (+29.6%). On Qwen2.5-7B-Instruct, Pass goes from 0.534 to 0.659 (+23.4%) and Succ. from 0.426 to 0.533 (+25.1%).

• UserGym gains, with uneven transfer: Average UserGym score rises from 0.559 to 0.622 on Qwen3-4B (+11.3%) and from 0.563 to 0.620 on Qwen2.5-7B-Instruct (+12.3%). EAU beats the best RL baseline in six of seven evaluated environments on Qwen3-4B, with large gains on Func., Pers., and Inten. On the held-out Inten. environment it reaches 2.023 versus InfoPO's 1.862, but on Tele. it scores 0.512, slightly below InfoPO's 0.542 — the paper explicitly notes the gains are not uniform.

• τ²-bench: EAU reaches an average of 21.5 on Qwen3-4B versus 18.5 for PPO (+16.2%), and versus 18.0 for InfoPO. It is best on Airline (28.8) and Telecom (21.3), while InfoPO remains best on Retail (24.4).

• Hybrid user simulation beats either alone: On ColBench with Qwen3-4B, the hybrid simulator (ε = 0.5) scores 0.294 ± 0.019 Pass and 0.213 ± 0.032 Succ., slightly above LLM-only (0.284 ± 0.046 / 0.200 ± 0.044); symbolic-only is worse (0.229 ± 0.004 / 0.177 ± 0.012).

• A generic meta-agent is not enough: A stripped-down baseline with the symbolic user and the intra-run intervention skill removed gives mixed results versus PPO — including a decrease in UserGym on Qwen2.5-7B-Instruct, where it scores 0.512 versus PPO's 0.568 — while EAU reaches 0.620 on that setting. Because two components are removed together, the comparison tests the combined design, not each part separately.

• Gains are not tied to one setup: Adding EAU to UserRL on τ²-bench lifts it from 13.8 to 20.8, and adding it to InfoPO lifts 18.0 to 21.0. Swapping the meta-agent to Codex (GPT-5.6-Sol) still improves over InfoPO alone, to 18.8, though by a smaller margin than the default meta-agent.

• Intra-run intervention matters: On ColBench, EAU proactively restarts its first three runs after detecting severe issues, explores individual modifications in run 4, and in run 5 — the second completed run — surpasses the strongest prior baseline. On UserGym, given identical experience from the first four runs, enabling intra-run intervention in run 5 produces a larger improvement than inter-run adaptation alone. The paper notes these curves show run-count progression, not an efficiency comparison under matched compute.

• Diagnostics surface real bugs: In two analyzed ColBench runs of 624 training steps each, the meta-agent logged 15 knob writes, two measured harms, and three retracted attributions. Examples: reducing max_steps from 10 to 6 lowered Pass by 0.246 and reverting raised it by 0.177; a min_questions counter that omitted turns which both asked and answered a question deadlocked 41% of episodes and was repaired in code. Four proposed max_steps increases were rejected by a write check requiring the recent training success rate to reach 0.80 before difficulty is increased.

Methodology in Plain English

The agent itself is a normal LLM trained with reinforcement learning — the authors fix PPO with a learned critic and Generalized Advantage Estimation, computed only over action tokens so that a turn's advantage is not diluted by surrounding observation text. What changes is everything around the agent.

A separate "meta-agent," instantiated as a coding agent (Claude Code with Claude Opus-5 by default), has access to the training code and logs on a local filesystem. At fixed intervals it reads trajectories, tool traces, rewards, and optimization statistics, and decides which of a prepared list of knobs to turn. Its intervention space spans four areas: the user simulator (rules, the ε mixing coefficient, temperature and persona prompts), the environment (adding or revising intermediate rewards, episode-length constraints), the agent's training dynamics (entropy and KL coefficients), and outright code repairs. Before a change is accepted, a bounded integrity gate validates it; accepted changes are hot-swapped into the running loop. The meta-agent also keeps an "experience pool" — a directory of documents in the filesystem — that it consults before intervening and updates with new findings, so knowledge carries across runs.

A key design choice is the symbolic user simulator: instead of relying entirely on a stochastic LLM to play the user, EAU can encode user behavior as explicit rules over parsed agent outputs, giving deterministic control over what information is revealed and when. Because hand-writing such rules is laborious, they are exposed as knobs for the meta-agent to revise. The hybrid ε-symbolic scheme mixes symbolic and LLM responses, with ε itself adaptable.

Evaluation discipline is deliberate: the meta-agent only sees training-side feedback. Evaluation instances, trajectories, scores, and derived summaries are never used to propose or accept interventions, update the experience pool, or guide subsequent runs; training-time changes do not alter the evaluation task set, user-simulator configuration, interaction limits, or scoring rules. Training used CoSER (a fine-tuned Llama-3.1-8B) as the default LLM simulator and GPT-4o-mini for evaluation, following prior work.

Why This Matters

Most work on stabilizing multi-turn agent RL tries to make better use of a noisy reward. This paper argues that some noise is a bug in the infrastructure, not a property of the signal, and that an automated supervisor can find and fix it. If that holds up, it changes what "training an agent" includes: the simulator and environment become part of the optimization target rather than a fixed substrate.

Real-world applications:

• Customer support agents that must ask clarifying questions before acting, where an inconsistent simulated customer would otherwise teach the wrong behavior. (UserGym is built around exactly this kind of interaction.)
• Travel booking assistants, where the agent must negotiate constraints and call reservation tools correctly (the τ²-bench Airline domain, where EAU scored highest at 28.8).
• Telecom troubleshooting agents that walk a user through diagnosis over many turns (τ²-bench Telecom, where EAU scored 21.3).
• Retail and order-management assistants handling returns, exchanges, and account changes (the τ²-bench Retail domain, where InfoPO at 24.4 remained best — a useful caveat).

Industry relevance: The method targets a practical pain point in applied RL for agents — that reward engineering and simulator engineering consume large amounts of human effort and go stale as the policy improves. Automating part of that loop, and persisting what was learned across runs, is directly relevant to teams running repeated training campaigns on user-facing assistants. The paper is also candid about cost: the stated episode and turn limits define the rollout and search setup, not a matched end-to-end compute budget, which would additionally need to account for meta-agent inference and all attempted training runs.

Future Directions

1. Cheaper, open meta-agents: The conclusion explicitly calls for investigating post-training open-source models to perform this co-evolution autonomously, rather than relying on frontier coding agents such as Claude Code and Codex.

2. Matched-compute and efficiency comparisons: The authors flag that run-level curves show progression by run count, not efficiency under matched total compute, and that their limits are not a matched end-to-end compute budget — leaving open how much the approach costs relative to simpler alternatives.

3. Separating the components of the design: The ablation removes the symbolic user and the intra-run intervention skill together, so the individual contribution of each remains an open question, as does the role of specific knobs (several, such as per_gym_scale and turn_reward_clip on ColBench, were not exercised at all).

4. Safety and correctness guarantees: The ethics statement notes that EAU can modify training code, rewards, and simulator behavior, creating risks of implementation errors, misleading learning signals, and unintended disclosure of task information, and that interface checks constrain admissible updates but do not guarantee behavioral correctness — nor does benchmark performance with simulated users establish safety with real users.

Target Audience

Researchers and engineers working on RL for LLM agents, particularly those building multi-turn, tool-using assistants trained against user simulators; practitioners frustrated by unstable or misleading rewards in interactive RL; and readers interested in meta-agents and automated machine-learning-system optimization. It is also useful for benchmark designers who want to understand how simulator and verifier artifacts propagate into reported agent performance. Readers without RL background will follow the motivation and results but will need to look up PPO, GAE, and critic-free methods like GRPO to fully engage with Section 4.4.

Authors’ abstract

Training LLM agents to interact with users over multiple turns is challenging because the learning signal depends not only on the agent, but also on the user simulator and environment, which jointly shape the agent's experience. Simulated users can be stochastic or inconsistent, while environments may contain unreliable tools, incomplete state coverage, and incorrect verifiers. Prior work mainly addresses the resulting instability through reward shaping or denser supervision, while treating these components as fixed. We instead introduce EAU, a meta-agent framework for environment--agent--user co-evolution. During training, EAU monitors trajectories, tool traces, rewards, and training statistics; diagnoses failures and uninformative interactions; and proposes targeted modifications to the user simulator, environment, reward design, and RL configuration. Validated changes are hot-swapped into the ongoing run, allowing the training setup to evolve together with the policy. These adaptations also persist across runs, so later training can build on previously discovered improvements rather than restarting from a fixed setup. We enable this process through an editable symbolic representation of user and environment dynamics, while the agent remains a neural LLM optimized with RL. Across three interactive-agent benchmarks and two backbone models, EAU consistently outperforms prior RL methods, improving the overall score over the strongest baseline by 14.7--15.0% and increasing successful task completion on ColBench by up to 36.7%, showing that co-evolving the training system with the policy can substantially improve multi-turn RL.