Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks

AI summary

Overview

• Research area: Reinforcement learning (RL) for large language model (LLM) agents operating in long-horizon, multi-turn interactive environments; specifically, environment design as a training-time intervention.
• Technical level: Intermediate. The motivation and strategy are accessible, but the paper assumes familiarity with agentic RL concepts such as reward sparsity, policy entropy, group-relative advantage estimation, and algorithms like GRPO, DAPO, and GSPO.
• Scope: The paper proposes and empirically validates Feedback-Enriched Environments (FEEs), a strategy for enriching environment feedback during RL training, tested on the SciWorld and BFCL benchmarks with Qwen3-4B and Qwen3-8B models.

What This Paper Is About

Training LLM agents for long-horizon tasks with RL is hampered by severe reward sparsity: agents get trapped in zero-reward trajectories and receive no learning signal. The usual fix is to warm up the agent first with supervised fine-tuning on expert trajectories, but such data is expensive to collect and over-optimizing it can constrain the exploration that RL needs. This paper flips the approach, adapting the environment rather than the agent by systematically enriching the feedback the environment returns during training.

Key Contributions

1. A paradigm shift from agent-side to environment-side adaptation. Instead of seeking a better-initialized agent, the authors enrich the environment's feedback signals, formalizing this as an intervention on the observation space that forms a new "enriched environment" alongside the original "standard environment."
2. A systematic feedback design strategy that specifies what to provide (action guidance versus observation enrichment) and when to provide it (during intra-episode exploration and inter-episode evolution across the training lifecycle). The resulting recommendation is to deliver action guidance in early stages and observation enrichment in later stages of both processes.
3. Construction and evaluation of FEEs on two benchmarks (SciWorld and BFCL-V3 Multi-Turn), trained across model scales (Qwen3-4B, Qwen3-8B) and RL algorithms (GRPO, DAPO, GSPO), yielding an average improvement of 2.82% over standard environments.
4. An in-depth analysis of training dynamics, covering training stability via policy entropy, state-space exploration, internalization of environmental guidance into policy weights, and the role of intra-group feedback consistency as a boundary for stable optimization.

Main Findings

• Action guidance generally beats observation enrichment: In the pilot study on SciWorld, curves for action guidance consistently stay above those for observation enrichment throughout training. The authors attribute this to action guidance pruning the search space with valid next steps, whereas observation enrichment imposes a higher cognitive load that the agent must learn to map to actions through causal reasoning.
• Timing depends on feedback type: Action guidance applied early (AG-Early) outperforms action guidance applied late (AG-Late) in earlier stages, while observation enrichment applied late (OE-Late) shows a sharp upward trend in later stages compared with OE-Early. Action guidance prunes the initial combinatorial search space but becomes redundant once basic navigation is mastered; observation enrichment needs a foundational policy to interpret the added semantics.
• A hybrid schedule works best: Applying AG-Early for the first 100 training steps and switching to OE-Late for the remaining 100 steps surpassed all other settings, achieving the highest success rate in the pilot study.
• RL training closes much of the gap to proprietary models: Qwen3-4B and Qwen3-8B start at average scores of 23.58% and 27.18%. After RL training, Qwen3-4B reaches 47.81% and Qwen3-8B reaches 48.11%, the paper reports, outperforming Qwen3-235B-Thinking (35.09% average) and closely approaching GPT-5.4 (52.07% average) and Kimi-K2-Thinking (53.51% average).
• FEEs improve results across scales, algorithms, and benchmarks: For example, Qwen3-8B with GSPO on SciWorld improves from 53.91% to 60.94%, and Qwen3-4B with GRPO on BFCL-Base improves by 10.00 percentage points (58.00% to 68.00%). DAPO with Qwen3-4B on BFCL-Base improves from 61.00% to 71.00%.
• Not all sub-tasks improve: The paper notes regressions on tasks requiring the agent to question input sufficiency. Qwen3-4B trained with DAPO drops from 41.00% to 34.00% on Miss Param tasks, and Qwen3-8B with GRPO declines from 52.00% to 48.00% on Miss Func tasks. The authors hypothesize that proactive guidance makes agents overly inclined to follow interaction cues and prioritize execution over clarification.
• FEEs stabilize training dynamics: Tracking policy entropy for Qwen3-4B over 300 training steps on SciWorld-related environments, without explicit entropy regularization the FEE-trained agent maintained steady entropy for all 300 steps while the standard environment suffered a sharp drop to zero at approximately 250 steps. With entropy regularization, FEEs sustained stable training for nearly 200 steps while the standard environment collapsed at around 130 steps.
• FEEs promote state-space exploration that transfers: On BFCL, the FEE-trained agent showed a higher density of high-success environments in later training. When 400 environments were partitioned into easy ([0.66, 1]), medium ([0.33, 0.66]), and hard ([0, 0.33]) tiers based on success rates of standard-trained models, the FEE-trained model outperformed the baseline across all difficulty levels, with a 4.3% improvement on "hard" environments.
• Guidance is internalized, not just an inference-time hint: Using multiple-choice probe questions on the GorillaFileSystem environment in BFCL, models trained with FEEs progressively assigned higher probabilities to the enriched feedback option across training, whereas standard-trained models remained biased toward the original tool output.
• Intra-group feedback consistency is required: In 200 steps of GRPO training of Qwen3-4B on SciWorld, consistent intra-group feedback produced a steady rise in success rate, while varying feedback within a sampling group produced erratic fluctuations with sharp plunges and sudden spikes. The authors attribute this to distortion of within-group advantage estimation.
• Effectiveness is not universal: On AppWorld, training Qwen3-4B and 8B with enriched feedback still failed to produce positive rewards, with training remaining trapped in zero-reward trajectories.

Methodology in Plain English

The authors begin by treating the environment as a partially observable Markov decision process with goals, states, actions, observations, transitions, and rewards. They define "enriched feedback" as an intervention on the observation space: the environment wraps its observation and the interaction history into an augmented observation before passing it to the agent. This changes the agent's action distribution without changing the underlying task.

They then run a pilot study on SciWorld to decide two design questions. The first is substance: action guidance (procedural hints that suggest valid next steps, effectively narrowing the search space) versus observation enrichment (extra semantic state information that makes hidden dynamics visible). The second is timing, at two scales: within a single episode's finite horizon (intra-episode) and across the whole training lifecycle (inter-episode). In the pilot, each SciWorld episode is limited to 15 steps; AG-Early and AG-Late supply action guidance at steps 1–3 and 6–10 respectively, and OE-Early and OE-Late supply observation enrichment over the same intervals, applied stochastically with a 0.5 probability throughout training. Training uses Qwen3-4B-Thinking-2507 with GRPO for 200 steps, 16 parallel environments, a 15-interaction rollout length, and a 1×10⁻⁶ learning rate, with success rate evaluated on the standard environment every 5 steps using a strictly disjoint task set.

For the main experiments, the authors build FEEs on SciWorld and BFCL-V3 Multi-Turn. In SciWorld, action guidance uses the ground-truth expert trajectories and observation enrichment uses the source code's task-progress tracking. In BFCL, action guidance is suggested actions appended to the user query and observation enrichment expands the raw return contents of invoked tools. They train for 200 steps with 0.5 enrichment probability, switching from AG-Early to OE-Late at step 100, again keeping evaluation tasks disjoint. They compare GRPO, DAPO, and GSPO across Qwen3-4B and Qwen3-8B, with evaluations every 5 steps, and use GPT-5.4, Kimi-K2-Thinking, and Qwen3-235B-Thinking as a performance ceiling. Finally, four analysis experiments probe entropy dynamics, exploration behavior, internalization via probe questions, and intra-group feedback consistency.

Why This Matters

This work reframes where the leverage lies in training long-horizon agents: rather than paying for expert demonstrations or accepting the constraints of supervised warm-up, it asks how the environment itself can scaffold learning. It also supplies a concrete, transferable design rule (guidance early, enrichment late) and documents limits, including regressions on tasks requiring clarification and failure on AppWorld.

Real-world applications:

• Tool-calling and API-driven assistants: The BFCL setting mirrors multi-turn customer-facing systems where an agent must invoke predefined APIs across dialogue turns.
• Scientific and laboratory workflow automation: The SciWorld setting involves designing and executing elementary science experiments in an interactive text environment, a proxy for procedurally constrained lab or instrument operation.
• Computer-use and terminal agents: The paper positions long-horizon tasks such as web navigation and terminal-based computer usage as the motivating class of problems.
• Training pipelines for enterprise agents: The intra-group consistency finding is directly actionable for teams using group-relative RL methods, since feedback randomization within a rollout group can destabilize optimization.

Industry relevance: the paper targets practitioners building RL training loops for LLM agents, reporting that a change to the environment's feedback channel improved results across three different RL algorithms and two model scales without altering the algorithms themselves, which makes the intervention composable with existing agent-side optimization stacks.

Future Directions

• Automating the stage boundaries: Early and late phases in both intra-episode exploration and inter-episode evolution were manually specified; their sensitivity and optimal configurations remain underexplored.
• Generalizing beyond two benchmarks: The authors state that broader evaluation across a wider range of agent benchmarks is needed to establish the strategy's generalization ability.
• Handling extremely sparse long-horizon settings: On AppWorld, enriched feedback still failed to produce positive rewards, suggesting richer environment adaptation strategies are needed for the hardest cases.
• Understanding and mitigating the clarification regression: The observed drops on Miss Param and Miss Func tasks raise the question of how to enrich feedback on tasks where the correct behavior is to question the input rather than act on it.

Target Audience

Researchers and engineers working on RL post-training for LLM agents, particularly those dealing with sparse-reward, multi-turn tasks; practitioners using group-relative algorithms such as GRPO, DAPO, or GSPO who want a training-time intervention that is orthogonal to algorithm changes; and benchmark or environment designers interested in how feedback structure shapes exploration and learning.

Authors’ abstract

Large Language Models demonstrate remarkable proficiency in static reasoning, yet training them as autonomous agents through Reinforcement Learning (RL) for long-horizon tasks is often hindered by severe reward sparsity. While conventional \textit{agent-side warming} up via supervised fine-tuning (SFT) can alleviate this, it is frequently limited by data scarcity and constrained exploration. To address this, we propose a paradigm shift to \textit{environment-side adaptation} by constructing \textbf{F}eedback-\textbf{E}nriched \textbf{E}nvironments (\textbf{FEEs}). Through a pilot study, we establish a feedback design strategy that reformulates environments by transitioning from action guidance to observation enrichment during the later stages of both intra-episode exploration and inter-episode evolution. Large-scale experiments on SciWorld and BFCL benchmarks using various Qwen3 model scales and RL algorithms such as GRPO, GSPO, and DAPO demonstrate that FEEs consistently yield performance improvements over standard settings. Furthermore, our analysis reveals that training with FEEs \textbf{(1)} stabilizes training dynamics by reducing entropy volatility, \textbf{(2)} facilitates proactive state-space exploration in difficult tasks, \textbf{(3) }ensures the internalization of environmental guidance into policy weights rather than acting as a mere inference-time prior, and \textbf{(4) }identifies intra-group feedback consistency as a critical boundary for stable optimization.