Agent Learning via Early Experience

AI summary

Overview

Research area: Training methods for large language model (LLM) agents — specifically a training paradigm positioned between supervised fine-tuning (imitation learning) and reinforcement learning (RL).

Technical level: Intermediate. The paper assumes familiarity with reinforcement learning basics (Markov Decision Processes, reward signals, imitation learning) and LLM fine-tuning, though the core idea is explained in accessible terms.

One-sentence scope: The paper proposes "early experience," a reward-free training paradigm in which an agent's own proposed actions and the resulting future states serve as supervision, and evaluates two strategies under this paradigm — implicit world modeling and self-reflection — across eight language-agent benchmarks and three instruction-tuned models.

What This Paper Is About

Training language agents with reinforcement learning is difficult because many real-world environments lack verifiable rewards (for example, websites that do not report whether a form was filled correctly), or require long-horizon rollouts that make credit assignment inefficient and unstable. As a result, most agents are trained by supervised fine-tuning on expert demonstrations, which is costly to scale, exposes the agent to limited environment diversity, and generalizes poorly because the agent never observes the consequences of its own actions. This paper introduces a middle-ground paradigm — early experience — where the agent acts in the environment, collects the resulting future states, and uses those states as supervision without any reward signal.

Key Contributions

1. Formalizing the early experience paradigm. The authors advocate and formalize early experience as a practical, scalable bridge between imitation learning and RL for language agents. It lets agents convert their own experience into learning signals without external rewards and can be integrated into existing training pipelines.

2. Two training strategies under the paradigm. They propose and systematically study (a) implicit world modeling, which uses collected future states to ground the policy in environment dynamics, and (b) self-reflection, which distills fine-grained lessons from the agent's own suboptimal actions.

3. Comprehensive empirical evaluation. They evaluate across eight environments spanning embodied navigation, web navigation, multi-turn tool use, and long-horizon planning, using multiple model families, and report gains in task effectiveness, out-of-domain generalization, and downstream RL performance.

4. Analysis of practical properties. They study how performance scales with the amount of expert data and with the branching factor (number of alternative actions rolled out per expert state), and compare against representative baselines including Long CoT, STaR, and DPO.

Main Findings

• Consistent gains over imitation learning across all eight benchmarks. Early experience improves over imitation learning in every setting and for every model size tested (Llama-3.2-3B, Qwen-2.5-7B, Llama-3.1-8B). Implicit world modeling (IWM) yields steady gains in structured environments (ALFWorld and ScienceWorld +2.3 to +5.5; WebShop +11.3 to +18.4), while self-reflection (SR) produces the largest jumps where tasks require multi-step reasoning and constraint satisfaction (TravelPlanner +12.8 to +15.0; ScienceWorld +13.3; BFCLv3 +8.0 on the Llama-3.2-3B model).

• Action-space dependence of which method wins. With closed and finite action sets (ALFWorld, ScienceWorld, TravelPlanner), IWM helps internalize transition regularities while SR adds targeted corrections for long-horizon plans. With structured but large action sets (BFCLv3, Tau-Bench), early experience reduces tool misuse and improves ordering, and SR often helps more when policy errors are logical. With open action sets (SearchQA, WebArena), the hardest regime, both methods still yield reliable gains.

• Observation-complexity robustness. Environments range from short clean textual scenes (ALFWorld) to procedural readouts (ScienceWorld), structured API schemas and tool outputs (BFCLv3, Tau-Bench), and noisy accessibility-tree web states with hundreds of DOM-like elements (WebArena). Gains hold regardless of this complexity; IWM excels where transitions are consistent and predictable (for example WebShop), and SR delivers larger gains when failures stem from reasoning errors or long-horizon plan repair (TravelPlanner, ScienceWorld).

• Out-of-domain generalization. OOD scores drop relative to in-domain across all tasks in Table 2, yet early experience consistently recovers a substantial portion of the gap. IWM helps most where dynamics are stable (for example ALFWorld, up to +14.8 on Llama-3.1-8B), and SR is strongest when distribution shifts alter tool availability or arguments (BFCLv3). In several benchmarks such as ALFWorld and SearchQA, OOD gains meet or exceed in-domain gains.

• Early experience is a stronger starting point for RL. Using GRPO with identical hyperparameters and training steps on three benchmarks (WebShop, ALFWorld, SearchQA), checkpoints from early-experience methods consistently lead to higher post-RL ceilings than imitation-only starts. In some cases the gap grows during RL training (ALFWorld); in others it narrows but never reverses. Running GRPO directly from the raw pretrained model performs worst across all tasks and shows unstable training dynamics.

• Data efficiency. On WebShop, using just 1/8 of the demonstrations already surpasses imitation learning trained on the full dataset; on ALFWorld, the same holds with 1/2 of the demonstrations. Early experience maintains a consistent lead over imitation learning at every data level.

• Branching factor behavior differs by method. IWM improves steadily as the branching factor K increases, consistent with learning richer transition regularities. SR improves at small to moderate K and shows diminishing returns at very large K, because comparing many alternatives occasionally includes other success-leading actions, reducing the contrast with the expert, and because current models have limited capacity to reason over many alternatives and outcomes in a single context. SR works best with a modest K (for example 2–4).

• Comparison to representative baselines (Llama-3.1-8B). On WebShop and ALFWorld, adding Long CoT to imitation learning severely degrades performance (0.0, a drop of 47.3, and 25.8, a drop of 54.7), STaR degrades it (25.0, a drop of 22.3, and 74.2, a drop of 6.3), and DPO improves it modestly (53.1, +5.8, and 82.8, +2.3). IWM reaches 58.6 (+11.3) and 85.9 (+5.4) respectively, exceeding all three baselines.

Methodology in Plain English

The problem is framed as a Markov Decision Process with states, actions, a transition function, a reward function, a discount factor, and an initial state distribution — but the reward function is treated as unknown or unverifiable during training. The agent starts from an expert dataset of state–action pairs.

For each expert state, the agent samples K alternative actions from its own current policy and executes them in the environment, recording the resulting next state. This produces a rollout dataset of (state, alternative action, resulting next state) triples. Because the agent itself proposes the alternatives, the rollout dataset can be up to an order of magnitude larger than the expert dataset.

Implicit world modeling turns next-state prediction into a training task. The model is trained with a next-token prediction loss to predict the state that follows a given state–action pair. The same model parameters are used for state prediction during world modeling and action prediction during policy training, so the policy internalizes environment dynamics without a separate simulator module. In practice this is a two-stage pipeline: first train with the world-modeling loss (which also includes the expert transitions), then fine-tune on the expert dataset.

Self-reflection asks the model to explain its own mistakes. For each alternative action, the same language model is prompted to generate a chain-of-thought explaining why the expert action is preferable, grounded in the actual differences between the two resulting next states. The agent is then trained to jointly predict that chain-of-thought followed by the expert action, conditioned on the state.

Evaluation covers eight benchmarks grouped into three families: embodied and scientific simulation plus travel planning (ALFWorld, ScienceWorld, TravelPlanner), multi-turn tool use (BFCLv3, Tau-Bench, SearchQA measured by F1), and web navigation (WebShop, WebArena-Lite). All three models are trained with the same prompt format and decoding strategy, and per environment the early-experience methods use exactly the same step budget as imitation learning.

Why This Matters

Impact on research. The paper defines a concrete middle ground between imitation learning and reinforcement learning for agents, arguing that scalable RL for language agents is not yet mature because environments lack reliable simulators, standard reset mechanisms, and scalable evaluation platforms. It shows that the agent's own actions and the resulting states — not just human demonstrations — can be a dense, reward-free supervision source, and that this supervision transfers forward into better RL outcomes. It also connects to mid-training and world-modeling literature, contrasting with prior work that treats the world model as a separate simulator.

Real-world applications.

• Web navigation agents: booking flights, filling forms, and general website interaction where platforms expose no ground-truth success feedback (WebShop, WebArena-Lite).
• Multi-turn tool and API assistants: agents that must select and sequence many typed tools with arguments and avoid tool misuse (BFCLv3, Tau-Bench).
• Embodied and household assistants: agents that navigate and manipulate in textual environments (ALFWorld).
• Scientific and procedural assistants: agents that carry out multi-step experiments and read procedural outputs (ScienceWorld), plus constraint-heavy itinerary planning (TravelPlanner) and open-ended search/QA (SearchQA).

Industry relevance. The work is produced by Meta Superintelligence Labs, FAIR at Meta, and The Ohio State University, and addresses a practical bottleneck for companies deploying agents: high-quality expert demonstrations are expensive to scale, and reward engineering is unavailable or brittle in many target environments. Early experience reduces dependence on both, achieves comparable or superior performance with only half or even less of the expert data, and applies across model scales tested (3B, 7B, 8B parameters).

Future Directions

• Scaling and method combination. The paper reports results on several instruction-tuned model families and notes that early experience applies seamlessly to larger models; exploring how IWM and SR interact when combined, and whether their complementary strengths (stable dynamics versus reasoning repair) can be captured jointly, is a natural next step.

• Why self-reflection saturates at large K. The authors attribute diminishing SR returns at very large branching factors to loss of contrast with the expert action and limited model capacity to reason over many alternatives in one context — an open problem for reflection design and model capacity.

• Extending early experience to new environments and RL setups. The paper positions early experience as a mid-training bridge, demonstrating post-RL gains on three infrastructure-ready environments (WebShop, ALFWorld, SearchQA). Extending this to environments that lack reliable simulators, reset mechanisms, and scalable evaluation platforms remains open.

• Replacing approximate supervision across longer horizons. Since many tasks involve long interaction sequences with delayed or ambiguous outcomes, further work could examine how reward-free early experience scales to much longer rollouts than those tested.

Target Audience

Researchers and practitioners working on LLM-based agents, agentic fine-tuning pipelines, and reinforcement learning for language models; engineers who need to train agents in environments without verifiable rewards; and readers interested in the broader debate about how agents can learn from their own experience. It is most useful to those with some background in imitation learning or RL, since the paper formalizes its setting as a Markov Decision Process and evaluates against RL baselines such as GRPO.

Authors’ abstract

A long-term goal of language agents is to learn and improve through their own experience, ultimately outperforming humans in complex, real-world tasks. However, training agents from experience data with reinforcement learning remains difficult in many environments, which either lack verifiable rewards (e.g., websites) or require inefficient long-horizon rollouts (e.g., multi-turn tool use). As a result, most current agents rely on supervised fine-tuning on expert data, which is challenging to scale and generalizes poorly. This limitation stems from the nature of expert demonstrations: they capture only a narrow range of scenarios, and expose the agent to limited environment diversity. We address this limitation with a middle-ground paradigm we call early experience: interaction data generated by the agent's own actions, where the resulting future states serve as supervision without reward signals. Within this paradigm, we study two strategies of using such data: (1) implicit world modeling, which uses collected states to ground the policy in environment dynamics; and (2) self-reflection, where the agent learns from its suboptimal actions to improve reasoning and decision-making. Evaluation across eight diverse environments and multiple model families shows that our approaches consistently improve effectiveness and out-of-domain generalization, highlighting the value of early experience. Moreover, in environments with verifiable rewards, our results provide promising signals that early experience offers a strong foundation for subsequent reinforcement learning, making it a practical bridge between imitation learning and fully experience-driven agents.