Co-Evolving Acting and Forecasting for Language Agents

AI summary

Overview

Research area: Artificial Intelligence — reinforcement learning (RL) for large-language-model agents, specifically sample-efficient training of long-horizon agents.

Technical level: Advanced. The paper assumes familiarity with policy gradient methods, GRPO, advantage normalization, and supervised fine-tuning of LLM backbones.

Scope: The paper introduces CoPE, a training method that adds multi-step action-forecasting as an auxiliary objective to GRPO so that a single LLM backbone learns to act and to forecast its own future actions at the same time, and evaluates it on five agent benchmarks.

What This Paper Is About

Training language agents is expensive because each episode requires many interaction steps, and standard agentic RL (such as GRPO) extracts learning signal only from a single success/failure scalar at the end of the trajectory. Prior attempts to use the intermediate steps rely either on value targets, which label every step with the same trajectory-level outcome, or on observation targets, which are long, noisy (HTML pages, terminal logs) and reach only one step ahead. CoPE proposes a third kind of target — the sequence of actions that actually followed each history in a successful rollout — and trains the policy to forecast those action chunks from earlier histories, while still acting one step at a time at inference.

Key Contributions

1. A third kind of supervision target — hindsight action chunks. CoPE defines the target distribution q(a_{t:t+K} | h_t) as the distribution over the next K+1 actions under the success-conditioned hindsight distribution Pr+(·|Y=1), with intermediate observations marginalized out, and shows these chunks can simply be read off successful on-policy rollouts without learning a world model.

2. A formal link between GRPO and single-step hindsight fitting. The paper shows that, up to a task-dependent positive scalar, the idealized GRPO update follows the gradient of a cross-entropy to the single-step (K=0) hindsight distribution, and then defines the action-forecasting loss as the multi-step (K>0) counterpart of that same objective.

3. A practical joint training algorithm with no inference overhead. The forecaster and the actor share one backbone; the forecasting loss is computed only on successful trajectories, normalized by the number of successes per group to prevent gradient scaling, and combined with GRPO as L = L_GRPO + λ·L_AF. At test time the agent acts with the policy alone, so no extra model, no separate pretraining stage, and no extra forward passes are needed.

4. An empirical demonstration of co-evolution across five benchmarks. CoPE improves aggregate metrics over GRPO and over observation-prediction baselines (ECHO, RWML) on three simulated environments and improves overall metrics on two realistic applications, with measured sample-efficiency and wall-clock numbers.

Main Findings

• Aggregate gains over GRPO on simulated environments: ALFWorld improves by 5.0 points (90.50 to 95.50 overall success); WebShop success rises by 12.5 points to 77.0% (score 84.07 to 89.40); ScienceWorld success rises by 8.0 points to 72.5% (score 52.78 to 53.40). CoPE is reported as the best among RL-based methods on ScienceWorld.

• Gains over observation-prediction baselines: CoPE exceeds the stronger of ECHO/RWML by 1.0 point on ALFWorld, by 4.4 average-score points and 5.5 success-rate points on WebShop, and by 5.0 success-rate points on ScienceWorld. The gains are not uniform across ALFWorld subtasks — ECHO or RWML performs better on Cool and Pick2.

• Realistic applications: On τ²-bench, overall success moves from 27.5% to 28.7% (Telecom 38.8 to 40.6, Retail 17.5 to 29.4, Airline 26.2 to 16.2). On AppWorld with Qwen2.5-14B-Instruct, success rises from 41.67% to 55.36% on the normal split and from 18.71% to 22.54% on the challenge split.

• Sample efficiency: CoPE is 1.6× and 2.0× more sample-efficient than GRPO on AppWorld and ScienceWorld respectively. On ScienceWorld, ECHO requires 53 and 114 steps to reach equivalent milestones, making CoPE 1.7× to 1.9× faster; CoPE reaches a 0.6 reward by step 86, a level neither baseline reaches within the first 150 steps.

• Low overhead: On ScienceWorld the forecasting pass adds 11% time per training step on average, while shorter rollouts reduce measured end-to-end step time by 9% relative to GRPO.

• Delay analysis of why multi-step targets help: Relative to GRPO, success-filtered SFT changes success rate by −7.5 points, K=0 forecasting by +2.0 points, K=0 with an invalid-action filter by +5.0 points, and full CoPE (K=2) by +8.0 points — an ordering the authors compare to delay-ensemble findings in robot action chunking.

• Evidence for implicit temporal ensembling: With a separately trained LoRA forecaster used to rerank eight candidate actions, explicit forecasting improves GRPO (56.5 to 70.0 at checkpoint G150, 64.5 to 72.0 at G250, and 72.5 to 73.0 at the final checkpoint) but adds essentially nothing to CoPE (−0.5 at the final checkpoint), suggesting CoPE's policy already contains that information.

• Backbone generality: The GRPO-to-CoPE gain holds on a second backbone, Qwen3-8B (WebShop score 83.4 to 87.4, success 64.0 to 72.5; ScienceWorld score 50.1 to 58.3, success 61.5 to 71.5).

• Why success conditioning is required: Without success conditioning, the authors report that the model is forced to imitate average or failed trajectories, creating an objective conflict that leads to zero reward after the first few steps of training.

• Negative result on τ²-bench Airline: The drop from 26.2% to 16.2% is attributed to outcome-conditioned imitation over-representing easy tasks; 38% of Airline tasks expect the final database to stay unchanged, versus 4.4% in Retail.

Methodology in Plain English

The agent is an LLM that sees a history of observations and past actions and picks the next action. Standard GRPO collects a group of trajectories per task, scores each with a single success/failure reward, and pushes the model toward actions in successful trajectories and away from actions in failed ones. The problem is that every action in a trajectory gets the same push, and the model is never asked to think ahead.

CoPE keeps GRPO but adds a second job for the same model. On trajectories that succeeded, it takes each intermediate history and asks the model — under a special "forecast the future actions" instruction — to reproduce the next K+1 actions that actually followed, skipping the observations in between. Because only successful trajectories are used, success acts as a filter that selects which trajectories to imitate rather than as a label pasted onto each step. Because the skipped observations are simply dropped, the loss is spent on short, decision-relevant action tokens instead of on long HTML pages or terminal logs.

The same model does both jobs, so the forecasting practice is written directly into the representation the acting policy uses. The authors describe this as an implicit temporal ensemble: an action taken at step t is also supervised as a forecast issued from each of the K preceding histories, so it gets trained from several vantage points rather than only from the moment it was executed. They also add a normalization term that averages the forecasting loss over the number of successful rollouts in a group, so that as the policy improves and more rollouts succeed, the auxiliary gradient does not grow and drown out the RL objective. At inference, the forecasting instruction is simply not used: the agent produces one action per step, exactly as a GRPO-trained agent would.

Why This Matters

Impact on research. The paper reframes auxiliary supervision for agentic RL: instead of predicting noisy environment observations or constructing step-level value estimates from a single outcome signal, it predicts actions over multiple horizons. It also gives a concrete bridge between LLM agent RL and the robot-learning literature on action chunking and delayed prediction, and shows that chunking-style benefits can be obtained without ever executing a chunk.

Real-world applications:

• Customer-service agents that must follow policy constraints while operating over long conversations (the τ²-bench Retail, Airline and Telecom domains).
• Web-navigation and online-shopping agents that act over long, token-heavy HTML observations (WebShop).
• Scientific or laboratory automation agents that must plan multi-step experimental procedures (ScienceWorld).
• App-automation and API-composition agents that write code to orchestrate consumer applications (AppWorld).

Industry relevance. The method requires no separate world model, no extra pretraining stage and no additional inference cost, which makes it cheap to bolt onto an existing GRPO training pipeline. The reported 11% per-step training cost against a 9% reduction in measured end-to-end step time is the kind of trade-off an engineering team can act on directly; the code is released at github.com/ulab-uiuc/CoPE.

Future Directions

• Reducing the easy-task over-representation of success filtering. The authors identify this as a known limitation of outcome-conditioned imitation and point to the Airline drop as an extreme case; how to reweight or diversify successful trajectories is left open.

• Choosing the forecast horizon K. The paper analyzes K=0 versus K=2 variants and motivates larger horizons through the delay-ensemble argument, but a principled way to set K per task or per environment is not established.

• Extending beyond a fixed single-step execution policy. CoPE deliberately avoids executing predicted chunks at inference; whether selective use of forecasts at test time (for instance when the policy is uncertain) could add further gains is not explored.

• Understanding the implicit ensemble inside the shared backbone. The LoRA-forecaster experiment provides behavioral evidence that forecasting changes what the policy knows, but the mechanistic account of how forecast gradients reshape the acting representation is left to future work.

Target Audience

Researchers and engineers working on RL post-training for LLM agents, particularly those already using GRPO-style critic-free methods and looking for denser supervision without extra inference cost. It is also relevant to robot-learning researchers interested in how action-chunking and delayed-prediction effects transfer from manipulation policies to language agents, and to practitioners deploying agents in web navigation, customer service, or API orchestration who need sample-efficient training on sparse success signals. Readers without a background in policy gradient methods will find the theoretical sections (Section 4) demanding.

Authors’ abstract

Training language agents is bottlenecked by the high cost of long-horizon rollouts, yet standard agentic reinforcement learning (RL) learns only from a terminal scalar reward, leaving the supervision in intermediate steps largely unused. Prior work exploits this supervision through value targets that estimate per-step success probabilities or observation targets that fit environment response. Both are hard to construct well: (1) value targets come from the same single scalar signal for each step in the trajectory, and (2) observation targets, such as web pages and terminal logs, are long and noisy and reach only one step ahead. We propose CoPE, which adds a third kind of multi-step action targets. On successful on-policy trajectories, CoPE trains the policy backbone to generate the action sequences that follows each history. Each action is also supervised when forecast from the preceding histories, forming a temporal ensemble that the shared backbone absorbs during training, and this enables acting and forecasting to co-evolve. The policy still acts one step at a time at inference, with no overhead. Across five benchmarks (ALFWorld, ScienceWorld, WebShop, $τ^2$-bench, and AppWorld), CoPE (1) achieves stronger aggregate performance than GRPO and RL baselines with predictive auxiliary targets across simulated environments, while improving overall performance on both realistic applications; (2) delivers up to $1.9\times$ and $2.0\times$ higher sample efficiency than ECHO and GRPO, respectively, on ScienceWorld; and (3) achieves these gains with a single policy backbone, without additional models, separate pretraining, or inference overhead. The codebase is available at https://github.com/ulab-uiuc/CoPE.