Agent-G$^2$: Gaussian Guidance for Agentic Reinforcement Learning

AI summary

Overview

Research area: Reinforcement learning for large language model agents, specifically hint-based RL for long-horizon, sparse-reward agentic tasks (web navigation and embodied control).

Technical level: Advanced. The paper assumes familiarity with GRPO-style policy optimization, reward sparsity, advantage estimation, and ablations over sampling distributions, though the core idea — draw the guidance depth per task from a Gaussian instead of fixing it — is easy to state.

One-sentence scope: The paper diagnoses how existing hint-based RL methods misallocate expert-prefix guidance depth, shows that useful depths form a Gaussian-like band rather than a single optimum, and proposes Agent-G$^2$, a framework that samples per-task depths online from rollout statistics at no extra rollout cost.

What This Paper Is About

Long-horizon agentic tasks give only a binary reward at termination, so agents almost never stumble onto a successful trajectory and the learning signal collapses. Hint-based RL fixes this by executing a prefix of an expert trajectory before each rollout, so the policy starts closer to success — but the method's behavior hinges entirely on how much of that trajectory to keep. Existing methods treat this "guidance depth" as one deterministic number, either shared across a whole batch or probed separately per task at significant rollout cost. Agent-G$^2$ argues this is a mismatch with the problem's actual structure and replaces the scalar with a per-task Gaussian sampled from statistics the training loop already produces.

Key Contributions

1. A diagnostic characterization of guidance depth as a band, not a point. The authors show that informative depths form a neighborhood around each task's optimum, and that the training-signal profile aligned by depth offset is unimodal, approximately symmetric, and well fit by a Gaussian ($\sigma = 0.22$, $R^2 = 0.92$). This explains structurally why shared-depth schedulers fail and why per-sample probing must trade rollout cost for selection accuracy.

2. Agent-G$^2$, a Gaussian guidance framework with a global-local decomposition. Each task's depth distribution center combines a global baseline tracking overall policy progress with a per-cluster correction for task difficulty; the spread tracks within-cluster variance of success rates. No probe rollouts and no learned depth predictor are used.

3. Online schedule adaptation from existing GRPO rollouts. The same rollouts that update the policy also refresh the global baseline and the per-cluster EMA statistics, so per-task depth variation is obtained at no additional rollout cost.

4. Empirical validation across two benchmarks and two model scales. Results on ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct show consistent gains over hint-based, hint-free, and auxiliary-supervision baselines, at under one-third the rollout cost of per-sample probing.

Main Findings

• Shared-depth scheduling is structurally mismatched, not mistuned. Among the shared-depth schedulers tested at step 50 on Qwen2.5-1.5B-Instruct / ALFWorld, the three step-based schedulers place only 15%–23% of rollout groups inside the in-range band, and most assignments are over-guided. Even the best shared scheduler, Target-acc, leaves 38% of assignments outside the in-range band. Across five additional checkpoints, Fix-step exceeds 48% mismatch at every checkpoint, Linear and Step-dec stay near 50%, and Target-acc averages the lowest mismatch at 39.8%.

• Per-sample probing buys accuracy only with extra rollouts. With 2 probe rollouts per candidate depth, Binary Search still leaves 75% of assignments mismatched. Enumeration with 4 probe rollouts reaches 47% mismatch at 2× the GRPO rollout budget, and near-zero mismatch requires 32 probe rollouts at 20× the budget.

• Useful guidance occupies a band with a Gaussian-like informativeness profile. Using Bernoulli variance $p_i(d)(1-p_i(d))$ as an informativeness proxy and aligning task-depth pairs by $\Delta d = d - d_i^\star$, the profile is unimodal and approximately symmetric, with Gaussian fit $\sigma = 0.22$ and $R^2 = 0.92$.

• ALFWorld results. Agent-G$^2$ reaches 95.3% overall success at 1.5B and 98.4% at 7B, improving over the strongest hint-based RL baseline by 1.5 and 2.3 points, over the strongest hint-free RL baseline by 3.9 points at both scales, and over RLVMR (the strongest auxiliary-supervision method) by +7.4 / +6.6 points. It beats Enumeration, the strongest probe-based baseline, by +9.3 / +2.3 points without auxiliary networks or probe rollouts.

• WebShop results. Agent-G$^2$ obtains a 92.3 reward score at both scales, with 78.9% (1.5B) and 84.4% (7B) final-purchase success. At 1.5B it outperforms Enumeration by +2.2 reward-score points.

• Schedule design substitutes for scale. The 1.5B Agent-G$^2$ surpasses all 7B non-probing baselines, including 7B BEACON, indicating that the gains are not merely compensating for limited model capacity.

• Gains are not explained by imitation. Full SFT reaches 56.3% and Sampled-Prefix SFT reaches 26.6% on ALFWorld, far below Agent-G$^2$'s 95.3%; the 68.4-point gap between Sampled-Prefix SFT and Agent-G$^2$ isolates the contribution of post-prefix RL.

• Stochastic sampling is essential. Removing sampling (setting $d_{i,j} = \mu_i$, $\sigma_i = 0$) drops overall success from 95.3% to 89.8%. Replacing the Gaussian with a variance-matched uniform distribution drops it to 88.3%, suggesting the gain comes from covering the informative band rather than from the exact distributional shape.

• Both the center and the spread matter, and clustering matters most on long tasks. Setting $\mu_i = \mu_{\text{global}}$ lowers overall success to 91.4%; setting $\sigma_i = \sigma_{\min}$ lowers it to 93.8% with a pronounced Long-task drop from 94.7% to 78.3%; collapsing to a single cluster ($K=1$) lowers overall success to 89.1% and degrades Long tasks by 29.3 points.

• The auxiliary prefix loss stabilizes but is not the main driver. Removing $\mathcal{L}{\text{aux}}$ costs 8.6 points overall, whereas removing $\mathcal{L}{\text{GRPO}}$ reduces the method to Sampled-Prefix SFT and drops performance to 26.6%, a 68.7-point decrease.

• The schedule emerges from training rather than being hand-specified. At $t=5$ the depth distribution favors deep prefixes, around $t=25$–50 it widens as cluster variance peaks, and by $t=200$ it concentrates near zero because every cluster has cleared the success threshold. The Long cluster retains a wider $\sigma_k$ for longer.

• Training efficiency. Agent-G$^2$ reaches the final accuracy of scheduled baselines in roughly half the gradient steps. Per-step cost on Qwen2.5-1.5B / ALFWorld is 88 seconds (ratio 1.00×), versus 57 s for Step decay (0.65×), 60 s for Cosine decay (0.68×), 61 s for Linear decay (0.69×), 80 s for Target acc (0.91×), 285 s for Binary Search (3.24×), and 425 s for Enumeration (4.83×).

Methodology in Plain English

Each training task has one expert trajectory of length $L_i$. A scheduler picks a guidance ratio $r_i \in [0,1]$, which converts to a prefix length $n_i = \min(\lceil r_i L_i \rceil, L_i - 1)$; the first $n_i$ expert actions are executed in the environment, and the policy runs $R$ independent rollouts from the resulting state. GRPO then turns the binary terminal rewards into a group-normalized advantage.

Agent-G$^2$ changes how $r_i$ is chosen. The scheduler keeps three quantities: a global baseline $\mu_{\text{global}}$ tracking the guidance level the current policy needs, plus per-cluster exponential moving averages $A_k$ (success level) and $V_k$ (within-cluster dispersion of success rates). Tasks are clustered offline by expert-trajectory length as a difficulty proxy. For a task in cluster $k$, the center is $\mu_i = \mathrm{clip}(\mu_{\text{global}} + \lambda(p_{\text{target}} - A_k), 0, 1)$ and the spread is $\sigma_i = \max(\gamma V_k, \sigma_{\min})$, with $p_{\text{target}} = 0.5$. One ratio is drawn per task as $z_i \sim \mathcal{N}(\mu_i, \sigma_i^2)$ and clipped to $[0,1]$. Tasks in the same cluster share parameters, but independent per-task draws create depth variation within the batch.

After rollouts, the policy is updated with $\mathcal{L}{\text{GRPO}} + \eta \mathcal{L}{\text{aux}}$, where $\mathcal{L}{\text{aux}}$ is a teacher-forced loss on the sampled expert prefixes. The same terminal rewards then refresh $\mu{\text{global}}$, $A_k$, and $V_k$ for the next batch. Schedule state is initialized from weak priors ($\mu_{\text{global}} \leftarrow 0.8$, $A_k \leftarrow 0$, $V_k \leftarrow 0$) that only determine the first batch. Agent-G$^2$-specific hyperparameters ($\Delta = 0.1$, $\alpha = 0.2$, $\gamma = 1.0$) are fixed across both benchmarks without per-task tuning, and results are averaged over 5 seeds on held-out test sets.

Why This Matters

Impact on research. The paper reframes guidance-depth selection from a search problem (find the one right depth) to a coverage problem (sample a neighborhood of useful depths), and shows the coverage signal is already sitting in the rollouts GRPO collects. That removes an entire class of auxiliary machinery — probe rollouts, depth predictors, auxiliary value heads — and it moves the evaluation of hint-based RL from mathematical reasoning, where tasks share uniform structure, to agentic benchmarks where difficulty varies widely within a single batch.

Real-world applications (derived from the benchmark domains and the agentic setting):

• Household and embodied assistants that must chain many sequential actions ("pick two objects," "clean then heat") where feedback arrives only at completion.
• Web and e-commerce agents that search, filter, and purchase against a natural-language specification, the WebShop setting.
• Any multi-step tool-use or workflow agent where an expert demonstration exists and a binary or terminal success signal is the only reward.
• Sim-to-real or production fine-tuning pipelines that already have logged expert trajectories and want to convert them into training signal without building a reward model or critic.

Industry relevance. The cost table is the practical argument: Agent-G$^2$ costs 88 seconds per gradient step versus 285–425 seconds for probing methods, and converges in roughly half the gradient steps of scheduled baselines. For teams training agentic policies, that is a direct reduction in compute per run, and it is achieved by refactoring how an existing GRPO loop is scheduled rather than by adding a new model. The finding that a 1.5B model can surpass 7B non-probing baselines also matters for deployment economics.

Future Directions

• Weaker supervision. Agent-G$^2$ requires one expert trajectory per training task. Extending it to suboptimal demonstrations or language hints is stated as an open direction, since the current framework cannot be applied when such trajectories are unavailable or costly.
• Richer distributional families. The Gaussian is motivated by an empirical fit on two benchmarks. Tasks with multimodal or heavily skewed depth profiles may need different parameterizations, though the uniform-distribution ablation suggests shape matters less than stochastic coverage.
• Online difficulty estimation. Clusters are currently fixed offline by expert-trajectory length, a simple proxy that does not adapt as the policy improves and task-relative difficulty shifts. Online difficulty estimation could improve adaptivity.
• Generalization beyond the two benchmark domains. The diagnostic and the method are demonstrated on ALFWorld and WebShop with two Qwen2.5 scales; whether the Gaussian band structure and the global-local decomposition hold in other long-horizon agentic settings is not established here.

Target Audience

Researchers and engineers working on reinforcement learning for LLM agents, particularly those dealing with sparse or terminal-only rewards in long-horizon settings. It is also relevant to practitioners building agentic training pipelines who already have expert trajectories and want to avoid the cost of probe-based depth search or auxiliary reward models. Readers without a background in policy-gradient methods and advantage estimation will find the diagnostics accessible but the method and ablations demanding.

Authors’ abstract

Hint-based reinforcement learning addresses reward sparsity in long-horizon agentic tasks by retaining a prefix of an expert trajectory before each rollout, letting the policy explore from a state closer to success. Its effectiveness hinges on the guidance depth: how much of the trajectory to keep. Existing methods treat this depth as a deterministic scalar. Scheduled approaches share one value across samples and ignore per-task heterogeneity; per-sample probing estimates it separately at the cost of extra rollouts. We find that useful guidance occupies a band of depths whose informativeness profile is approximately Gaussian around the band center, rather than concentrating at a single optimal point. We propose Agent-G$^2$, a Gaussian guidance framework that draws the depth per task from a Gaussian whose center and spread are estimated online from rollouts already collected for policy optimization, requiring no probe rollouts or learned depth predictor. The center combines a global baseline with per-cluster difficulty, and the spread tracks within-cluster variance. We evaluate Agent-G$^2$ on ALFWorld and WebShop on Qwen2.5-1.5B / 7B-Instruct. Agent-G$^2$ outperforms the strongest hint-based, hint-free, and Aux-RL baselines on ALFWorld by 2.3 / 3.9 / 7.4 points at under one-third the rollout cost of per-sample probing.