Mosaic: Multi-Session Agentic Reinforcement Learning

AI summary

Overview

Research area: Reinforcement learning for large language model (LLM) agents — specifically "agentic RL," where a model is trained through multi-turn interaction with tools and environments — applied to web-research agents.

Technical level: Advanced. The paper assumes familiarity with policy-gradient methods, Group Relative Policy Optimization (GRPO), and modern agent harnesses (context compaction, subagent delegation).

Scope: The paper proposes a training setting and algorithm ("Mosaic") for agent trajectories that are split into multiple sessions, and evaluates it on the BrowseComp benchmark with Qwen3.5-4B.

What This Paper Is About

Most agentic reinforcement learning methods assume a single session: one continuously accumulated context that produces one training segment. Modern agent techniques break this assumption — context compaction rewrites the prefix to free context space, and subagent forking creates branch sessions — so trajectories at inference time span multiple sessions while training still treats them as one. The paper formalizes this gap as Multi-Session Agentic Reinforcement Learning and proposes Mosaic, a critic-free hierarchical method that assigns better-targeted learning signals across both the whole trajectory and its individual sessions.

Key Contributions

1. A new problem setting. The authors define Multi-Session Agentic RL, which supports an arbitrary number of sessions N_i within a single trajectory, and show that the widely used single-session assumption creates a train/inference mismatch for compaction and subagent forking.

2. A diagnosis of "broadcasting." They formalize broadcasting — replicating a trajectory-level advantage to every token in every session — as the standard multi-session baseline, and argue it gives only coarse supervision with no control over how sessions are used.

3. Session-aware trajectory-level advantage estimation. A bounded modulation of the GRPO advantage based on where a trajectory's session count ranks among trajectories with the same outcome, plus fallback rules for groups where every rollout succeeds or every rollout fails.

4. Session-level advantage shaping. A critic-free per-session reward computed as the marginal gain in the model's average log-likelihood of the ground-truth answer before versus after a session, normalized separately by session type (normal, compaction, subagent) and added to the trajectory advantage.

Main Findings

• Multi-session training beats single-session training. With context compaction, Mosaic reaches 38.86 accuracy versus 34.68 for single-session-trained GRPO, 35.31 for broadcasting, and 24.80 for the untrained Qwen3.5-4B base model.

• Fewer sessions, better accuracy. Mosaic improves accuracy by up to 10.1% relative over the broadcasting baseline while using fewer sessions per trajectory on average (3.17 → 2.35) in the compaction setting; in the subagent setting it scores 32.46 with 3.77 sessions versus 31.44 with 4.43 sessions for broadcasting.

• Multi-session ability does not emerge for free. Applying a single-session-trained model to multi-session inference gives only modest gains (e.g., 34.68 with compaction and 31.91 with subagent for GRPO), so training in the multi-session setting matters.

• Mosaic beats the information-gain baseline IGPO. IGPO scores 28.52 (single-session), 31.36 (compaction) and 24.64 (subagent) and uses notably more sessions under compaction (4.12); the paper also reports that computing marginal gains at the session level rather than the turn level substantially reduces wall-time.

• Both components contribute, but unevenly by scenario. Ablation shows trajectory-level session-aware estimation and session-level shaping each help; combining them gives a clear additional gain under compaction but only a modest gain under subagent forking, which the authors attribute to subagent trajectories being non-linear (DAG-like) so a session's immediate marginal gain may differ from its true contribution.

• Session signals behave as designed. In mixed-outcome groups, advantages approximately decrease with session count for successful trajectories and increase for unsuccessful ones, while successful trajectories keep higher advantages — widening within-group advantage variance and improving sample efficiency. In all-success or all-failure groups, the fallback introduces session-dependent variation where outcome rewards are identical.

• Type-conditioned normalization matters. Raw marginal information gains differ by session type, with compaction sessions having a lower mean; global normalization would systematically downweight them, whereas per-type normalization lets each type receive both positive and negative signals.

• Gains come from session utility, not session count. From base model to broadcasting to Mosaic under compaction: sessions 1.70 → 3.17 → 2.35, share of positive-information-gain sessions 37.36% → 47.74% → 59.92%, and information gain per session 0.30 → 0.86 → 0.89. Under subagent forking the same metrics go 2.98 → 4.43 → 3.77 sessions, 27.39% → 41.15% → 51.04%, and 0.41 → 0.74 → 0.80.

Methodology in Plain English

The setup is a web-research agent built on Qwen3.5-4B running a ReAct-style loop with two tools, search and visit (via the Serper and Jina Reader APIs). Training uses REDSearcher_RL_1k, a set of 1,000 deep-search tasks, for one epoch with rollout batch size 16 and group size 8; each session is capped at 128K tokens; deepseek-v4-flash serves as the judge for verifying answers; and evaluation is on BrowseComp, 1,266 hard questions requiring persistent web navigation. Hyperparameters are ρ = 0.2, β₀ = 0.1, β₁ = 0.2, and λ_r = 0.1.

The algorithm has two layers, both built on GRPO (which compares rollouts of the same prompt and needs no learned value network):

Trajectory layer. Instead of asking only "did the rollout succeed?", Mosaic asks "did it succeed with a sensible number of sessions?" It splits a prompt's rollouts into the successful and unsuccessful subgroups, ranks each rollout by its session count inside its own subgroup, and converts that rank into a bounded score between −1 and 1. That score nudges the original advantage up or down within a range of 1 − ρ to 1 + ρ, so it can change the size of the reward signal but never its sign. The design encodes three rules: successful trajectories should be rewarded for using fewer sessions (less cost), unsuccessful ones for using more (more exploration), and reward always dominates session count. When every rollout in a group gets the same outcome — a case where the standard advantage collapses to zero — two fallback terms create differentiation, with the all-failure case kept conservative so no failing trajectory gets a positive advantage.

Session layer. Each session is treated as a state transition, and its value is measured by how much it improves the model's average log-likelihood of producing the correct answer given the prefix. A session that surfaces useful evidence raises this quantity; one that discards key facts or injects noise lowers it. These marginal gains are computed with gradients stopped, standardized separately within each session type, and set to zero for the final session (whose output usually contains the answer and would distort the distribution). The normalized signal is added to the trajectory advantage with weight λ_r.

The logic is a long-term, outcome-oriented signal at the top and a short-term, local signal at the bottom.

Why This Matters

Impact on research. The paper reframes session count as a trainable decision variable rather than a fixed property of the harness, and shows that the relevant question is not "how many sessions?" but "how much useful progress per session?" It also supplies a reusable recipe — critic-free, group-relative, hierarchical — that any GRPO-style pipeline can adopt without adding a value model.

Real-world applications:

• Deep-research and competitive-intelligence agents that must browse for hours and compress their own history to stay within context limits.
• Coding and computer-use agents orchestrated by harnesses that spawn subagents and merge results, where execution does not follow a single continuous context.
• Enterprise assistants that must hand off state between phases (retrieval, analysis, drafting) while preserving critical facts.
• Cost-aware deployment, where the number of model calls per task is a direct billing line and the paper's accuracy-per-session trade-off is the operative metric.

Industry relevance. Context compaction and subagent forking are now standard in production agent harnesses, and the paper's discussion names partially opaque harnesses such as Claude Code as a target. Training signals that survive broken context continuity are directly useful to anyone shipping agent products, and the reported reduction in wall-time for computing marginal gains speaks to training cost.

Future Directions

• Open data and environments. The authors note that public training data for multi-session trajectories are scarce, limiting scale and reproducibility, and call for open datasets and matching execution environments.
• Cross-session state transfer as a first-class objective. Compaction, forking and merging all ask what information should be carried forward under a limited context budget; a unified treatment, including information loss and error accumulation over repeated transfers, remains open.
• Asynchronous and partial-rollout training. Multi-session execution amplifies variance in trajectory length and completion time and increases policy staleness; prioritizing finished rollouts could bias learning toward shorter, easier trajectories.
• RL inside black-boxed harnesses. Mechanisms that break context continuity — dropping reasoning tokens or tool responses, choosing a new session boundary versus lossy concatenation — need systematic study, as does the non-linear case where a subagent session feeds several later sessions, which the ablations suggest is not yet well handled.

Target Audience

Researchers and engineers working on reinforcement learning for LLM agents who already understand policy-gradient methods and group-relative advantages, and who build or train agents whose inference-time execution spans multiple contexts. It is also relevant to practitioners designing agent harnesses, who will find the session-utility framing and the accuracy-versus-session-count results directly actionable, and to readers interested in process-level reward design without a learned critic.

Authors’ abstract

Most existing agentic reinforcement learning methods assume single-session trajectories, where interaction proceeds within one continuously accumulated context. However, agentic techniques such as context compaction and subagent forking may produce trajectories spanning multiple sessions, creating a mismatch between the single-session training assumption and inference-time execution. Therefore, we introduce Multi-Session Agentic Reinforcement Learning, a more general setting that supports an arbitrary number of sessions within a trajectory. Under this setting, a widely adopted method is to broadcast trajectory-level advantages to all sessions, but this does not distinguish contributions from various sessions or guide session allocation within trajectories. To address this issue, we propose Mosaic, a multi-session agentic RL method with critic-free hierarchical optimization. At the trajectory level, it incorporates session-usage signals into advantage estimation. At the session level, it shapes each session's advantage according to the estimated marginal contribution. Experiments on BrowseComp with Qwen3.5-4B show that Mosaic outperforms broadcasting in both context compaction and subagent forking scenarios while using fewer sessions on average. Further analyses show that our method achieves improvements in session utility and allocation, which we suggested is a promising target for future advances in multi-session agentic RL.