Scaling In-Context Online Learning Capability of LLMs via Cross-Episode Meta-RL

AI summary

Overview

Research area: Large language model agents, in-context reinforcement learning (ICRL), and meta-reinforcement learning.

Technical level: Intermediate to Advanced — the paper assumes familiarity with Markov decision processes, policy-gradient optimization, and LLM post-training, though the core idea is stated plainly.

Scope: The paper introduces Orbit, a multi-task, multi-episode meta-RL training framework that teaches LLMs to improve their behavior across repeated attempts at the same unseen task using only the interaction history in their context window.

What This Paper Is About

LLMs are strong when all task-relevant information is given upfront, but they struggle in settings where the important information must be discovered through interaction and where feedback arrives only after actions are taken. The authors target a specific regime they call multi-episode in-context online learning: an agent gets several fresh attempts at the same unchanged task, and a competent learner should use early failed attempts to gather information and do better in later ones, all without updating model weights. The goal is to test whether that general capability can be trained into an LLM rather than prompted into it.

Key Contributions

1. Orbit, a multi-episode meta-RL training framework. A multi-task, multi-episode meta-reinforcement learning recipe that trains a pretrained LLM across a distribution of decision-making environments and across multiple episodes per task, rewarding the model for learning-to-learn within its context rather than for solving a single episode.

2. A deliberately minimal design. The framework avoids external memory, retrieval-augmented generation, and extensive prompt engineering for summarization or reflection, so that any measured improvement can be attributed to multi-episode meta-learning alone.

3. Demonstrated transfer to completely unseen tasks. After meta-training on five training environments, an open-source Qwen3-14B model shows substantially improved in-context online learning on Maze and Mastermind, which are entirely unseen during training, matching GPT-5.2 and outperforming standard RL fine-tuning by a large margin.

4. Evidence of emergent exploration and scaling. The paper provides both qualitative traces and a quantitative count of newly explored states showing that Orbit learns to "try something different" after failure, plus scaling results across Qwen3-4B, 8B, and 14B showing consistent gains with model size.

Main Findings

• In-context improvement across episodes on unseen tasks: Orbit (8B) achieves substantially higher success rates than the base model on unseen Maze and Mastermind, and continues to improve across successive episodes, whereas the base model does not.

• Advantage over standard RL post-training: A standard RL baseline trained on the same task distribution but with single-episode completion rewards shows only modest early-episode gains and then saturates or degrades. Orbit instead improves monotonically across episodes on both tasks.

• Episode 3 success rates (Table 2): On Maze, RL reaches 0.45 (+0.23 over base) and Orbit reaches 0.55 (+0.33). On Mastermind, RL reaches 0.21 (−0.06 versus base) and Orbit reaches 0.59 (+0.32).

• Learned exploration under failure (Table 4, Qwen3-8B): Newly explored states in Episode 2 given failure in Episode 1: base 1.64, RL 4.11, Orbit 4.69. Newly explored states in Episode 3 given failure in both Episodes 1 and 2: base 0.94, RL 1.27, Orbit 1.48.

• Emergent reflection-like behavior: In a qualitative Maze trace, after failing in Episodes 1 and 2, Orbit enters Episode 3, spontaneously summarizes earlier failures inside its thinking block, and selects an action that differs from previous attempts, ultimately reaching the goal. The paper stresses that these behaviors are not prescribed by any explicit "reflect" instruction or auxiliary module.

• Scaling with model size: Scaling the Qwen3 backbone from 4B to 8B to 14B produces consistent gains on unseen Maze and Mastermind, with the largest improvements in Episode 3, then Episode 2, and only modest improvement (potentially even a drop for 14B) in Episode 1. The authors interpret this as larger models allocating early interaction toward exploration and capitalizing on accumulated evidence later.

• Comparison context: Figure 1 compares average success rate across Maze and Mastermind over episodes for Orbit, GPT-4o, GPT-5.2 (high reasoning effort), and oracle algorithms for the test environments. The paper also notes that prior work PAPRIKA reports relatively weak performance on several environments, including 2% success rates on Mastermind.

• Evaluation scope: The evaluation uses two unseen environments, Maze and Mastermind, each with 256 distinct instances. The authors deliberately do not evaluate on training tasks or intra-environment variants, since success there saturates quickly and makes it hard to separate genuine in-context adaptation from environment memorization.

Methodology in Plain English

Each task is modeled as a partially observable, finite-horizon episodic decision process. The agent starts with an empty history, acts, observes a new state and reward, and appends everything to its context window. When an episode ends, the environment resets to a fresh initial state but keeps the same underlying rules, and the agent retains the entire cross-episode transcript. The agent's policy is just the LLM's forward pass conditioned on that transcript — no weights change at test time.

Training treats the whole multi-episode process as the unit of optimization. The researchers sample a task from a training set of five environments (Rock-Paper-Scissors, Minesweeper, Hangman, Wordle, Blackjack), let the model play several episodes, and score the resulting trajectory. Instead of using whatever shaped rewards each game provides, they ignore task-specific process rewards and use a single unified completion reward, with the trajectory-level reward equal to the number of successful task completions within the interaction budget. This prevents games with large intrinsic reward scales from dominating the gradient signal.

Optimization uses Group Relative Policy Optimization (GRPO), a policy-gradient method that compares groups of trajectories and uses their relative rewards as a baseline, with PPO-style clipped updates. GRPO never learns an explicit value function, so the learning signal depends only on aggregated outcomes rather than intermediate progress. The authors argue this outcome-driven structure fits their sparse completion reward and avoids reward hacking across heterogeneous tasks, and they cite the Kimi K1.5 technical report's observation that suppressing step-wise credit assignment can encourage exploration of diverse reasoning paths.

The concrete training configuration: Qwen3-8B as the base model for most experiments (Qwen3-4B, 8B, and 14B for the scaling study), a maximum context length of 32,768 tokens, temperature 1.0 and top-p 1.0 during training, a group of K = 4 trajectories per instance, batch size 64 trajectories across all model sizes, 100 optimization steps, learning rate 1×10⁻⁶, asymmetric clipping bounds of (0.2, 0.28), entropy and KL regularization both disabled, and thinking mode enabled. Trajectories exceeding the context limit are truncated and assigned zero reward, and each task instance is restricted to three episodes so that complete histories fit in context. Evaluation uses the recommended Qwen3 inference settings (temperature 0.6, top-p 0.95) with thinking mode on. Sections 4.3 and 4.4 report the checkpoint after 100 optimization steps; Section 4.5 reports the best-performing checkpoint within the first 100 steps. All experiments use the RLLM training framework.

Why This Matters

Impact on research. The paper reframes in-context reinforcement learning as a trainable capability rather than a prompt-time trick, and argues it is the first to demonstrate substantial and transferable improvements in in-context RL across diverse, unseen tasks. The authors also point out that the closest concurrent meta-RL work evaluates on environments identical to those seen during training, which makes it hard to isolate in-context learning from environment memorization — a distinction this paper's unseen-task protocol is designed to address.

Real-world applications (as suggested by the paper's framing and examples):

• Enterprise tool use. Figure 2 illustrates an agent probing unfamiliar APIs with rate limits and schemas, refining its strategy after errors, and then exploiting the resulting mental model on a later attempt — all without weight updates.

• Navigating unfamiliar interfaces or software. A deployed agent can treat early attempts as information gathering and improve within the same session.

• Operating in environments whose rules are not fully specified upfront, where hidden constraints cause sensible-looking first attempts to fail.

• Long-horizon decision-making agents that must recover from mistakes and refine strategy as experience accumulates, the "learn-at-inference-time" setting the paper argues current LLMs fail at.

Industry relevance. The framework is deliberately simple, avoids external memory and prompt-engineering scaffolding, and works on a relatively small open-source model, which matters for teams that cannot rely on frontier closed models. The paper reports that a meta-trained Qwen3-14B matches GPT-5.2's performance on the evaluated unseen tasks, and that scaling the backbone yields consistent gains — both of which are direct signals for practitioners about where to invest compute.

Future Directions

• Longer interaction horizons. The 32k-token context window of the Qwen3 base models is described as the primary constraint limiting the number of episodes (three per game). Extending to longer horizons with larger-context or memory-augmented architectures is called out as a natural next direction for richer temporal reasoning and long-term credit assignment.

• Scaling the number and diversity of training environments. The authors train on only five environments and say scaling both quantity and diversity is the next step toward understanding empirical scaling laws governing generalization.

• Improving training efficiency and stability. The paper notes significant potential to improve computational efficiency and stability through more advanced optimization and credit-assignment techniques.

• Isolating ICL from memorization more rigorously. The unseen-task protocol is a step in this direction, but the concurrent meta-RL framework the authors cite raises the open question of how much apparent in-context learning reflects genuine adaptation versus familiarity with the environment.

Target Audience

This paper is most useful for reinforcement learning and LLM post-training researchers working on agentic decision-making, in-context RL, and meta-learning; engineers building deployed agents that must operate in unfamiliar environments without retraining; and readers interested in whether in-context online learning is a trainable capability rather than an emergent property of scale. It will be hardest to follow without prior exposure to MDPs, policy-gradient methods, and GRPO-style optimization, though the central intuition — train the model to get better across repeated attempts at the same task — is accessible without that background.

Authors’ abstract

Large language models (LLMs) achieve strong performance when all task-relevant information is available upfront, as in static prediction and instruction-following problems. However, many real-world decision-making tasks are inherently online: crucial information must be acquired through interaction, feedback is delayed, and effective behavior requires balancing information collection and exploitation over time. While in-context learning enables adaptation without weight updates, existing LLMs often struggle to reliably leverage in-context interaction experience in such settings. In this work, we show that this limitation can be addressed through training. We introduce ORBIT, a multi-task, multi-episode meta-reinforcement learning framework that trains LLMs to learn from interaction in context. After meta-training, a relatively small open-source model (Qwen3-14B) demonstrates substantially improved in-context online learning on entirely unseen environments, matching the performance of GPT-5.2 and outperforming standard RL fine-tuning by a large margin. Scaling experiments further reveal consistent gains with model size, suggesting significant headroom for learn-at-inference-time decision-making agents. Code reproducing the results in the paper can be found at https://github.com/XiaofengLin7/ORBIT.