ACLArena: Agent Continue Learning in Multi-stage Post-training

AI summary

Overview

Research area: Continual learning for LLM agents trained through multi-stage post-training, spanning reinforcement learning (RL), on-policy distillation (OPD), self-distilled fine-tuning (SDFT), and weight-space model merging.

Technical level: Advanced. The paper assumes familiarity with RL post-training, KL-divergence-based distillation, LoRA adapters, and model merging.

Scope: The paper introduces ACLArena, a controlled testbed for studying Agent Continual Learning (ACL), diagnoses why sequentially trained agents forget earlier capabilities, compares three practical consolidation paradigms, and proposes a new recipe called Mixture of Low-Rank Experts (MLE).

What This Paper Is About

Industrial-grade LLM agents must acquire different capabilities (reasoning, tool use, instruction following) in separate training stages, and each new stage tends to overwrite what came before. The paper builds a reproducible multi-stage pipeline to measure that forgetting and transfer, then asks whether previously trained checkpoints can be reused in behavior space or parameter space to recover lost skills without sacrificing newly learned ones. Its goal is to replace ad-hoc post-training recipes with a controlled comparison and a practical mitigation strategy.

Key Contributions

1. A realistic benchmark. ACLArena is a multi-stage post-training pipeline over heterogeneous environments that exposes both forgetting and generalization, using a fixed curriculum (Math → Search → E-commerce → Instruction Following).
2. A mechanistic study. Using sequential training as a diagnostic baseline, the authors characterize forgetting and transfer at both the model level (PCA of checkpoints, cosine similarity) and the token level (prediction consistency conditioned on predictive entropy), and provide optimized comparisons of three ACL paradigms: Multi-teacher Mixed On-policy Distillation (MMOPD), Self-distilled Fine-tuning (SDFT), and Model Merging (MM).
3. A practical mitigation strategy. Guided by the analysis, they propose Mixture of Low-Rank Experts (MLE), which pairs offline replay over high-quality trajectories with a routed network of multiple LoRA experts, each specialized via RL.
4. An empirical evaluation. Experiments across four reasoning and agentic tasks, under both in-domain and out-of-domain settings, with all main results reported as mean ± standard deviation over three independent runs.

Main Findings

• Sequential training is non-monotonic. Capability evolution is "highly task-dependent": early stages transfer beneficially (Seq-Math raises NQ from 13.3 to 22.0, and Seq-Search largely preserves math), but the E-commerce stage causes pronounced interference, dropping AIME26 from 23.33 to 6.04 and NQ from 45.2 to 14.6, with multi-hop search falling from 37.4 to 9.4.
• Every capability decays after its own stage. Search is forgotten most abruptly (NQ 45.2 → 14.6 at the start of E-commerce, recovering only to 33.5); math decays gradually then collapses (25.83 → 23.33 after search → 6.04 after E-commerce, rebounding only to 10.21); E-commerce is the most durable but still declines (τ³-Retail 34.0 → 29.6).
• The final checkpoint does not dominate its predecessors. Seq-Final (Seq-IF) ends below earlier peaks on AIME26 (10.21 vs 25.83), NQ (33.5 vs 45.2) and τ³-Retail (29.6 vs 34.0), while reaching 84.8 on IF-Eval.
• Model-level diagnosis: partially aligned parameter updates. In the PCA projection, the four single-task oracles occupy clearly distinct parameter-displacement directions; each stage moves toward its own oracle but never aligns perfectly, and the more stages a checkpoint has passed through, the further its starting point has drifted from Base — evidence that accumulated cross-stage interference, rather than single-task difficulty, blocks recovery.
• Token-level diagnosis: change is localized. Under fixed-prefix evaluation, low-entropy positions (the large majority) remain highly stable across stages; almost all prediction change is carried by the small high-entropy minority where the reference checkpoint is uncertain.
• MMOPD recovers math substantially. Starting from Seq-Final, MMOPD raises AIME26 from 10.21 to 21.25, NQ to 45.2, single-hop search to 53.7, and multi-hop search to 32.8.
• SDFT reaches the highest AIME26 among consolidation methods. SDFT gives AIME26 22.29 and NQ 48.3, but τ³-Retail falls to 22.3 and IF-Eval drops to 53.6.
• Plain model merging is the weakest consolidator. Uniformly averaging the four sequential-stage checkpoints yields AIME26 14.79, NQ 39.0 and τ³-Retail 11.8, though it preserves MMLU (80.3) and GPQA (41.8).
• MLE gives the most balanced profile. MLE scores AIME26 21.04, NQ 49.7, τ³-Retail 32.9, IF-Eval 85.0, GPQA 42.4, single-hop search 57.7, multi-hop search 38.6, τ³-Mock 66.7 and IF-Bench 29.1, with the best values in the table's GPQA, single-hop, multi-hop and τ³-Mock columns.
• Specialist oracles do not transfer. Independently trained per-task experts show clear specialization — the search expert reaches 49.9 on NQ and 56.0 on single-hop search, the IF expert reaches 86.2 on IF-Eval — but their gains do not carry consistently across domains.
• Three design principles. (i) Task-specific post-training induces distinct, only partially aligned optimization directions; (ii) consolidating heterogeneous capabilities in one shared model creates persistent trade-offs; (iii) SFT and RL are complementary — SFT establishes valid task behavior with broader, more directionally consistent updates, while RL makes smaller, policy-local refinements.
• MOPD's motivation. The authors argue training collapse on agentic search tasks is structural to reverse KL: mode-seeking is desirable at low-entropy tokens but prunes viable branches at high-entropy decision points, so they add a mass-covering forward-KL term only where teacher entropy exceeds a threshold α, evaluated in closed form over a top-q token set.

Methodology in Plain English

The authors start from a base model (Qwen3-8B-Base) and train it through four stages in a fixed prerequisite order: math reasoning first, then search (a single tool), then E-commerce (up to fifteen tools), then instruction following. Math and instruction following are trained directly with a critic-free RL algorithm; the agentic stages first use rejection-sampled fine-tuning as a cold-start to teach tool-use format, then critic-free RL. This sequential run produces four checkpoints, Seq-Math, Seq-Search, Seq-E-commerce and Seq-IF (the final one, called Seq-Final).

They then run two diagnostics. At the model level they unit-normalize checkpoints and project them onto the first two principal components, comparing against independently trained single-task oracles. At the token level they fix a sequence generated by Seq-Math and force every later checkpoint to score the same positions, recording whether Seq-Math's original token is still ranked first, split by the reference model's predictive entropy.

For consolidation they compare three routes. MMOPD treats all four stage checkpoints as frozen teachers and the final checkpoint as the student, training on a balanced mixture of prompts drawn from every stage's distribution, with each prompt routed to the teacher that owns its domain (rather than ensembling teacher logits); environment tokens are masked out. SDFT trains a per-task oracle policy in isolation, filters its rollouts on final-reward correctness, well-formed tool invocation and protocol compliance while deduplicating near-identical traces, then fine-tunes the consolidated student on the filtered prompt–trajectory pairs, with ablations over mixing ratio and curriculum ordering. Model merging averages weights using task vectors, θ_merge = θ_0 + Σ λ_k (θ_k − θ_0), explored under uniform, grid-searched and capability-balanced weighting.

The proposed MLE method first rapidly learns multiple domains' behaviors through SDFT, then attaches LoRA adapters and refines each with RL, so each stage yields its own expert set that a routing mechanism composes into a single deployable model. Implementation uses the Slime RL framework with an agentic rollout loop, SDFT rollouts with group size 8, two NVIDIA H200 nodes. Evaluation spans AIME 2026 (avg@16), GPQA-Diamond and MMLU-Redux for math; NQ in-domain with PopQA and TriviaQA for single-hop and 2WikiMultiHopQA, Bamboogle, HotpotQA and MuSiQue for multi-hop search; τ³-Bench retail (in-domain) with telecom and mock domains (out-of-domain); and IF-Eval (in-domain) with IF-Bench (out-of-domain).

Note: the provided text is truncated partway through Section 5.2, so the remaining discussion and the appendices (hyperparameters, cosine-similarity analysis, token-analysis details, the agentic-search collapse study) are not available here.

Why This Matters

The paper reframes industrial agent training as a continual-learning problem rather than a series of independent fine-tuning jobs, and shows that stacking stages can leave an agent worse on several capabilities than an early checkpoint. Its controlled comparison makes the trade-off surface explicit — MMOPD and SDFT each recover different parts of the capability profile, model merging trades away task success for stability, and MLE balances them — which gives practitioners evidence for choosing a consolidation route instead of guessing.

Real-world applications:

• E-commerce agents that must coordinate up to fifteen tools while retaining search and reasoning behavior.
• Retail and telecom customer-service agents evaluated through τ³-Bench, where task success rate is the deployment metric.
• Search and retrieval assistants requiring both single-hop and multi-hop answering without losing instruction adherence.
• General-purpose assistants where a final instruction-following and safety-alignment stage must not erase reasoning or tool-use skills.

Industry relevance centers on cost: the paper notes that classical continual-learning approaches are impractical for agent training because replay requires retaining costly environments and reward pipelines, task-specific modules complicate deployment, and joint retraining becomes increasingly expensive as capabilities accumulate. A recipe that consolidates capabilities through LoRA experts plus routing, or through filtered trajectory replay, is attractive precisely because it avoids re-running every environment.

Future Directions

• Curriculum generality. The authors deliberately study one fixed curriculum (Math → Search → E-commerce → IF) rather than arbitrary task permutations; how sensitive the findings are to stage order, and whether a difficulty- or prerequisite-aware ordering changes forgetting, remains open.
• Routing and expert scaling. MLE composes per-stage LoRA experts through a routing mechanism; how routing behaves as the number of stages, domains and experts grows — and whether it can be extended beyond LoRA — is not settled.
• Closing the gap to oracles. Seq-Math reaches the best AIME26 in the tables (25.83) and the search expert the best NQ (49.9), while no consolidation method matches all oracles at once; how close a single deployed model can get to the union of specialist performance is unresolved.
• Better mixing and ordering for SDFT. The paper flags that SDFT is highly sensitive to the task-mixing distribution and the presentation order of training data, and reports ablations ranging from fully interleaved shuffling to staged, difficulty-graded schedules — refining those schedules is an obvious next lever.

Target Audience

Researchers and engineers working on LLM post-training, RL fine-tuning and agent development who need to merge capabilities learned in separate stages; also practitioners building multi-tool industrial agents, and students of continual learning interested in how classical forgetting concepts reappear in modern LLM training pipelines.

Authors’ abstract

Building general-purpose agents for industrial deployment requires integrating multiple capabilities, each typically acquired at a distinct stage of training. Yet there is currently no well-established recipe for Agent Continual Learning (ACL), with little understanding of the trade-offs among existing integration paradigms. To address this gap, we introduce ACLArena, a framework for comprehensively studying, analyzing, and evaluating ACL. We first build a sequential training pipeline and conduct an in-depth analysis that explains the mechanisms of forgetting and generalization from two complementary perspectives, the model level and the token level. Guided by these analyses, we systematically compare multi-teacher on-policy distillation, self-distilled fine-tuning, and model merging to assess their ability to recover previously learned capabilities while preserving newly acquired ones. Through extensive experiments, we develop a detailed understanding of how capabilities transfer across stages. Finally, we propose a new ACL recipe that combines offline replay over high-quality trajectories with a routed network of multiple LoRA experts each specialized via RL, substantially improving the agent's ability to learn across multiple domains. Comprehensive experiments on four reasoning and agentic tasks, evaluated under both in-domain and out-of-domain settings, demonstrate the value of our analysis and the effectiveness of our approach.