Long-Horizon RL Needs Muon

AI summary

Overview

Research area: Reinforcement learning for long-horizon LLM agents, specifically software-engineering agents that resolve repository-level issues, with a focus on optimizer choice and training-recipe design.

Technical level: Advanced. The paper assumes familiarity with policy-gradient methods (PPO-style clipping, importance sampling), advantage standardization, and the spectral-norm interpretation of matrix-valued optimizers such as Muon.

Scope: The paper evaluates a single-rollout asynchronous training recipe that pairs the Muon optimizer with token-averaged loss, truncated importance sampling, and auxiliary length and tool-error penalties, comparing it against SPO baselines across two model families.

What This Paper Is About

Long-horizon agents that fix software issues must chain together reasoning, tool calls, and environment feedback, but the only reliable learning signal is a sparse binary reward from whether a final patch passes tests. Because trajectories vary in length and rollouts are collected asynchronously from the policy being updated, the authors ask how the choice of optimizer interacts with reward shaping and loss aggregation under these conditions. Their goal is to determine whether Muon, combined with a specific set of recipe components, learns better than the Adam-based SPO baselines they compare against.

Key Contributions

1. A single-rollout asynchronous training recipe that combines Muon, token-averaged policy loss, truncated importance sampling (TIS), and auxiliary length and tool-error penalties for repository-level issue resolution.
2. A direct optimizer comparison between Adam (Kingma and Ba, 2014) and Muon (Jordan et al., 2024) within Single-stream Policy Optimization (SPO) (Xu and Ding, 2025), holding data, rollout budgets, termination rules, and loss aggregation fixed.
3. An integration of Muon with SPO++ (Ruan et al., 2026b) advantage standardization across valid tokens, plus asymmetric PPO clipping using the DAPO Clip-Higher settings of epsilon_low = 0.2 and epsilon_high = 0.28.
4. Ablations isolating the length penalty, the tool-error penalty, TIS, and token-level versus sequence-level loss aggregation, evaluated on two model families (MiniCPM5-2B and Qwen3.5-4B).

Main Findings

• Full recipe beats both baselines: The full recipe reaches higher training success rates than both SPO baselines later in training for both model families. The gain over SPO + Muon specifically supports combining the optimizer with the full set of recipe changes rather than using Muon alone.
• Loss aggregation interacts with the length penalty: Without the length penalty, the relative performance of token-level and sequence-level averaging varies across seeds, so token-level averaging is presented as part of the full recipe rather than as an unconditionally superior choice.
• Adam configurations decline: Across the recipe comparisons, training success continues to improve with the full recipe, while several Adam configurations decline. Figures 4 and 5 mark collapses in red and the full recipe in green.
• Ablations leave the tool interface untouched: Penalty ablations keep the tool interface and the binary outcome reward unchanged, so differences are attributable to the penalty terms rather than to changes in task setup.
• Reported summaries: Numerical summaries use recorded values, and matched comparisons use complete windows from runs with the same seed; Figure 2 uses mean success over updates 90–100 when that window is complete, and the last observed value otherwise. The truncated content provided does not report the specific success-rate percentages.

Methodology in Plain English

The agent is given a repository issue and interacts with the codebase through tools inside the Claude Code harness (version 2.1.118), using training tasks from R2E-Gym. When it finishes, its patch is run against the project's tests. It earns a reward of 1 only if it passes all fail-to-pass (F2P) tests, which check that the issue is resolved, and all pass-to-pass (P2P) tests, which check that nothing else broke; otherwise the reward is 0. Empty test sets and evaluations interrupted by infrastructure failures remain unscored.

Because the reward is just a yes-or-no signal at the very end, the recipe adds shaping terms at each turn. Invalid actions and designated tool, command, session, or execution errors produce a tool-error term of −0.5, and long responses incur a length penalty reaching −0.5 for responses exceeding 2,048 tokens, with the trajectory total proportionally capped; the final turn contributing to the policy loss is exempt from this penalty. The combined length and tool-error term is clipped to between −0.25 and 0.25 and added to each valid token's advantage in that turn before standardization. Responses containing predefined command-rejection messages trigger an additional trajectory-level penalty.

Two further pieces address the asynchronous setup. First, because rollouts are generated by a sampling policy that may differ from the policy the training backend evaluates, truncated importance sampling reweights each token by the ratio of the two policies' probabilities, capped at C = 2, applied directly to the masked token-level PPO losses without batchwise normalization. Second, the loss is averaged over tokens across the batch (token-level averaging) rather than averaging each trajectory's mean equally (sequence-level averaging), which changes how much weight long trajectories carry.

The Muon update is applied to hidden weight matrices, using Newton–Schulz iterations to approximate orthogonalization of a momentum-based update, while embeddings, the output head, normalization parameters, and vector parameters use AdamW. Both optimizers collect task-specific outcome statistics online, without SPO's offline initialization. Every configuration collects its own trajectories through asynchronous rollouts, and comparisons pair runs with the same seed.

Why This Matters

Impact on research: The paper argues that optimizer choice is not a neutral implementation detail in long-horizon agentic RL; Muon's spectral-norm-constrained update direction behaves differently from Adam when combined with sparse outcome rewards, asynchronous rollouts, reward shaping, and particular loss aggregation. This makes optimizer selection a first-class design decision in agentic recipes and connects the Muon literature, previously studied in mathematical and interactive RL, to multi-turn software repair.

Real-world applications:

• Automated repair of bugs filed against production code repositories.
• Codebase maintenance tooling that proposes and validates patches against existing test suites.
• Developer-assistant agents that navigate, edit, and test code as part of an IDE or CI workflow.
• General long-horizon agent systems that coordinate tool use over extended interactions where only end-of-episode feedback is available.

Industry relevance: Repository-level issue resolution is directly tied to engineering productivity, and the training signal used here — executable tests — is already available in most software organizations. Recipes that learn reliably under asynchronous, single-rollout collection are cheaper to run than synchronous multi-trajectory schemes, and the paper's emphasis on training success rate during the run speaks to the practical question of when a training job is stable enough to keep.

Future Directions

• Whether the advantage of Muon over Adam holds at model scales, rollout budgets, or task suites beyond MiniCPM5-2B and Qwen3.5-4B is not reported and remains open.
• The interaction between the length penalty and loss aggregation varies across seeds; identifying the conditions that determine which averaging scheme wins is unresolved.
• The paper's scope is training success rate, and the truncated content does not report held-out or final evaluation performance, leaving the generalization question open.
• The relative contribution of each recipe component is studied through ablations, but the content provided does not report which component drives most of the gain when penalties, TIS, and aggregation are varied jointly rather than one at a time.

Target Audience

Researchers and engineers working on reinforcement learning for LLM agents, particularly those training multi-turn agents against execution-based or test-based rewards. It is also relevant to practitioners selecting optimizers and loss aggregation schemes for asynchronous RL training pipelines, and to readers already familiar with PPO-style policy optimization who want to understand how Muon's update geometry interacts with reward shaping and importance correction.

Authors’ abstract

Long-horizon agents must coordinate reasoning, tool use, and environment feedback over extended interactions. Learning from sparse outcome rewards requires weighting trajectories of different lengths and updating the policy while rollouts are still being collected. We examine a single-rollout asynchronous recipe combining Muon, token-averaged loss, truncated importance sampling, and auxiliary length and tool-error penalties. Single-rollout sampling allows asynchronous learning without waiting for multiple trajectories for the same issue. We study optimizer choices and recipe components for repository-level issue resolution across two model families. The evaluated training runs show higher training success with the full recipe than with the baselines.