Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models

AI summary

Overview

Research area: Post-training methods for Looped Language Models (LoopLMs) — models that reuse the same Transformer blocks across multiple recurrent computation steps before emitting each token — with a focus on on-policy distillation, reinforcement learning with verifiable rewards, and cross-loop supervision.

Technical level: Advanced. The paper combines a practical training recipe with formal analysis involving reverse-KL gradients, Lipschitz-gradient descent bounds, and empirical tangent kernels.

Scope: The paper introduces LoopOPD and Dynamic LoopOPD (D-LoopOPD), two cross-loop on-policy distillation methods that use a LoopLM's own terminal loop as a "compute-privileged" teacher for an intermediate loop, plus theory and experiments on Ouro-Thinking models.

What This Paper Is About

Looped Language Models gain reasoning ability by running the same parameters through more recurrent computation steps ("loops"), but existing post-training methods either give sparse reward signal only at the final loop, require memory that grows with the number of loops, or depend on an external stronger teacher that may not exist. This paper asks whether a LoopLM can supervise its own intermediate loops using its extra recurrent computation as the source of teacher signal. The proposed methods distill the terminal loop's next-token distribution into an intermediate loop on student-generated rollouts, and then continually refresh that teacher as training proceeds.

Key Contributions

1. LoopOPD. A cross-loop on-policy distillation framework that treats an intermediate loop policy as the student and a frozen terminal loop policy as a compute-privileged teacher, distilling the terminal loop's next-token distributions on student-generated rollouts — with no external teacher or privileged information. In experiments, the student loop is ℓ = 2 and the terminal loop is L = 4.

2. Dynamic LoopOPD (D-LoopOPD). A variant that refreshes the terminal loop teacher at each optimization step, since student and teacher share parameters. This forms a closed teacher–student feedback loop the authors call recurrent self-improvement.

3. Theoretical analysis of cross-loop update transport. The paper characterizes how a distillation update propagates across loop depths. Proposition 1 gives a descent guarantee for the step-frozen teacher objective under an M_t-Lipschitz gradient and step size 0 < η_t < 2/M_t. Theorem 1 derives sufficient conditions (positive gradient alignment γ_ℓ,t > 0 and γ_L,t > 0 at both loop depths, plus a step-size bound) under which a single update decreases task surrogates locally at both the intermediate and terminal loops simultaneously.

4. Empirical validation on Ouro-Thinking models. Experiments at Ouro-1.4B-Thinking and Ouro-2.6B-Thinking show improved mathematical reasoning over GRPO, RLTT, OPSD, and (for the 1.4B scale) OPD, with cross-loop gains at every loop depth and transfer to general reasoning and code generation despite mathematical-only training.

Main Findings

• Mathematical reasoning improves on Ouro-2.6B-Thinking. Averaging Avg@16 over AIME 2024, AIME 2025, and AIME 2026, the vanilla model scores 55.49 and GRPO scores 56.60, while LoopOPD reaches 59.37 and D-LoopOPD reaches 62.36.

• Largest single-benchmark gain on AIME 2024. D-LoopOPD raises terminal loop AIME 2024 Avg@16 from the vanilla 62.08 to 71.67. D-LoopOPD improves over LoopOPD by 6.46, 1.46, and 1.05 points on AIME 2024, AIME 2025, and AIME 2026, respectively.

• Competitive with larger non-looped models on AIME 2024. D-LoopOPD's 71.67 Avg@16 surpasses vanilla Qwen3-8B at 71.25 and approaches vanilla Qwen3.5-9B at 72.08, with fewer parameters.

• Cross-loop gains at every depth. On Ouro-2.6B-Thinking, D-LoopOPD raises Avg@16 from 39.16 to 47.50 at the directly supervised student loop 2 (a gain of 8.34 points), and by 5.41 and 9.59 points at loops 3 and 4, reaching 66.87 and 71.67.

• Gains transfer beyond mathematics. Although trained only on mathematical data (DAPO-Math-17k), D-LoopOPD improves over the vanilla Ouro-2.6B-Thinking on all five evaluated non-math benchmarks: +0.58 on MMLU, +1.06 on MBPP, +1.33 on MBPP+, +0.61 on HumanEval, and +0.61 on HumanEval+. It achieves the best results in the model group on MMLU, MBPP, and MBPP+.

• Benchmarks where the gain is smaller. On Ouro-2.6B-Thinking, D-LoopOPD's BeyondAIME score of 34.00 is described as comparable to the vanilla model and GRPO (both 35.00). On OlympiadBench, D-LoopOPD is the best in the Ouro-2.6B group — reported as 65.77 in the body text and 65.78 in Table 2.

• Higher training reward and validation than frozen-teacher distillation. Training dynamics show D-LoopOPD achieving higher training reward and higher validation Avg@16 than LoopOPD, both measured at the student loop ℓ = 2.

• Dynamic teachers maintain higher entropy. D-LoopOPD maintains higher teacher and student token-distribution entropy than LoopOPD through most of training, with the gap narrowing toward later steps, which the authors link to more diverse student rollouts and broader exploration.

• Student depth matters, and loop 2 is best. In an ablation using loop 1, loop 2, or loop 3 as the student with the teacher fixed at loop 4, loop 2 gives the best terminal performance (loop 4 Avg@16 from 62.08 to 71.67). Loop 3 starts close to the initial terminal policy (61.46 versus 62.08), leaving less room for distillation. Loop 1 has the largest teacher–student gap but very low standalone accuracy: its student Avg@16 remains at 0.63 after training, while the terminal loop improves to 68.54.

• Ouro-1.4B-Thinking also benefits. LoopOPD and D-LoopOPD outperform the vanilla model, GRPO, and OPD on average across the three AIME benchmarks; LoopOPD achieves the best OlympiadBench score and D-LoopOPD the best BeyondAIME score in that group.

• Baseline behaviors in the tables. RLTT and OPSD do not beat the vanilla Ouro models on the averaged metrics shown; GRPO gives modest gains over vanilla.

Methodology in Plain English

The setup. A LoopLM applies its shared Transformer blocks for L iterations before predicting each token, so it can produce a next-token probability distribution at every loop depth. Standard generation uses only the terminal loop (loop L). The intuition is that the terminal loop, having done more computation, is a stronger policy than an intermediate loop — so it can act as a teacher without needing extra parameters or extra information.

LoopOPD. Pick an intermediate loop ℓ (the paper uses ℓ = 2) as the student and the terminal loop L = 4 as a frozen teacher. Sample responses from the student policy itself, then for every prefix of each response, minimize the reverse KL divergence between the student's loop-ℓ distribution and the frozen teacher's loop-L distribution. Because both distributions are computed on the same prefixes from the same model family, this avoids the teacher–student context mismatch that affects on-policy self-distillation with privileged information.

D-LoopOPD. Since student and teacher share the same parameters, updating the student also changes the terminal loop. D-LoopOPD exploits this: at each optimization step it uses the current terminal loop policy as the teacher, with a stop-gradient applied to the teacher distribution so gradients flow only through the student. The refreshed teacher then supervises the next step, forming a feedback loop.

Why not simpler alternatives. GRPO gives reward credit only at the terminal loop. RLTT spreads reward across loops but retains per-loop log-probabilities, so memory scales linearly with loop count. OPD needs an external, stronger teacher — and for Ouro-2.6B-Thinking no stronger same-family LoopLM is available.

Theory. The authors analyze one gradient step. They show that if the empirical distillation loss has a Lipschitz gradient and the step size is small enough, the batch-averaged KL to the step-frozen teacher strictly decreases (Proposition 1). They then show how the parameter update transports into first-order logit changes at both loop depths, expressed via a cross-loop block of the empirical tangent kernel. Defining alignment terms γ_d,t between the distillation gradient and a task surrogate gradient at depth d, they give conditions under which one update decreases the surrogate at both the intermediate and terminal loops at once (Theorem 1).

Evaluation. All post-training methods are trained on DAPO-Math-17k. Evaluation uses AIME 2024, AIME 2025, AIME 2026, OlympiadBench, and BeyondAIME for mathematics; MMLU for general reasoning; and MBPP, MBPP+, HumanEval, and HumanEval+ for code. All evaluations use a maximum response length of 31,744 tokens. AIME results are reported with Avg@16 and Pass@16, OlympiadBench and BeyondAIME with accuracy, MMLU with accuracy, and the code benchmarks with Pass@1.

Why This Matters

Impact on research. The paper reframes loop depth as a source of supervision rather than just a source of compute. It offers an alternative to external-teacher distillation and to reward-based methods whose memory grows with loop count, and it provides a formal account of when a distillation update helps at more than one loop depth. It also adds a concrete data point to the ongoing question of how to post-train recurrent-depth and looped architectures effectively.

Real-world applications (potential, extrapolated from the method):

• Post-training compact reasoning models for deployment where memory and teacher-model budgets are constrained.
• Improving mathematical and quantitative reasoning assistants used in education or technical workflows.
• Transfer-style training pipelines where only domain-specific data (for example, mathematics) is available but broad capability is desired.
• Code generation and general question-answering systems that can benefit from latent recurrent compute without added parameters.

Industry relevance. The method requires no external teacher checkpoint and no privileged information, which lowers the cost and complexity of post-training pipelines. Because the intermediate-loop student and terminal-loop teacher hold the same parameter count, memory pressure comes from activation and rollout handling rather than a second large model. The reported results on Ouro-1.4B and Ouro-2.6B also suggest a route to reasoning gains at parameter scales that are practical to serve.

Future Directions

• Choosing the student loop systematically. The ablation shows loop 2 is best among loops 1–3 for Ouro-2.6B-Thinking with teacher loop 4; a principled rule for picking ℓ (and L) across model sizes and tasks is left open.

• Extending the alignment conditions into practice. Theorem 1 requires positive gradient alignment γ_ℓ,t and γ_L,t plus a step-size bound; how to monitor or enforce these quantities during real training is not reported.

• Broadening beyond mathematical training data. The observed gains on MMLU, MBPP, MBPP+, HumanEval, and HumanEval+ come from mathematical-only training; whether direct training on other domains compounds these gains is not reported.

• Scaling to more loops and larger models. Experiments use L = 4 and models at 1.4B and 2.6B parameters; behavior at greater loop counts or larger parameter scales is not reported.

• Releasing code and checkpoints. The paper states that code and model checkpoints will be released upon acceptance, which would enable independent reproduction and comparison.

Target Audience

Researchers and engineers working on large language model post-training, reinforcement learning from verifiable rewards, knowledge distillation, and recurrent or looped architectures. It is also relevant to readers interested in latent/recurrent computation and test-time compute scaling, and to practitioners who need an external-teacher-free method for training reasoning models under memory constraints. Readers without a background in KL-based policy objectives or gradient analysis will find the method sections accessible but the theory section demanding.

Authors’ abstract

Looped Language Models (LoopLMs) offer a parameter efficient approach to scaling reasoning by reusing shared parameters across recurrent computation steps. Despite their promise, effective post-training of LoopLMs remains challenging. Existing approaches either provide reward based supervision that is sparse or costly to extend across loops, or rely on external teachers or privileged information, leading to limited teacher availability or teacher-student context mismatch. To address these limitations, we introduce LoopOPD, a cross-loop on-policy distillation framework that uses additional recurrent computation within a LoopLM as its own source of supervision. LoopOPD uses a frozen terminal loop policy as a compute privileged teacher for an intermediate loop student on student generated rollouts, providing dense supervision without an external teacher or privileged information. We further propose Dynamic LoopOPD (D-LoopOPD), which continually refreshes the terminal loop teacher as the shared model parameters are updated, enabling recurrent self-improvement. We characterize how distillation updates propagate across loop depths and derive sufficient conditions under which a single update yields simultaneous local improvement at both loop depths. Experiments on Ouro-Thinking models show that LoopOPD improves mathematical reasoning, while D-LoopOPD yields further gains through dynamic teacher updates. Despite being trained only on mathematical data, the resulting models also improve on general reasoning and code generation benchmarks, demonstrating that recurrent computation can serve as an effective source of supervision for LoopLMs. Our code and model checkpoints will be released upon acceptance.