Wait, Wait, Wait... Why Do Reasoning Models Loop?

AI summary

Overview

Research area: Machine learning / large language model reasoning — specifically the failure mode of repetition ("looping") in chain-of-thought generation.

Technical level: Intermediate. The paper is readable with a basic grounding in LLM inference (greedy decoding, temperature sampling, distillation), though the synthetic graph task and probability arguments require some comfort with machine learning formalism.

Scope: The paper combines a large-scale empirical study of looping in open reasoning models with a controlled synthetic graph-reasoning testbed to isolate the mechanisms by which training errors cause looping, and to assess whether temperature is a genuine fix or merely a workaround.

What This Paper Is About

Reasoning models such as DeepSeek-R1 produce long chains of thought, but at low temperatures or under greedy decoding they frequently get stuck repeating the same text — a failure mode that model providers universally recommend avoiding by raising the temperature. The paper asks why looping happens in the first place, and whether temperature addresses the root cause or only masks it. To answer this, the authors argue that looping stems from systematic errors in how a model learns its training distribution, and they build a minimal synthetic task where those errors can be studied in isolation.

Key Contributions

1. Empirical characterization of looping in open reasoning models. A systematic study across DeepSeek-distilled Qwen/Llama, OpenThinker3, Phi-4-reasoning, and QwQ-32B on AIME problems, showing that looping is universal at low temperature, decreases with model capacity, is far worse in distilled students than their teachers, and is exacerbated by problem difficulty.

2. Identification of learning errors as the root cause. The paper argues that the capacity gap between teacher and student, and the difficulty dependence of looping, both point to imperfect learning of the training distribution rather than a decoding artifact.

3. A synthetic star-graph reasoning testbed. Building on Bachmann and Nagarajan (2024), the authors define a random-walk path-finding task in which "progress-making" and "cyclic" actions can be precisely defined, allowing controlled experiments with small Transformers trained from scratch.

4. Two concrete looping mechanisms plus an analysis of temperature. They show (i) risk aversion induced by hardness of learning, and (ii) an inductive bias toward temporally correlated errors — and argue that temperature mitigates symptoms without repairing the underlying learning errors.

Main Findings

• Looping is pervasive at low temperature and shrinks with temperature. Every open reasoning model tested looped under greedy or near-greedy decoding, and looping rates fell monotonically as temperature increased — consistent with earlier "neural text degeneration" results for non-reasoning models.

• Larger models loop less; distilled students loop far more than their teachers. Within the Qwen family, looping ranged from 76% at 1.5B parameters down to 37% at 32B. OpenThinker3-1.5B looped in roughly 30% of greedy responses while its teacher QwQ-32B looped in only about 4%. A perfectly distilled student could not substantially exceed its teacher, so this gap is evidence of learning error.

• Harder problems elicit more looping. After ranking AIME problems by accuracy at temperature 0.8, the hardest quintile looped dramatically more at temperature 0 (e.g., 51% vs. 13% for OpenThinker3-1.5B). Phi-4-reasoning was the lone exception.

• Risk aversion arises when a hard progress action competes with an easy cyclic action. Proposition 3.1 formalizes this: if a learner cannot distinguish n "hard" actions from one another, maximum-likelihood training spreads the probability mass on each hard action by a factor of 1/n, while the easy action retains its full mass p. Once p exceeds (1−p)/n, greedy decoding selects the cyclic action indefinitely.

• Transformers overestimate the easy cyclic action. In the graph task, at roughly 91% of root positions the trained model assigned more than the training probability (0.3) to the reset action, with a mean of about 0.33 — an additional force that strengthens looping beyond the 1/n spreading effect.

• A second mechanism: temporally correlated errors. Even with no hardness at the decision point, the trained model's small deviations at the root (e.g., 0.2 ± 0.05 instead of exactly 0.2 for n = 5) persist across time. Across consecutive visits to the root, the top-probability child was the same 98.7% of the time and the second-highest child 72.8% of the time — far above chance, and not explainable by self-conditioning, since the second-highest child is never actually sampled under greedy decoding.

• Margins reduce but do not eliminate the inductive bias. Training traces that explicitly discourage revisiting already-visited root children cut looping and temporal correlation substantially (agreement dropped to 75.3% and then 52.5% for the top child at margins of 0.05 and 0.1), but agreement never reached the 0% a perfect greedy student would achieve.

• Repetition is self-reinforcing. As a model repeats text, its confidence in continuing the repetition grows, acting as a catalyst that amplifies whichever looping mechanism is already active.

• Temperature is a stopgap, not a solution. At temperature 1 in the graph task, the learned model's average response length was 108.1 tokens versus 24.4 for a perfect learner — roughly a 4.4× blow-up even after looping had largely disappeared. The same pattern appears in real models: OpenThinker3 students produce longer chains than their teachers even at high temperature. The model still assigns too little probability to progress-making actions; temperature only adds exploration noise on top.

Methodology in Plain English

The work proceeds in two stages.

First, measurement on real models. The authors take a set of open reasoning models spanning several sizes and training recipes, run them on AIME 2024 and 2025 problems, and sample 20 chains per (problem, model, temperature) triple across temperatures from 0 to 1.0. They define a response as "looping" if some 30-token sequence appears at least 20 times within it, then plot looping rate, accuracy, and response length against temperature. This definition is deliberately blunt but robust — it captures hard repetition of long spans rather than incidental phrase reuse.

Second, controlled experiments. Because the training distribution of a real reasoning model is too complex to analyze directly, the authors construct a stripped-down alternative: path-finding on a "star graph." A star graph has a root node with several spokes leading to distinct leaves, plus (in their modification) a separate start node connected only to the root. Each training example is a token sequence containing an edge list, a start, a goal, and a randomly generated walk. The walk moves toward the goal with probability 0.7 and resets to the start with probability 0.3, mimicking a chain of thought that sometimes progresses and sometimes backtracks. Small decoder-only Transformers (about 85M parameters) are trained from scratch on these traces.

This setup gives the authors precise vocabulary: an action is the choice of next node, a cyclic action returns to an already-visited node, and a progress-making action moves closer to the goal. By adjusting graph depth and the walk's transition probabilities, they can independently turn on or off the two mechanisms they want to study — hardness at the root in one experiment, plain uniform child selection in another — and then measure looping as a function of temperature.

Why This Matters

Impact on research. The paper reframes looping from a decoding quirk into a symptom of distributional learning error. That shift matters because it implies the standard mitigation — crank up the temperature — is treating the symptom. It also connects reasoning-model behavior to classical questions about whether randomness is fundamentally necessary for search, and it provides a reusable, minimal testbed for studying chain-of-thought pathologies without the confound of a real model's unknowable training distribution.

Real-world applications:

• Deployment of open reasoning models, where practitioners need principled guidance on sampling settings rather than folklore, and where students distilled onto small hardware are precisely the models most prone to looping.
• Distillation pipelines, which could bias teacher-generated traces toward exploring fresh actions at repeated decision points — a training-time intervention the paper's margin experiments directly support.
• Inference cost planning, since high-temperature chains that avoid looping are still longer than necessary, meaning token budgets and latency estimates for reasoning models are systematically miscalibrated.
• Evaluation and monitoring, where looping rate at low temperature could serve as a cheap diagnostic for how well a model has learned its training distribution.

Industry relevance. Reasoning models are expensive to serve, and a substantial fraction of that cost is wasted on repeated text. Understanding that the waste stems from learning error rather than an easily tuned sampling parameter gives labs a reason to invest in training-time fixes — better data curation, repetition-aware losses, or trace design — instead of shipping high-temperature defaults that trade one failure mode for another.

Future Directions

• Training-time interventions that directly reduce learning errors. The paper only sketches these; concrete proposals include unlikelihood-style losses penalizing cyclic continuations, curriculum designs that make progress-making actions easier to distinguish, and distillation traces biased toward new actions at repeated decision points.

• Extending beyond the synthetic task. The star-graph setup isolates mechanisms cleanly but strips away the linguistic and semantic complexity of real chain-of-thought. Whether risk aversion and temporal correlation fully account for looping in production models remains open.

• The conjecture that some problems are hard enough to loop any model. The observation that harder problems induce more looping suggests there may be no model size that eliminates it entirely. Testing this would require problems of controlled and escalating difficulty at scale.

• A principled theory of when temperature helps and by how much. The paper shows the benefit of temperature scales with the size of the underlying learning error, but a quantitative relationship between error magnitude and required sampling entropy has not been established.

Target Audience

This paper is most valuable to researchers working on reasoning models, chain-of-thought training, and knowledge distillation, as well as engineers responsible for deploying reasoning models where inference cost and output reliability matter. It also speaks to anyone studying repetition and degeneration in autoregressive generation more broadly, and to theoretically inclined readers interested in how maximum-likelihood training behaves when a target action is indistinguishable from many alternatives. Readers looking for a ready-made drop-in fix will not find one here — the paper's contribution is diagnostic, and its proposed remedies are training-time directions rather than immediate patches.

Authors’ abstract

Reasoning models (e.g., DeepSeek-R1) generate long chains of thought to solve harder problems, but they often loop, repeating the same text at low temperatures or with greedy decoding. We study why this happens and what role temperature plays. With open reasoning models, we find that looping is common at low temperature. Larger models tend to loop less, and distilled students loop significantly even when their teachers rarely do. This points to mismatches between the training distribution and the learned model, which we refer to as errors in learning, as a key cause. To understand how such errors cause loops, we introduce a synthetic graph reasoning task and demonstrate two mechanisms. First, risk aversion caused by hardness of learning: when the correct progress-making action is hard to learn but an easy cyclic action is available, the model puts relatively more probability on the cyclic action and gets stuck. Second, even when there is no hardness, Transformers show an inductive bias toward temporally correlated errors, so the same few actions keep being chosen and loops appear. Higher temperature reduces looping by promoting exploration, but it does not fix the errors in learning, so generations remain much longer than necessary at high temperature; in this sense, temperature is a stopgap rather than a holistic solution. We end with a discussion of training-time interventions aimed at directly reducing errors in learning.