Stop Unnecessary Reflection: Training LRMs for Efficient Reasoning with Adaptive Reflection and Length Coordinated Penalty

AI summary

Overview

Research area: Efficient reasoning for Large Reasoning Models (LRMs) — reinforcement learning approaches that shorten chain-of-thought generation without sacrificing answer accuracy. Published on arXiv (2602.12113v2, cs.AI), with affiliations at Zhejiang University and MYbank, Ant Group.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning policy optimization, reward shaping, and the <think> / </think> reasoning-trace format used by R1-style models.

Scope: The paper identifies "over-reflection" in LRMs, proposes a reinforcement learning framework (ARLCP) that penalizes reflection and length adaptively per problem, and evaluates it on five mathematical benchmarks plus MMLU and two additional model families.

What This Paper Is About

Large Reasoning Models often generate very long thinking traces containing repetitive self-questioning ("wait"), unproductive hesitation ("hmm"), and circular reasoning — tokens that increase latency and computation without improving the answer. The authors observe that this over-reflection gets worse as problem complexity increases, and that correct answers tend to be shorter with fewer reflection tokens than incorrect ones. The goal is to train models to stop unnecessary reflection while preserving the reflection that actually helps, so that responses get shorter without losing accuracy.

Key Contributions

1. Identification of the over-reflection phenomenon. The authors quantify reflection token counts across datasets of increasing difficulty and show a positive correlation between reflection tokens and problem complexity, plus a decline in accuracy as reflection token counts increase — particularly for smaller models.

2. The ARLCP framework. A reinforcement learning method with two coordinated penalties: an adaptive reflection penalty whose coefficient depends on the problem's estimated complexity, and a length penalty that covers verbosity not captured by reflection alone. Complexity is estimated online from the model's own reflection token counts (RTC), not from a fixed label.

3. Threshold-based complexity conditioning. Problems are segmented into simple, moderate, and hard via RTC thresholds n1 and n2, each mapped to a different penalty weight (λ1, λ2, λ3), with the remaining penalty budget α2 = α − α1 assigned to the length penalty.

4. Broad empirical validation. Evaluations on five mathematical reasoning benchmarks with DeepSeek-R1-Distill-Qwen-1.5B and 7B, plus out-of-distribution results on MMLU and transfer experiments on Qwen3-1.7B and DeepSeek-R1-Distill-Llama-8B.

Main Findings

• Primary efficiency–accuracy result (1.5B): ARLCP reduces average response length by 53.1% while improving accuracy by 5.8% over the vanilla DeepSeek-R1-Distill-Qwen-1.5B. In the per-benchmark table the figure is reported as ΔAcc = 5.81 and ΔLength = −53.05%.

• Primary efficiency–accuracy result (7B): A 35.0% length reduction with a 2.7% accuracy gain over vanilla DeepSeek-R1-Distill-Qwen-7B (table values: ΔAcc = 2.69, ΔLength = −34.96%).

• Gains concentrate on hard problems. For the 1.5B model, ARLCP scores 73.28% on AMC 2023, 34.17% on AIME 2024, and 26.46% on AIME 2025, which the authors attribute to the adaptive reflection penalty suppressing unnecessary reflection more aggressively on complex tasks.

• Reflection and length decline during training while accuracy rises. Tracking over training steps (Figure 4), both average reflection-word count and output length decrease while accuracy improves.

• Both penalties are necessary. Ablation on the 1.5B model: full ARLCP reaches 5.8 ΔAcc / −53.1% ΔLength; removing the length penalty gives 5.6 / −29.6%; removing the reflection penalty gives 1.4 / −58.1%; fixing both coefficients at 0.1 gives 5.5 / −47.5%.

• Fixed coefficients underperform adaptive ones. The Fixed Coefficients ablation (α1 = α2 = 0.1) is used to argue that adapting penalty strength to problem complexity matters.

• Threshold sensitivity. On the 1.5B model trained for 100 steps, thresholds (n1, n2) of (20, 40) give 3.36 Δacc / −20.2%; (60, 100) give 2.67 / −19.7%; (40, 100) give 3.36 / −20.0%; and the chosen (40, 80) gives 3.58 / −21.7%. n1 shows a rise-then-decline pattern peaking at 40, while n2 has a smaller effect, with 80 best.

• Penalty weight sensitivity. With α = 0.1: 3.48 Δacc / −9.5%; α = 0.3: 2.26 / −30.7%; α = 0.2 (chosen): 3.58 / −21.7%. Varying the λ values (0.025/0.05/0.075 gives 3.39 / −18.2%; 0.1/0.15/0.2 gives 3.08 / −18.8%) produces only minor fluctuations relative to the chosen 0.05/0.1/0.15.

• Out-of-distribution generalization (MMLU). On MMLU (14k multiple-choice questions across 57 subjects), the 7B model with ARLCP reaches 64.1 accuracy at 742 tokens, for +0.7 ΔAcc and −41.0% ΔLength, versus vanilla at 63.4 accuracy / 1257 tokens, Nothinking at 51.2 / 128, SFT at 62.8 / 1321, AdaptThink at 63.6 / 856, and TLMRE at 63.9 / 872.

• Transfer to other model families. Qwen3-1.7B with ARLCP achieves +3.00 ΔAcc and −38.19% ΔLength; DeepSeek-R1-Distill-Llama-8B achieves +1.99 ΔAcc and −34.01% ΔLength.

Methodology in Plain English

The authors start by measuring how often models produce reflection-triggering keywords such as "wait", "hmm", and "alternatively" (reflection token count, RTC) alongside total response tokens (LEN). They find RTC rises with dataset difficulty and that wrong answers carry more reflection than right ones. They treat RTC as a proxy the model itself uses to signal difficulty.

ARLCP then turns this signal into a reward. During training, for each problem the policy samples m rollouts. For rollouts that produced the correct answer, the method computes the mean and standard deviation of RTC and of LEN, and normalizes both quantities through a sigmoid. Each rollout's RTC is bucketed into simple, moderate, or hard using thresholds n1 = 40 and n2 = 80, which selects the reflection-penalty coefficient α1 from λ1 = 0.05, λ2 = 0.1, λ3 = 0.15. The length-penalty coefficient is the remainder, α2 = α − α1, with α = 0.2. The reward for a rollout is its correctness (0 or 1) multiplied by 1 − α1·f(RTC) − α2·f(LEN) — so only correct answers receive credit, and among correct answers the shorter, less-reflective ones score higher. Because autoregressive sampling is non-differentiable, the reward is optimized with policy gradients.

Training uses the DeepScaleR dataset (40K math problem–answer pairs compiled from AIME 1984–2023, AMC prior to 2023, Omni-MATH, and Still), which does not overlap the evaluation benchmarks. The implementation uses the VeRL framework with REINFORCE Leave One Out (RLOO) rather than GRPO; the authors state GRPO is sensitive in non-standard settings involving length penalties and can trigger abrupt policy collapses. All experiments ran on 8 NVIDIA A100 GPUs with batch size 128, learning rate 2e-6, and a 16K token context limit.

Why This Matters

• Research impact: The paper reframes "overthinking" not as generic verbosity but as a measurable, complexity-dependent reflection behavior, and shows that a penalty shaped by that behavior can improve accuracy and efficiency at the same time rather than trading one for the other. This contrasts with prior length-penalty RL and SFT-based compression, which the authors say tend to suppress reflection or discard the thinking process and thereby hurt accuracy.

• Real-world applications:

  • Real-time assistants and chat interfaces where inference latency from long thinking traces is directly visible to users.
  • Resource-constrained or on-premise deployments, where the token overhead of over-reflection translates into hardware and energy cost.
  • Cost-sensitive high-volume serving of reasoning models, where per-query token count is the dominant expense.
  • Question-answering over mixed domains, given that ARLCP also improved the length–accuracy trade-off on the out-of-distribution MMLU benchmark.

• Industry relevance: The work is co-authored with MYbank, Ant Group, and the measured effects (a reported 53.1% length reduction on 1.5B and 35.0% on 7B, alongside accuracy gains) map onto serving-cost metrics that operators track directly. The code is released at the GitHub repository listed in the paper, and the method is implemented on the open VeRL stack with RLOO, which lowers the barrier to reproducing the training pipeline.

Future Directions

• Scaling beyond the tested sizes. The paper evaluates 1.5B, 1.7B, 7B, and 8B models; whether the same adaptive penalty design behaves the same way on substantially larger LRMs is not reported.

• Reflection detection beyond keyword matching. RTC is computed by matching reflection-trigger keywords. Whether learned or semantically defined reflection detectors would change the trade-off is an open question.

• Threshold and hyperparameter transfer. n1 = 40, n2 = 80, and α = 0.2 were tuned on the 1.5B model. How these thresholds should be set for a new base model or a new domain is not resolved by the reported sensitivity analysis.

• Wider domain coverage. Beyond mathematics and the MMLU out-of-distribution check, the paper does not report results on other reasoning-heavy domains such as code generation or long-horizon agentic tasks.

Target Audience

Researchers and practitioners working on efficient inference and reinforcement-learning post-training for reasoning models — particularly those who need to reduce token cost in deployed systems. It is also useful for readers studying chain-of-thought behavior analysis, since the paper's diagnostic work on reflection token counts and problem complexity stands somewhat independently of the proposed training method. Readers without a background in policy-gradient methods should expect the methodology section to require additional reading.

Authors’ abstract

Large Reasoning Models (LRMs) have demonstrated remarkable performance on complex reasoning tasks by employing test-time scaling. However, they often generate over-long chains-of-thought that, driven by substantial reflections such as repetitive self-questioning and circular reasoning, lead to high token consumption, substantial computational overhead, and increased latency without improving accuracy, particularly in smaller models. Our observation reveals that increasing problem complexity induces more excessive and unnecessary reflection, which in turn reduces accuracy and increases token overhead. To address this challenge, we propose Adaptive Reflection and Length Coordinated Penalty (ARLCP), a novel reinforcement learning framework designed to dynamically balance reasoning efficiency and solution accuracy. ARLCP introduces two key innovations: (1) a reflection penalty that adaptively curtails unnecessary reflective steps while preserving essential reasoning, and (2) a length penalty calibrated to the estimated complexity of the problem. By coordinating these penalties, ARLCP encourages the model to generate more concise and effective reasoning paths. We evaluate our method on five mathematical reasoning benchmarks using DeepSeek-R1-Distill-Qwen-1.5B and DeepSeek-R1-Distill-Qwen-7B models. Experimental results show that ARLCP achieves a superior efficiency-accuracy trade-off compared to existing approaches. For the 1.5B model, it reduces the average response length by 53.1% while simultaneously improving accuracy by 5.8%. For the 7B model, it achieves a 35.0% reduction in length with a 2.7% accuracy gain. The code is released at https://github.com/ZeweiYu1/ARLCP .