Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement

AI summary

Overview

• Research area: Natural Language Processing — reinforcement learning (RL) post-training of large reasoning models (LRMs), with a focus on the internal quality of generated reasoning text.
• Technical level: Advanced (builds directly on GRPO-style RL optimization, advantage normalization, and LLM-as-a-judge evaluation).
• Scope: This paper introduces a "self-rewriting" framework that lets an LRM rewrite its own reasoning passages during RL training, using selective rewriting and a batched implementation to improve reasoning quality and the accuracy–length tradeoff with roughly 10% extra computation.

What This Paper Is About

Current large reasoning models are trained with RL using only outcome correctness rewards, which supervise whether the final answer is right but say nothing about the quality of the reasoning that produced it. As a result, reasoning texts contain four recurring flaws the authors name over-thinking, under-thinking, redundant-thinking, and disordered-thinking. The paper's goal is to give the model a way to improve its own internal reasoning process by having it rewrite its own reasoning during RL training and then learn from those rewrites.

Key Contributions

1. Self-rewriting integrated into RL post-training. The authors incorporate reasoning rewriting into the LRM RL post-tuning framework, letting models learn from their own rewritten reasoning while preserving the flexibility and scalability of the original GRPO algorithm.
2. Selective rewriting. Only "simple" samples — queries for which the first half group of rollouts are all correct — are rewritten; all other queries keep the standard GRPO treatment, so original reward signals are preserved and non-rewritten samples receive no special credit.
3. Efficient batched implementation. Rewriting and vanilla generation are compiled into a single batch and the continuation step generates only the text after the </think> token, adding about 10% time overhead compared with vanilla GRPO.
4. Fine-grained analysis of rewritten text. The paper analyzes length-ratio distributions and per-flaw LLM judge scores to show that rewriting produces more diverse length candidates and mitigates the four named reasoning flaws.

Main Findings

• Accuracy–length tradeoff improves. On Qwen3-8B, self-rewriting reaches an average accuracy of 78.8 (+0.6) with -46% reasoning length across the four evaluation tasks, versus 78.2 (+0.0) / 4243 (+0%) for the original model and 78.2 (+0.0) / 3887 (-8%) for GRPO.
• Internal reasoning quality improves substantially. Qwen3-8B self-rewriting scores 79.3 (+7.2) on the LLM judge metric averaged over the four tasks, the largest gain among all methods compared.
• Gains hold across model sizes. Qwen3-4B: 76.5 (+0.4) accuracy, 2217 (-45%) length, 77.4 (+10.3) judge. Qwen3-1.7B: 66.2 (-0.7) accuracy, 2835 (-35%) length, 73.5 (+8.1) judge.
• Length reduction happens without being asked. The rewriting prompt contains no instruction to shorten the text, yet the method shortens reasoning, and the average judge scores are higher than those of methods explicitly optimizing for length.
• Online RL outperforms offline RL. Online methods (LenPen1, LenPen2, ShorterBetter, self-rewriting) generally beat offline methods (LPO, TOPS) on final accuracy and judge scores, and they shorten reasoning more proportionally across out-of-domain tasks — offline methods over-truncated GPQA reasoning and under-truncated simpler ARC tasks.
• Rewriting ratios differ structurally from length preference. Analyzing the first 20 steps (5K samples) on Qwen3-8B, the median preferred/rejected length ratio is about 0.5 for self-rewriting versus about 0.7 for vanilla length-control sampling. Self-rewriting shows greater variance, roughly 10% of rewritten samples become longer than the originals, and the distribution is bimodal rather than unimodal.
• Per-flaw judge scores rise. On 1K queries sampled from DeepMath, Qwen3-8B scores move from Original to Rewritten as: over-thinking 76.3 → 84.0, under-thinking 82.0 → 87.3, redundant-thinking 67.9 → 82.1, disordered-thinking 61.6 → 77.1. On 1K MMLU queries: 64.4 → 78.8, 67.8 → 74.0, 60.6 → 80.8, 56.2 → 83.4. Redundant- and disordered-thinking improve most; over- and under-thinking improve more modestly.
• Selective rewriting matters. On Qwen3-8B, replacing selective rewriting with randomly rewriting 50% of samples gives 77.9 accuracy / 2691 length / 78.8 judge, and rewriting 100% gives 77.8 / 2120 / 78.6 — both below the selective version's 78.8 / 2293 / 79.3.
• Self-rewriting keeps improving under longer training. Continuing online RL methods to 20K samples, self-rewriting consistently gives higher accuracy than the other online RL baselines at every length budget tested (5K, 10K, 15K, 20K).
• Direct rewriting scores higher than the trained framework, but the framework keeps accuracy. The paper notes that direct rewriting yields larger judge-score improvements than fine-tuning with self-rewriting, which it attributes to the correctness reward preventing overfitting to rewritten outputs.

Methodology in Plain English

The method starts from GRPO, an RL algorithm that samples a group of answers per query and reinforces the ones that are correct. The authors split each rollout group in half. The model first generates the first half normally and a verifier checks whether all of those answers are correct.

• If all first-half answers are correct, the query is considered "simple." The model then rewrites its own reasoning passage using a neutral editing prompt that asks for a more organized, coherent, accurate version while preserving the core ideas, and then continues generation from that rewrite to produce a final answer.
• If any answer is wrong, the second half is generated normally, exactly as in GRPO.

Rewards are then assigned accordingly: when the group is not all-correct, correctness rewards apply as usual; when the group is all-correct, the rewritten response gets a reward of 1 and the others get 0. Because GRPO normalizes rewards within a group, giving a constant reward to non-rewritten correct responses would be equivalent, so the authors assign 0 for simplicity. They further divide all advantages in a rewritten batch by 5 — mathematically equivalent to using a learning rate of 1/5 for those samples — to limit the policy's preference for rewrites over correct non-rewrites.

For efficiency, vanilla generation samples (from non-all-correct queries) and rewriting samples (from all-correct queries) are compiled into one batch for joint inference, and the continuation step only generates the short tail after the </think> token. This keeps the overhead at about 10% over vanilla GRPO.

Evaluation uses MATH-500 (math), GPQA-Diamond (science), ARC-Challenge (logic), and MMLU-Pro (knowledge), with accuracy and token length measured as pass@1 averaged over 4 sampled runs at temperature 0.6 and max length 32K. Internal reasoning quality is scored by stronger LLMs (DeepSeek-V3, with others in the appendix) on a 1-to-5 scale per flaw, rescaled to 100. Training uses 10K samples randomly drawn from DeepMath-103K, 1 epoch, batch size 256, GRPO learning rate 3e-6 and rollout size 8; offline baselines use SimPO with learning rate 1e-6, beta = 2.0, gamma = 0.3. Models tested are Qwen3-1.7B, Qwen3-4B, and Qwen3-8B.

Why This Matters

• Research impact: The paper shifts reasoning-improvement research beyond the single axis of length control toward the internal quality of the reasoning process itself, and extends self-rewarding work from a discriminative (scoring) form to a generative (rewriting) form of self-supervision.
• Real-world applications:
  • Deploying reasoning models that answer correctly with far fewer generated tokens, reducing inference cost and latency.
  • Producing more readable and interpretable reasoning traces for human oversight and auditing of model behavior.
  • Improving out-of-domain generalization of reasoning models, since the online method adjusts reasoning length appropriately across tasks rather than over- or under-truncating.
  • Providing a general recipe that can be adapted with targeted rewriting prompts for application-specific reasoning styles.
• Industry relevance: The roughly 10% training overhead and the fact that it fits inside the standard GRPO pipeline make the method practical for post-training production reasoning models, particularly at companies already running RL-based post-training. The work was carried out with ByteDance China involvement and supported by the National Natural Science Foundation of China (Grant No. U21B2009).

Future Directions

• The authors leave exploration of specialized targeted rewriting to future research; the current work deliberately uses only general rewriting instructions and does not tell the model the specific evaluation criteria or to be concise.
• Understanding why self-rewriting's judge-score gains are smaller than direct rewriting's while preserving accuracy — the paper attributes this to the correctness reward preventing overfitting, which could be probed further.
• Extending the framework to larger model scales and training budgets beyond the 20K-sample checkpoints analyzed here.
• Investigating how rewriting prompts could be conditioned on application requirements (for example, style, audience, or domain-specific reasoning conventions) without destabilizing the RL objective.

Target Audience

This paper is most useful for researchers and engineers working on RL post-training of large reasoning models, particularly those interested in reward design, reasoning efficiency, and reasoning-trace quality evaluation. It also suits readers studying self-rewarding and self-supervised improvement methods, and practitioners who need to reduce inference token cost while preserving or improving accuracy. Some familiarity with GRPO and LLM-as-a-judge evaluation is helpful.

Authors’ abstract

Through reinforcement learning (RL) with outcome correctness rewards, large reasoning models (LRMs) with scaled inference computation have demonstrated substantial success on complex reasoning tasks. However, the one-sided reward, focused solely on final correctness, limits its ability to provide detailed supervision over internal reasoning process. This deficiency leads to suboptimal internal reasoning quality, manifesting as issues like over-thinking, under-thinking, redundant-thinking, and disordered-thinking. Inspired by the recent progress in LRM self-rewarding, we introduce self-rewriting framework, where a model rewrites its own reasoning texts, and subsequently learns from the rewritten reasoning to improve the internal thought process quality. For algorithm design, we propose a selective rewriting approach wherein only "simple" samples, defined by the model's consistent correctness, are rewritten, thereby preserving all original reward signals of GRPO. For practical implementation, we compile rewriting and vanilla generation within one single batch, maintaining the scalability of the RL algorithm and introducing only ~10% overhead. Extensive experiments on diverse tasks with different model sizes validate the effectiveness of self-rewriting. In terms of the accuracy-length tradeoff, the self-rewriting approach achieves improved accuracy (+0.6) with substantially shorter reasoning (-46%) even without explicit instructions in rewriting prompts to reduce reasoning length, outperforming existing strong baselines. In terms of internal reasoning quality, self-rewriting achieves significantly higher scores (+7.2) under the LLM-as-a-judge metric, successfully mitigating internal reasoning flaws.