DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains

AI summary

Overview

Research area: Large Reasoning Models (LRMs), reinforcement learning for chain-of-thought reasoning, and inference-time efficiency.

Technical level: Intermediate to Advanced. The paper assumes familiarity with reinforcement learning for LLMs (GRPO, DAPO, policy entropy, reward shaping) and with LLM reasoning benchmarks.

Scope: The paper proposes DeepCompress, a reinforcement-learning framework that dynamically classifies math problems as "Simple" or "Hard" during training and applies opposing length rewards to each class, aiming to improve accuracy and token efficiency simultaneously.

What This Paper Is About

Large Reasoning Models produce long chains of thought but waste effort — "overthinking" easy problems and "underthinking" hard ones. Prior work that shortens reasoning chains with supervised fine-tuning or length-penalized reinforcement learning gains efficiency but usually loses accuracy. This paper argues that always favoring shorter responses is the wrong objective, showing that longer responses cover a wider range of correct solutions for difficult problems, and proposes an adaptive reward that compresses reasoning on well-mastered problems while extending it on hard ones.

Key Contributions

1. Dual length reward. A reward mechanism that pushes toward shorter responses for "Simple" questions and toward longer, more exploratory responses for "Hard" questions, modulating the standard outcome reward.
2. Model-aware difficulty classification. A real-time, annotation-free way to label question difficulty: a question is "Simple" when its group pass ratio exceeds the batch pass ratio, and "Hard" otherwise, so difficulty tracks the model's evolving capability across training.
3. Robustness enhancements. A correctness-conditioned length reward (length terms apply only to responses with correct final answers) to prevent reward hacking, and an exponential moving average (EMA) smoothing of the batch pass ratio for training stability.
4. Empirical demonstration and analysis. State-of-the-art results on seven mathematical benchmarks plus GPQA-Diamond, MMLU-STEM, and Big-Bench Hard, together with analysis of policy entropy, response length, and "aha moment" reflection behaviors.

Main Findings

• Accuracy gains on math benchmarks: In Table 1, DeepCompress-Zero-3B reaches an average accuracy of 36.6 versus 34.6 for DeepMath-Zero-3B, and DeepCompress-Zero-7B reaches 48.7 versus 46.0 for DeepMath-Zero-7B — described as average absolute improvements of +2.0 points (3B) and +2.7 points (7B).
• Large gains on the hardest benchmarks: DeepCompress-Zero-7B improves over DeepMath-Zero-7B by +4.1 absolute points on AIME 24 (23.5 vs 19.4) and by +6.5 on AIME 25 (19.6 vs 13.1).
• Token compression: Average response length falls by 57.9% for the 3B model and 16.6% for the 7B model relative to DeepMath-Zero models. On AIME 24, DeepCompress-Zero-3B uses 37.6% fewer tokens for a +5.2 improvement, and DeepCompress-Zero-7B uses 35.2% fewer tokens for a +4.1 improvement.
• Generalization beyond mathematics: On GPQA-Diamond (biology, physics, chemistry), MMLU-STEM, and Big-Bench Hard, DeepCompress models improve over baselines — for example, DeepCompress-Zero-7B scores 43.9 on GPQA-Diamond overall, 75.5 on MMLU-STEM, and 85.7 on Big-Bench Hard, versus 42.6, 72.7, and 85.0 for DeepMath-Zero-7B (GPQA biology dips slightly, 57.6 vs 58.6).
• Longer responses can be better: The preliminary study found that for pass@1, shorter responses perform better, but for pass@k, longer responses catch up and surpass shorter ones, except on DeepMath-Zero-7B at AIME25, where the conclusion held with a larger k (e.g., k=64).
• Entropy and adaptive behavior: Models trained with a length bonus show higher policy entropy; length penalty variants maintain stable, low entropy. DeepCompress's entropy and response length both rise initially (exploration) and then stabilize/reduce (efficiency), while test performance keeps growing.
• Smarter, not just more, thinking: On a hard problem set where Qwen2.5-3B and Qwen2.5-7B failed, DeepCompress reflects more often than the baseline yet produces shorter responses — DeepCompress-Zero-7B: 2.64 reflections, 5,942 tokens, 13.81 pass@1; DeepMath-Zero-7B: 2.59 reflections, 7,180 tokens, 11.35 pass@1.
• Hyperparameter sensitivity: Performance is robust to the reward weight α, with all configurations beating the DeepMath-Zero-3B baseline; a larger EMA smoothing factor (λ = 0.99) is reported as crucial for optimal performance.

Methodology in Plain English

The authors start from Zero RL training with the DAPO algorithm, where a rule-based verifier gives +1 for a correct final answer and −1 otherwise. On top of that binary reward, they add a length reward.

For each question, the model samples a group of responses (group size 32). The authors compute each response's standardized length by subtracting the group mean response length and dividing by the group standard deviation. That standardized length is passed through a sigmoid function whose sign is controlled by a parameter β: positive β rewards shorter responses, negative β rewards longer ones. A weight α scales the magnitude of this length reward.

The key idea is how β is set. For every question, the authors compute the group pass ratio (fraction of the group's responses that are correct) and the batch pass ratio (average group pass ratio over the batch). β is simply the difference between the two. If a question's group pass ratio is above the batch average, it is treated as "Simple" and β is positive, favoring brevity. If it is below average, it is treated as "Hard" and β is negative, favoring longer exploration. Because the batch pass ratio reflects the model's current overall ability, the difficulty label changes as the model improves — no human difficulty annotations are needed.

Two refinements improve robustness. First, the length reward is added only to responses whose final answer is correct, so the model cannot gain reward by shortening wrong answers. Second, the batch pass ratio is smoothed with an exponential moving average (initialized at 1.0) so that early-training instability does not misclassify questions as simple and prematurely cut reasoning short.

Training used Qwen2.5-3B and Qwen2.5-7B with verifiable-reward RL for 600 steps, a learning rate of 1e-6, a training batch size of 512, a PPO mini-batch size of 32, clip ratios of 0.20/0.28, rollout n of 32, α = 0.2, and λ = 0.99. A validation set of 60 MATH questions and 60 AIME 2022–2023 questions selected the best checkpoint. Evaluation sampled 16 responses per question with temperature 0.6, top-p 0.95, and max_tokens 32,768, and all baselines were re-evaluated under the same settings.

Why This Matters

Impact on research: The paper challenges the dominant "shorter is better" assumption in reasoning-efficiency research, providing evidence that pass@k — the signal most RL algorithms actually optimize — can improve with longer responses on hard problems. This reframes length control as an adaptive resource-allocation problem rather than a compression problem, and the model-aware difficulty signal offers a reusable, annotation-free alternative to curated difficulty labels.

Real-world applications:

• Deploying reasoning assistants where inference cost scales with generated tokens, so shorter answers on routine queries directly reduce serving cost.
• Tutoring and math-education systems that must handle both routine and competition-level problems without a fixed reasoning budget.
• Scientific and engineering question answering (the GPQA-Diamond, MMLU-STEM, and Big-Bench Hard results suggest transfer to non-math domains).
• Agentic or multi-step pipelines where saved tokens on easy steps can be reallocated to hard steps.

Industry relevance: Token efficiency translates directly into serving economics for reasoning models. A method that improves both accuracy and average output length is attractive to anyone deploying LRMs at scale, and the approach is built on existing open training recipes (verl framework, DAPO, Qwen models) with code and models to be released at the linked GitHub repository.

Future Directions

• Dependence on length variation: The paper states as a limitation that the method relies on sufficient length variation among responses sampled within the RL group; what happens when the model produces nearly uniform lengths is not resolved.
• The 10k token cap: To keep training efficient, maximum generation length was capped at 10k tokens, which the authors say may have restricted exploration of more complex or longer-form solutions. Re-running with a higher cap is a natural next step.
• Beyond the single β formulation: Whether the difficulty signal should use other statistics than group-versus-batch pass ratio differences, or vary α and β per question rather than using the reported global settings, is left open.
• Broader domain validation: The generalization evidence covers GPQA-Diamond, MMLU-STEM, and Big-Bench Hard; extending the dual-reward idea to non-mathematical, non-scientific reasoning tasks (and to code or open-ended generation) is untested here.

Target Audience

Researchers and engineers working on reinforcement learning for large reasoning models, chain-of-thought efficiency, and inference cost reduction. It is most useful to readers already comfortable with GRPO/DAPO-style training and pass@k evaluation, and to practitioners deciding whether to apply length penalties in production reasoning systems.

Authors’ abstract

Large Reasoning Models (LRMs) have demonstrated impressive capabilities but suffer from cognitive inefficiencies like "overthinking" simple problems and "underthinking" complex ones. While existing methods that use supervised fine-tuning (SFT) or reinforcement learning (RL) with token-length rewards can improve efficiency, they often do so at the cost of accuracy. This paper introduces DeepCompress, a novel framework that simultaneously enhances both the accuracy and efficiency of LRMs. We challenge the prevailing approach of consistently favoring shorter reasoning paths, showing that longer responses can contain a broader range of correct solutions for difficult problems. DeepCompress employs an adaptive length reward mechanism that dynamically classifies problems as "Simple" or "Hard" in real-time based on the model's evolving capability. It encourages shorter, more efficient reasoning for "Simple" problems while promoting longer, more exploratory thought chains for "Hard" problems. This dual-reward strategy enables the model to autonomously adjust its Chain-of-Thought (CoT) length, compressing reasoning for well-mastered problems and extending it for those it finds challenging. Experimental results on challenging mathematical benchmarks show that DeepCompress consistently outperforms baseline methods, achieving superior accuracy while significantly improving token efficiency.