Think Less, Locally: Local Preference Learning for Reasoning Compression

AI summary

Overview

Research area: Efficient inference for large reasoning models (LRMs) — specifically chain-of-thought (CoT) compression and preference optimization.

Technical level: Intermediate. The method builds on DPO and step-level reasoning segmentation, so readers should be comfortable with preference learning and LLM decoding.

Scope: The paper introduces LoCo, a method that learns where in a reasoning trace the model should stop, rather than globally rewarding shorter traces, and evaluates it on five math and three code benchmarks.

What This Paper Is About

Large reasoning models often keep deliberating after they already have enough information to answer, a behavior called "overthinking" that inflates inference cost. Existing compression methods mostly compare whole reasoning trajectories (short correct solution vs. long correct solution) or delete/shorten pieces of an existing trace, which gives a global preference for brevity but little signal about which specific reasoning state makes further thinking unnecessary. LoCo instead detects "overthinking states" in the model's own traces, freezes the reasoning prefix at that state, regenerates a shorter verified continuation from the same state, and trains the model with DPO to prefer concluding over continuing at that exact point.

Key Contributions

1. A reframing of the compression problem. The authors argue that effective reasoning compression requires state-aware decisions about when further reasoning is unnecessary, not a blanket preference for shorter trajectories.
2. The LoCo framework. A three-stage pipeline: (a) step-level overthink detection combining model confidence screening with a stronger judge model, (b) verified continuation generation from frozen reasoning states, and (c) local preference learning with DPO on state-conditioned pairs that share an identical prefix.
3. Empirical validation across domains. LoCo is trained only on self-generated mathematical traces but evaluated on five math benchmarks and three code benchmarks, improving the accuracy-efficiency trade-off on math and transferring to code.
4. Ablations isolating design choices. Comparisons against a global (trajectory-level) preference, externally generated continuations from GPT-5.4, and random/evenly spaced cutpoints show that both local preference construction and detector-selected cutpoints drive the gains.

Main Findings

• Strong compression on mathematics with maintained or improved accuracy. LoCo reduces completion length by 41.7%–45.4% across the five math benchmarks while matching or exceeding base accuracy on each. On MATH-500 it is essentially flat (91.80 vs. base 91.88) with a 44.1% token reduction.
• Gains grow on harder math benchmarks. On Olympiad, AIME24 and AIME25, LoCo attains the highest accuracy, highest AUC, and 44.6%–45.4% shorter completions among compared methods (e.g., AIME24: 54.17% accuracy vs. base 50.83%, tokens 7364 vs. 13493, AUC 46.82 vs. 40.33).
• Baseline contrast. DAST gives only mild compression and sometimes increases completion length on harder benchmarks (e.g., AIME24 tokens 14423 vs. base 13493); TokenSqueeze compresses aggressively but loses more accuracy as difficulty rises (AIME24 accuracy 48.75 vs. base 50.83).
• Cross-domain transfer from math-only training. On code, LoCo reduces tokens by 23.4%–36.3%: MBPP accuracy is unchanged at 84.05 with 36.3% fewer tokens, LiveCodeBench improves to 42.51 from 41.77 with 24.7% fewer tokens, and CodeContests improves to 20.61 from 18.48 with 23.4% fewer tokens. The abstract states LoCo achieves the best AUC across all code benchmarks.
• Local beats global preference. A trajectory-level preference variant ("Global") with the same queries and thresholds cuts tokens by only 5.2% on AMC23, 11.1% on AIME24, and 15.8% on AIME25, versus LoCo's 41.7%, 45.4%, and 45.2%.
• Self-generated continuations beat external ones. Replacing chosen continuations with GPT-5.4 regenerations gives lower compression (35.0% on AMC23, 31.8% on AIME24, 36.0% on AIME25) and lower AUC than LoCo's own-model continuations.
• Cutpoint selection matters. Random (RandCut) and evenly spaced (UniCut) cutpoints achieve similar token reductions (roughly 39.5%–43.2%) but incur larger accuracy losses, especially on AIME24 and AIME25.

Methodology in Plain English

Step 1 — Find where the model overthinks. The model is fed its own complete reasoning trace and, for each token, its top-1 probability is recorded. Tokens are grouped into steps at paragraph breaks. A step's "confidence" is the average top-1 probability over its tokens, and "volatility" measures how much confidence changed from the previous step, computed as (c_i - c_{i-1})^2 / 4. A step is flagged as a candidate when it has simultaneously low confidence (at or below the 25th percentile) and high volatility (at or above the 75th percentile), with percentiles calibrated once on the model's own traces. Because these statistics use only information up to the end of the step, the detector is causal and never sees the continuation it removes.

Step 2 — Verify with a judge. Confidence alone cannot separate redundant deliberation from genuine difficulty, so each flagged step is re-checked by Qwen3-30B, which labels it as overthink, normal, or final. Only judge-confirmed overthink steps survive. Each trace keeps at most 10 candidate states, chosen by uniform sampling.

Step 3 — Regenerate shorter correct endings. At each surviving state, the reasoning prefix is frozen and 8 independent continuations are sampled. A continuation is kept only if its answer is symbolically equivalent to the gold answer; the shortest valid one becomes the "chosen" response, and the model's original continuation becomes the "rejected" one.

Step 4 — Filter and train. Pairs are kept only if the token savings are at least 300, the prompt fits within 6k tokens, and each continuation fits within 12k tokens; at most 3 pairs are kept per question (largest savings first) to prevent heavily-flagged questions from dominating. DPO is then applied with these shared-prefix pairs, so the comparison is conditioned on the identical reasoning state. Training uses DeepSeek-R1-Distill-Qwen-7B with a LoRA adapter (rank 256, alpha 512, dropout 0.05) for two epochs, learning rate 1e-5, effective batch size 64, DPO temperature β = 0.1, on four H20 GPUs with a maximum sequence length of 14,336 tokens. Data come from the math14k pool of 14,027 questions.

Why This Matters

Impact on research. The paper pushes reasoning compression from trajectory-level heuristics toward state-level decision learning, and its ablations give concrete evidence that where you compress matters as much as whether you compress. It also shows that a compression preference learned purely on math generalizes to code, supporting the view that the method reshapes a reasoning-length preference rather than injecting task knowledge.

Real-world applications:

• Reducing decoding latency and memory cost when deploying reasoning models that generate thousands of tokens per query.
• Lowering inference spend for math tutoring, symbolic problem solving, and assessment tools built on long-CoT models.
• Speeding up code generation assistants where the model currently deliberates well past the point of a correct solution.
• Enabling longer context or higher throughput in batch pipelines constrained by token budgets.

Industry relevance. The approach uses a self-contained data pipeline (self-generated traces plus a judge model) and LoRA fine-tuning on a 7B distilled model, making it practical for teams that cannot re-train base reasoning models but want to cut serving costs without giving up accuracy.

Future Directions

• Extending local preference learning beyond math and code to other long-CoT domains such as science QA or agentic tool use, where the paper reports no evaluation.
• Testing whether the confidence-plus-volatility detector remains reliable for base models with different reasoning styles; the paper calibrates percentiles on one model's own traces and does not report detector precision or inter-annotator agreement with the judge.
• Replacing the judge-model re-checking stage with a cheaper or self-contained verification signal, since Qwen3-30B is currently required to filter candidate states.
• Investigating whether the state-conditioned preference pairs compose with reinforcement-learning objectives, which the paper does not combine with LoCo.

Target Audience

Researchers and practitioners working on efficient LLM inference, chain-of-thought compression, and preference optimization. It is most useful to readers already familiar with DPO and step-level reasoning segmentation who want a concrete recipe for building compression supervision from a model's own traces, and to engineers deciding whether to invest in data-pipeline-based compression versus simpler length-penalty baselines.

Authors’ abstract

Large reasoning models achieve strong performance on challenging tasks, but their long chain-of-thought often incurs substantial inference costs by continuing to deliberate after sufficient information has been obtained. Existing reasoning compression methods primarily operate on reasoning trajectories, paying less attention to the local decision of whether further reasoning is necessary given the current state. We propose LoCo, a local preference learning method for reasoning compression. LoCo detects candidate overthinking states from the model's own traces, freezes the corresponding reasoning prefixes, regenerates shorter verified continuations from the same states, and trains on the resulting state-conditioned preference pairs with DPO. This formulation teaches the model when to continue reasoning and when to conclude, rather than simply encouraging shorter trajectories overall. We construct training data entirely from self-generated mathematical traces and evaluate LoCo on five mathematical and three code generation benchmarks. LoCo reduces completion length by 41.7\%--45.4\% on mathematics and 23.4\%--36.3\% on code, while maintaining or improving accuracy and achieving the best AUC across all code benchmarks. LoCo consistently improves the accuracy-efficiency trade-off on mathematics and transfers effectively to code generation despite math-only training, demonstrating the cross-domain generality of local preference learning for reasoning compression. The code and model will be publicly released at https://github.com/MuyaoYuan/LoCo.