HE-SNR: Uncovering Latent Logic via Entropy for Guiding Mid-Training on SWE-bench

AI summary

Overview

Research area: Machine learning — large language model training and evaluation, specifically mid-training metrics for agentic software engineering tasks.

Technical level: Advanced. The paper assumes familiarity with token-level probability distributions, perplexity, entropy, Top-k sampling, RoPE scaling, YaRN, MoE architectures, and the SFT/RL post-training pipeline.

Scope: The paper proposes a mid-training evaluation metric (HE-SNR) grounded in a fine-grained analysis of Top-10 entropy distributions over filtered "Action" tokens from SWE-bench trajectories, validated on models up to 560B parameters across 32K and 128K context windows.

What This Paper Is About

Because reinforcement learning on base models is difficult for complex multi-turn tasks, frameworks for SWE-bench build on instruction-tuned models, making a Supervised Fine-Tuning (SFT) phase a prerequisite for evaluating performance. But assessing mid-training progress currently depends on post-SFT scores — a lagging indicator that is expensive and obscured by SFT alignment stochasticity. The paper's goal is a metric that can directly predict a base model's latent software engineering potential from mid-training checkpoints, without waiting for SFT, and that resists the distortion the authors call the "Long-Context Tax."

Key Contributions

1. Data-Efficient Evaluation Protocol. A token-granular filtering strategy that achieves high correlation with SWE-bench scores using only 500 trajectories totaling approximately 12.5M tokens.

2. Theoretical Insight: "Shift to ln 3" and Entropy Compression Theory. The authors propose the Entropy Compression Hypothesis, identifying a distributional shift of high-entropy tokens toward ln 3 as a signature of superior reasoning, and reframing intelligence as "reasonable hesitation" rather than scalar Top-1 compression.

3. Novel Metric: High-Entropy Signal-to-Noise Ratio (HE-SNR). Validated across models with up to 560B parameters and a 10x scaling span, HE-SNR circumvents the Long-Context Tax and shows a strict linear relationship with downstream capabilities.

4. Empirical Insight into the SFT Alignment Tax. SFT improves global PPL but significantly degrades performance on critical high-entropy tokens, suggesting SFT prioritizes superficial pattern matching at the expense of complex reasoning.

Main Findings

• PPL tracks replication, not potential. On the LLM-generated Thought and Action components of the curated SWE-bench test dataset, PPL shows a strong linear correlation with Top-1 accuracy that visibly diminishes as k increases. Since Top-1 accuracy consistently exceeds 90%, PPL measures replication precision rather than latent potential.

• The Long-Context Tax is scale-dependent. Extending context via base frequency scaling causes a transient regression. It is negligible for MoE-A3B but pronounced in the larger MoE-A26B: at Step 200 of the 128K phase, PPL and Top-1 accuracy degrade significantly while Top-10 accuracy remains stable alongside improved SWE-bench performance. The tax erodes confidence in previously certain predictions rather than competence.

• Three entropy distribution patterns. Top-1 tokens concentrate near 0. Non-Top-1 tokens peak at ln 2 and ln 3, with ln 2 particularly prominent and consistent across scales. Non-Top-2 tokens show a broad distribution centered around ln 4 in weaker models, but superior models display a distinct peak at ln 3.

• The "Shift to ln 3" generalizes. Qwen2.5-72B-Base (Dense) and DeepSeek-V3-Base (MoE) on SWE-bench, plus 5,000 rigorous Math QA samples across four base models, all exhibit multi-modal entropy peaks at ln 1, ln 2, and ln 3 along with the shift trend.

• SFT acts as a potent entropy regularizer. Post-SFT distributions collapse uncertainty, reinforce the shift toward the ln 3 boundary and below, and boost Top-1 density. In MoE-A26B, the Top-1 accuracy gap at Step 200 vanishes post-SFT, while the ln 3 peak at Step 200 exhibits remarkable robustness to the Long-Context Tax.

• Observation tokens behave differently. The MoE-A26B entropy distribution on Observation tokens lacks the ln 3 peak, showing instead a prominent peak at ln 10, attributed to stochastic numerical sequences such as line numbers and random IDs. This justifies excluding Observations from metric computation.

• HE-SNR outperforms PPL and HE-PPL. HE-SNR (Filtered) demonstrates superior linearity and robustness compared to PPL baselines, mitigating the Long-Context Tax while maintaining strict rank consistency and linearity across the vast majority of checkpoints.

• Action tokens dominate Thinking tokens. In the ablation study, Pearson r rises from 0.5581 to 0.9666 and Kendall τ from 0.5192 to 0.9440 when shifting evaluation from Thinking to Action tokens. The full pipeline (XML tag removal, whitespace and symbol removal, AST-based comment removal) pushes Kendall τ to its peak of 0.9794, with Pearson r at 0.9649.

• The threshold is robust. Across a threshold range of ln t where t ∈ [2,5], the chosen ε = (ln 3 + ln 4)/2 sits close to the empirical optimum for both metrics, with a robust plateau rather than a single magic number.

• SNR decouples quantity from mastery. During 128K training of MoE-A26B, |ℋ| spikes at Step 200, confirming the Long-Context Tax inflates the volume of uncertain tokens and degrades HE-PPL. HE-SNR instead maintains a consistent upward trend aligned with downstream performance.

• The Alignment Tax appears in high-entropy decisions. Global PPL improves post-SFT, but both HE-PPL and HE-SNR degrade within the High-Entropy Decision Set, suggesting SFT sacrifices prediction accuracy on critical high-entropy tokens for stylistic patterns.

Methodology in Plain English

The researchers start from a curated test set rather than a training corpus. They take 500 successful trajectories from SWE-bench-Verified, generated with the same synthesis methodology as the task-aligned SFT data to ensure distributional alignment, and restricted to a 32K token limit so they can be evaluated against models with 32K or 128K context windows.

A multi-turn SWE trajectory is formalized as τ = (o₁, r₁, a₁, …, r_T, a_T): an observation, then chain-of-thought reasoning and an action repeated per turn, with actions encapsulated in an XML protocol. Metrics are computed only on the Action components — Observations are masked as input context and Thoughts are excluded because they are dominated by stylistic artifacts such as filler words like "Now" or "Let." The rationale is that functional execution (what the model does) matters more than narrative style (what it says).

Actions then go through progressively stricter filtering: regular expressions strip XML tags and markdown, AST parsing removes code comments as natural-language semantics distinct from executable logic, and a noise-reduction pass removes redundant whitespace, newlines, isolated symbols, and decorative artifacts. Because filtering happens at the character level, the authors map character-level tags to token granularity via offset alignment.

The core measurement is Top-k entropy, computed only over the model's top-10 re-normalized probabilities rather than the full vocabulary. The authors justify k = 10 as a balance between computational cost and coverage, noting it encompasses over 99.6% of target tokens in the test corpus while avoiding the cost and noise of full-vocabulary entropy. By Lemma 4.1, entropy over a top-k distribution is bounded by ln k with equality only under uniformity, which explains the empirical peaks at ln 2 ≈ 0.69 and ln 3 ≈ 1.10 and maps entropy values directly onto the effective number of competing candidates.

From this they define an Entropy-Compressed State of order k: probability mass distributed approximately uniformly over the top-k candidates, manifesting as a peak near ln k. The conventional high-confidence state near 0 is the trivial collapse to order k = 1. The metric itself, HE-SNR, averages the ratio of the ground-truth token probability to its Top-10 entropy over a High-Entropy Decision Set ℋ = {t | H_top10(x_t) > ε and x_t ∈ C₁₀(x_t)}. The membership condition x_t ∈ C₁₀(x_t) restricts evaluation to instances where the ground truth is plausible, filtering out tokens with severe distributional divergence that often stems from stylistic mismatch rather than semantic error. The threshold ε = (ln 3 + ln 4)/2 sits at the boundary distinguishing 3-candidate uncertainty from that of 4 or more, targeting the recalcitrant uncertainty that resists SFT regularization.

Evaluation uses multiple mid-training checkpoints from MoE-A3B (68B total, 3B active, linear RoPE scaling) and MoE-A26B (560B total, 26B active, YaRN) across 32K and 128K phases. Each checkpoint undergoes SFT on over 10,000 SWE-related trajectories for 3 epochs, with SWE-bench-Verified Pass@1 averaged over three independent runs. Two baselines are compared alongside the target metric: standard PPL over all tokens, and HE-PPL defined as PPL computed exclusively over the high-entropy token set ℋ.

Why This Matters

Impact on research. The paper reframes the Compression-Intelligence Hypothesis from scalar information minimization to a distributional view, arguing that true intelligence involves retaining a structured distribution over valid reasoning paths rather than minimizing loss alone. It also offers a mechanism-level account of the Alignment Tax located in high-entropy decision points, which gives alignment researchers a specific target for mitigation strategies beyond general statements that SFT trades capability for format.

Real-world applications:

• Training compute budgeting. By identifying sub-optimal mid-training candidates early, teams can abort unproductive runs rather than paying for SFT and full SWE-bench evaluation each iteration.
• Coding assistant development. The metric targets agentic repository-level issue resolution — tool invocation, multi-turn environment interaction, and patch generation — which is the setting real coding assistants operate in.
• Context-window extension decisions. Practitioners extending context via linear RoPE scaling or YaRN can use HE-SNR to distinguish genuine capability growth from the transient entropy spike of the Long-Context Tax, which raw PPL conflates.
• Alignment pipeline design. Understanding which token subsets degrade post-SFT could inform how SFT data and objectives are constructed for software engineering agents.

Industry relevance. The work is directly tied to industrial deployment: the authors are employed by or interning at Meituan, which leads development of the LongCat-Flash-Lite and LongCat-Flash models evaluated; the conflict-of-interest disclosure states this explicitly. The models span 68B and 560B total parameters, reflecting production-scale rather than academic-scale training.

Future Directions

1. Replace the static threshold. HE-SNR currently relies on a fixed ε and data-specific patterns; the authors aim to develop adaptive thresholding mechanisms that capture varying convergence rates.

2. Separate logic from style. Because SFT is sensitive to code style, future work will explore code canonicalization or style transfer so that high-entropy tokens reflect logical uncertainty rather than stylistic divergence.

3. Test broader architectures and domains. The authors plan to extend validation to diverse architectures and broader logical domains such as mathematics.

4. Search for higher-order states. A specific open question is whether Entropy-Compressed States of order k ≥ 4 emerge in tasks with greater intrinsic complexity.

Target Audience

This paper is most useful to ML researchers and engineers working on LLM training pipelines for agentic and software engineering tasks — particularly those who must decide which mid-training checkpoints are worth carrying forward into SFT. It also suits alignment researchers interested in the mechanistic sources of the Alignment Tax, and evaluation researchers looking for alternatives to perplexity in long-context settings. Readers need a working understanding of token-level probability distributions, entropy, and the SFT/RL post-training stack to follow the derivations; the empirical results are readable without that background, but the metric semantics are not.

Authors’ abstract

SWE-bench has emerged as the premier benchmark for evaluating Large Language Models on complex software engineering tasks. While these capabilities are fundamentally acquired during the mid-training phase and subsequently elicited during Supervised Fine-Tuning (SFT), there remains a critical deficit in metrics capable of guiding mid-training effectively. Standard metrics such as Perplexity (PPL) are compromised by the "Long-Context Tax" and exhibit weak correlation with downstream SWE performance. In this paper, we bridge this gap by first introducing a rigorous data filtering strategy. Crucially, we propose the Entropy Compression Hypothesis, redefining intelligence not by scalar Top-1 compression, but by the capacity to structure uncertainty into Entropy-Compressed States of low orders ("reasonable hesitation"). Grounded in this fine-grained entropy analysis, we formulate a novel metric, HE-SNR (High-Entropy Signal-to-Noise Ratio). We validate our approach on models with up to 560B parameters across different context windows (32K/128K). This work provides both the theoretical foundation and practical tools for optimizing the latent potential of LLMs in complex engineering domains.