Probability-Entropy Calibration: An Elastic Indicator for Adaptive Fine-tuning

AI summary

Overview

Research area: Machine learning — supervised fine-tuning (SFT) of large language models, specifically token-level loss reweighting for mathematical reasoning and reasoning transfer.

Technical level: Advanced. The paper builds on sequential guessing theory, rank statistics, information-theoretic entropy bounds, and the Cauchy Mean Value Theorem, then translates them into a practical training loss.

Scope in one sentence: The paper proposes RankTuner, a token-reweighting method that combines ground-truth token probability and token entropy through a rank-based "Relative Rank Indicator," and evaluates it against probability-only and entropy-only reweighting baselines on mathematical reasoning, out-of-distribution reasoning, and a controlled noise-injection diagnostic.

What This Paper Is About

Supervised fine-tuning normally treats every token in a target response as equally important, but some tokens are noisy, ambiguous, or easily replaceable, while others are genuinely critical. Existing fixes pick one signal: either the ground-truth probability (how well the model matches the label) or token entropy (how uncertain the model is). The authors argue that using either signal alone misidentifies which tokens deserve emphasis, and they build a single signal — the Relative Rank Indicator — that calibrates probability against entropy so that updates concentrate on truly under-learned tokens.

Key Contributions

1. A joint analysis of the two reweighting paradigms. The paper formally characterizes Prob-Dominant weighting (a function of the ground-truth probability $p_t$) and Entropy-Dominant weighting (a monotone increasing function of token entropy $H_t$), and argues that treating them in isolation over-emphasizes noisy tokens and intrinsically replaceable tokens.

2. The Relative Rank Indicator and Relative Scale. The authors introduce a rank-based signal, $\mathcal{I}_t = g(f(R_t) - f(\mathbb{E}[R_t]))$, comparing the ground-truth token's rank $R_t$ with its expected rank $\mathbb{E}[R_t]$ under the model's own prediction distribution, and use its inverse $\mathcal{S}_t = \mathcal{I}_t^{-1}$ as a per-token multiplier on the fine-tuning objective.

3. A theoretical bridge between probability, entropy, and rank. Propositions 4.4 and 4.5 establish that $R_t \le 1/p_t$ and that $\mathbb{E}[R_t]$ is lower bounded by a function of $H_t$ ($\tfrac{1}{4}2^{H_t}+1$ when $H_t \ge 2$; $2-p_{\max,t}$ when $H_t < 2$), and the Cauchy Mean Value Theorem is used to derive a power-law form linking $\mathcal{I}_t$ to a "relative competence" ratio.

4. Empirical validation across backbones and benchmarks. RankTuner is tested on Qwen2.5-Math-7B and Qwen3-8B against standard SFT, EAFT, OverTone, DFT, and TALR, with ablations separating the probability-aware and entropy-aware components.

Main Findings

• Noise sensitivity is substantially lower. In the controlled noise-insertion diagnostic (Table 1), RankTuner reports token noise precision@10% of 2.16%, token recall@10% of 26.39%, and sequence hit@10% of 9%, versus Entropy-dominant (4.54%, 4.54%, 77%) and Prob-dominant (3.25%, 39.65%, 77%). Lower is better in this table.

• Consistent mathematical reasoning gains over the original models. On Qwen2.5-Math-7B, RankTuner improves MATH-OAI Pass@1 by ↑36.81 (31.79 to 68.60) and Pass@16 by ↑1.00 (87.80 to 88.80); Minerva Math Pass@1 by ↑25.67 (7.63 to 33.30) and Pass@16 by ↑17.28 (42.28 to 59.56); OlympiadBench Pass@1 by ↑23.40 (9.49 to 32.89) and Pass@16 by ↑14.22 (47.85 to 62.07); AIME24 Pass@1 by ↑0.83 (6.25 to 7.08) and Pass@16 by ↑0.00 (23.33 to 23.33); and AMC23 Pass@1 by ↑24.06 (20.47 to 44.53), with Pass@16 ↓2.50 (85.00 to 82.50).

• Gains also hold on a second backbone. On Qwen3-8B, RankTuner improves MATH-OAI Pass@1 by ↑7.24 (65.14 to 72.38) and Pass@16 by ↑2.80 (87.40 to 90.20); Minerva Math Pass@1 by ↑6.87 (31.39 to 38.26) and Pass@16 by ↑16.91 (48.53 to 65.44); OlympiadBench Pass@1 by ↑9.06 (27.19 to 36.25) and Pass@16 by ↑12.89 (51.11 to 64.00); AIME24 Pass@1 by ↑4.17 (6.04 to 10.21) and Pass@16 by ↑0.00 (26.67 to 26.67); and AMC23 Pass@1 by ↑10.94 (35.62 to 46.56) and Pass@16 by ↑10.00 (75.00 to 85.00).

• The comparison against the strongest baseline is mixed, not uniformly positive. The paper's own $\Delta_{\mathrm{Best}}$ rows show RankTuner below the best non-RankTuner baseline on Qwen2.5-Math-7B for MATH-OAI Pass@1 (↓0.55), Minerva Math Pass@1 (↓2.38), and Minerva Math Pass@16 (↓0.73), and on Qwen3-8B for Minerva Math Pass@1 (↓2.20). The authors state RankTuner achieves best or near-best performance across the majority of benchmark–metric pairs.

• AIME24 behavior is framed as robustness. On AIME24, a harder benchmark where the paper notes several baselines show substantial degradation, RankTuner keeps the original Pass@16 while still improving Pass@1.

• Out-of-distribution transfer improves. With temperature 0.8, 16 candidate responses per query, and a 3072-token budget on Qwen2.5-Math-7B, RankTuner reaches 53.58 Pass@1 on ARC-C (Original 13.46, SFT 42.30, DFT 26.50, EAFT 48.57, TALR 52.54) and 29.64 on GPQA (Original 7.86, SFT 25.00, DFT 27.90, EAFT 25.63, TALR 29.29).

• Ablations support complementarity of the two components. Figure 4 compares RankTuner with tuned Alpha Power Loss ($\alpha=0.5$) and Entropy Regularization ($\alpha=0.02$) on AIME24 and OlympiadBench, plots AIME24 Pass@k alongside two RankTuner variants (w/o Prob, w/o Entropy), and measures average inference entropy on AIME24 for Qwen2.5-Math-7B. The paper states the ablations highlight complementary roles of the probability- and entropy-aware terms; the numeric values of these ablation panels are not contained in the provided content.

• The indicator separates token types in practice. On Qwen3-8B chain-of-thought tokens from Minerva Math, replaceable pronouns such as "them" and "all" fall in a neutral region ($\mathcal{I} \approx 1$), while critical tokens including the fraction operator "frac", the key result "0", and the delimiter "{" concentrate in the low-$\mathcal{I}$ region ($\mathcal{I} < 1$).

• Empirical adherence to the theoretical bounds. Visualization on Qwen3-8B / Minerva Math tokens shows $R$ versus $p$ following the upper envelope $R = 1/p$, and $\mathbb{E}[R]$ versus $H$ aligning with the lower bound of Eq. (5). The error distribution of $\frac{1}{s(H)} - \frac{1}{\mathbb{E}[R]}$ is concentrated near zero.

• Code generation is claimed in the abstract but not reported in the provided experimental sections. The abstract mentions an improvement on code generation performance; the setup, tables, and RQ1–RQ3 sections contained in the provided content cover math reasoning and ARC-C/GPQA only.

Methodology in Plain English

The starting point is a simple observation: probability and entropy answer different questions. Probability tells you how much the model agrees with the labeled answer at a position — that is task alignment. Entropy tells you how many plausible alternatives exist there — that is intrinsic difficulty inherited from pre-training. A filler word like "umm" can have high entropy and be wrongly boosted; a word like "essentially" can have low probability simply because "basically" would do just as well, and be wrongly punished.

To combine them fairly, the authors convert both into the same currency: rank. For each token position, they compute (1) the rank $R_t$ of the actual ground-truth token when the vocabulary is sorted by predicted probability, and (2) the expected rank $\mathbb{E}[R_t] = \sum_{\hat{i}} \hat{i} \cdot p_{t,\hat{i}}$, which is the expected number of guesses needed to find the target if you sampled from the model's own distribution (the "guessing problem" of Massey, 1994). Rank and expected rank are directly comparable because both count guesses.

The Relative Rank Indicator is $\mathcal{I}_t = g(f(R_t) - f(\mathbb{E}[R_t]))$ with $f(x) = 1/\log_2(x+1)$ and $g(x) = 2^x$, so that $R_t = \mathbb{E}[R_t]$ gives a neutral value of 1. The indicator falls when the realized rank is worse than expected (an under-learnt token) and rises when the position is intrinsically hard. Because of the logarithmic compression in $f$ and the exponential rescaling in $g$, the indicator saturates near 1 when both ranks are large, creating a neutral regime that de-emphasizes low-likelihood tokens in high-uncertainty contexts.

To justify that construction, the authors prove two tight bounds — rank is at most $1/p_t$, and expected rank is at least a function $s(H_t)$ of entropy — and use the Cauchy Mean Value Theorem to rewrite the indicator as a power law $(\mathbb{E}[R_t]/R_t)^{K(\xi_t)}$, with $K(\xi_t) \approx 0.5$ for typical reasoning tokens. Substituting the bounds yields the objective actually optimized: the Relative Scale $\mathcal{S}_t \approx (p_t \cdot s(H_t))^{-K(\xi_t)}$ with $K(\xi_t) = [\log_2(\xi_t+1)]^{-2}$ and $\xi_t = \max{R_t, s(H_t)}$. Note that $\mathcal{I}_t$ operates as an intrinsic reward signal: it increases when you get right what probability and entropy suggest you should have got right, so more tokens can be correctly classified as replaceable or noisy.

For training, the token weight becomes $\widetilde{w}_t = w_t \cdot \mathcal{S}_t$; the authors set $w_t = p_t$ for math reasoning datasets and $w_t = 1$ for general fine-tuning tasks. Experiments train on the first 10k instances of NuminaMath-CoT using the verl framework on four NVIDIA A800-SXM4-80GB GPUs, with AdamW, learning rate 5e-5, global mini-batch size 256, maximum input length 2048 tokens, and cosine decay with a warm-up ratio of 0.1. Evaluation uses 16 decoding runs at temperature 1.0 and a maximum generation length of 4096 tokens, reporting Pass@1 and Pass@16 on MATH-OAI, Minerva Math, OlympiadBench, AIME 2024, and AMC 2023.

Why This Matters

Impact on research. The paper reframes token reweighting as a calibration problem rather than a choice between two scalars, and supplies a theoretical route from probability and entropy to a single comparable quantity via rank statistics. It also introduces a diagnostic — controlled noise insertion measuring precision/recall@10% and hit@10% — that exposes how one-dimensional weighting surfaces injected noise.

Real-world applications.

• Training math and science tutors where small reasoning errors on tokens like operators, digits, and delimiters matter far more than stylistic word choices.
• Fine-tuning on noisy or human-written corpora, where filler words and rephrasing should not dominate the gradient signal.
• Domain adaptation of general-purpose models to specialized reasoning tasks without over-sharpening behavior on already-confident tokens.
• Building cheaper fine-tuning pipelines, since the reported setup fits on four A800-SXM4-80GB GPUs.

Industry relevance. Any team doing supervised fine-tuning on limited or imperfectly labeled data faces the same trade-off: push hard on the tokens the model gets wrong, or risk over-fitting to ambiguity. RankTuner's selling point is elasticity — the same token can receive a large or small weight depending on the position's intrinsic uncertainty — plus compatibility with existing objectives, since it is implemented as a multiplier on an already-defined loss and trained under the standard verl framework.

Future Directions

• Injecting the Relative Scale into RL-style post-training. The authors state RankTuner is complementary to PPO and GRPO, and describe an extension that injects the Relative Scale into PPO/GRPO-style token-level policy ratios, or uses relative rank and entropy for token selection, explicitly leaving these variants to future work.

• Validating the claimed code generation gains. The abstract mentions improved code generation performance, but the experimental sections contained here report math reasoning and ARC-C/GPQA; a dedicated code evaluation is an obvious next step.

• Robustness of the transformation functions and $\xi$ approximations. The authors note that $(f,g)$ was chosen for stability and a closed form and is not claimed to be optimal, that alternative monotone choices lead to broadly stable results (App. C.5), and that other approximations of $\xi$ are discussed (App. C.6).

• Broader backbone coverage. Supplementary cross-architecture results are referenced (Table 9 of the appendix), pointing to wider testing across model families as a natural extension.

Target Audience

This paper suits machine learning researchers and engineers working on supervised fine-tuning, loss reweighting, and data-centric training pipelines; readers with a background in information theory, rank statistics, or sequential guessing will follow the derivations most easily. It is also useful for practitioners deciding whether to adopt a probability-based, entropy-based, or combined token-weighting scheme, since it reports where RankTuner beats and where it trails the strongest baseline. Readers looking only for a drop-in training recipe can use the Relative Scale definition and the stated hyperparameters, but the argument for why it works requires the rank and bound derivations in Sections 4.1–4.4.

Authors’ abstract

Token-level reweighting is a simple yet effective mechanism for controlling supervised fine-tuning, but common indicators are largely one-dimensional: the ground-truth probability reflects downstream alignment, while token entropy reflects intrinsic uncertainty induced by the pre-training prior. Ignoring entropy can misidentify noisy or easily replaceable tokens as learning-critical, while ignoring probability fails to reflect target-specific alignment. RankTuner introduces a probability--entropy calibration signal, the Relative Rank Indicator, which compares the rank of the ground-truth token with its expected rank under the prediction distribution. The inverse indicator is used as a token-wise Relative Scale to reweight the fine-tuning objective, focusing updates on truly under-learned tokens without over-penalizing intrinsically uncertain positions. Experiments on multiple backbones show consistent improvements on mathematical reasoning benchmarks, transfer gains on out-of-distribution reasoning, and pre code generation performance over probability-only or entropy-only reweighting baselines.