Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging

AI summary

Overview

• Research area: Natural Language Processing; specifically post-training of large language models, calibration, and weight-space model merging.
• Technical level: Intermediate. The method itself is conceptually simple (weight interpolation), but the evaluation involves calibration metrics, multiple model families, and loss-landscape interpretation.
• Scope: The paper argues that the "alignment tax" of instruction tuning includes a severe loss of calibration alongside accuracy changes, and shows that interpolating model weights between a pre-trained (PT) and instruction-tuned (IT) model can yield merged models that Pareto-dominate both parents on accuracy and calibration.

What This Paper Is About

Post-training makes LLMs more helpful and safe, but it also degrades performance on some benchmarks, an effect usually called the "alignment tax." This paper argues that framing the tax only as an accuracy drop misses a second, equally serious cost: instruction tuning makes models badly miscalibrated, meaning they become far more confident without becoming correspondingly more accurate. The authors show that by simply blending the weights of a base pre-trained model and its instruction-tuned counterpart, one can trace a continuous frontier between them and find "sweet spot" merges that improve accuracy relative to both parents while substantially recovering the calibration that alignment destroyed.

Key Contributions

1. Reframing the alignment tax to include calibration. The authors show that instruction tuning universally degrades calibration (measured by Expected Calibration Error, ECE), with ECE values increasing by an order of magnitude across every model they test, while accuracy effects are mixed.
2. Demonstrating a Pareto-superior frontier via model merging. By interpolating between PT and IT weights with a coefficient λ, they consistently find merged models that exceed both parents in accuracy on some benchmarks while recovering a substantial portion of the lost calibration — a result they describe as not a zero-sum trade-off.
3. Establishing generality and scaling behavior. The pattern holds across Gemma-3, Qwen2.5, and Llama-3.1-8B families, across benchmarks, and across three merge algorithms (Linear, SLERP, DARE-TIES). The payoff, robustness, and predictability of merging all improve with model scale.
4. Identifying the mechanism and auditing safety. They trace the miscalibration to confidence inflation (GPQA mean confidence rising from around 40% in PT models to over 90% in IT models without commensurate accuracy gains), give a geometric interpretation via Linear Mode Connectivity, and report safety sweeps on ToxiGen and WMDP across the full λ range.

Main Findings

• Calibration is universally degraded by instruction tuning. On MMLU-Pro, ECE rises sharply for every model tested: Gemma-3-1B (0.07 to 0.66), Gemma-3-4B (0.02 to 0.64), Gemma-3-12B (0.02 to 0.53), Gemma-3-27B (0.04 to 0.48), Qwen2.5-1.5B (0.06 to 0.33), Qwen2.5-3B (0.04 to 0.47), and Qwen2.5-7B (0.06 to 0.45). Accuracy over the same comparison is mixed: it improves for Gemma-3-1B (11.2 to 14.2), Gemma-3-4B (27.9 to 29.8), and Qwen2.5-3B (32.1 to 32.8), but degrades for Gemma-3-12B (42.4 to 39.8), Gemma-3-27B (49.4 to 47.8), Qwen2.5-1.5B (28.7 to 28.1), and Qwen2.5-7B (43.6 to 43.1).

• Merged models dominate the naive PT-IT trade-off. The path traced by merged models strictly dominates a naive linear interpolation between the base and instruct endpoints, and the authors mark an optimal merge coefficient λ* at which accuracy is comparable to or exceeds either parent while calibration is partly restored.

• Concrete Pareto-dominant examples. On Llama-3.1-8B with SLERP at λ=0.9, BBH accuracy reaches 51.0% versus PT 46.5% and IT 50.7%, with ECE of only 0.074 versus IT 0.198. On GPQA, the λ=0.7 merge reaches 33.0%, exceeding both PT (31.5%) and IT (29.1%), while keeping ECE at 0.065 versus IT 0.320. On Gemma-3-27B MMLU-Pro, the SLERP merge at λ=0.5 reaches 51.91% accuracy versus PT 49.39% and IT 47.80%, with ECE 0.219 versus IT 0.478.

• The mechanism is confidence inflation. Instruction tuning sharply inflates prediction confidence without a matching accuracy gain. On GPQA, mean prediction confidence for instruction-tuned models surges from around 40% to over 90%, which the authors identify as the direct driver of the observed miscalibration.

• Benefits grow with model scale. The peak accuracy gain over the instruction-tuned parent is marginal for small models but exceeds 4 percentage points for the 12B and 27B Gemma-3 models. Larger models also show smoother, concave performance landscapes (so λ choice is less critical) and more predictable optimal coefficients, with λ* converging around 0.4–0.5 for MMLU-Pro and GPQA; a default of λ=0.5 generalizes well across benchmarks.

• The effect holds across merge algorithms. Linear, SLERP, and DARE-TIES trace distinct but similar Pareto-superior paths, indicating the result is not an artifact of one merging method.

• Gains extend to generative diversity and simulation. On NoveltyBench (curated subset) for Gemma3-12B, merging resolves the trade-off between the over-confident IT model's low diversity and the PT model's diffuse outputs. On SimBench, the optimal merged model scores 20.4, substantially outperforming both PT (7.7) and IT (18.2).

• Safety does not regress below the base model. In sweeps over λ from 0.1 to 0.9 for Gemma-3-12B and Qwen-2.5-7B, ToxiGen safety scores decrease as base weights are introduced, stabilizing around the base model level (roughly 43–60%), while WMDP hazardous knowledge scores remain relatively flat (Qwen stays between 60–62%). No merged model is less safe than the publicly available base model.

• Amplifying task vectors is destructive. Setting λ>1 causes a performance cliff: IFEval drops from 75.4% at λ=1.1 to 10.3% at λ=2.0, and MATH Level 5 collapses from 55.1% to 0.6%, with ECE worsening throughout.

Methodology in Plain English

The authors take open-weight model families — Gemma-3 and Qwen2.5 (both in pre-trained and officially released instruction-tuned versions), plus Llama-3.1-8B — and blend their weights using a coefficient λ that ranges from 0 (pure pre-trained) to 1 (pure instruction-tuned). Blending is done with MergeKit using Spherical Linear Interpolation (SLERP) as the primary method, with linear interpolation and DARE-TIES run as robustness checks. The procedure is entirely post-hoc: it requires no additional training, no gradient-based optimization, and can be performed without GPUs.

Each resulting model is evaluated along two axes. Task performance is measured as accuracy on MMLU-Pro, GPQA, BBH, MATH (Level 5 problems only), and IFEval. Calibration is measured by Expected Calibration Error. All inference is run in BF16 through the LM Evaluation Harness, using Open LLM Leaderboard configurations for few-shot counts but without accuracy normalization, since raw accuracy is needed for calibration. The authors also test generative diversity on NoveltyBench and distributional human-behavior prediction on SimBench, and they run safety evaluations on ToxiGen and WMDP across the λ sweep. For interpretation, they connect the results to loss-landscape geometry via Linear Mode Connectivity, arguing that instruction tuning lands in sharp, low-entropy minima while pre-trained models sit in flatter, higher-entropy basins.

Why This Matters

Impact on research. The paper argues that calibration degradation is a central, previously under-characterized part of the alignment tax rather than a side issue, and that prior merging work on this axis (such as Lin et al., 2024) diagnosed the tax only through perplexity and downstream accuracy. It positions simple weight interpolation as a strong, computationally cheap baseline against which more elaborate calibration methods must be compared — noting that Temperature Scaling cannot improve accuracy because it is a monotonic transformation, and that Self-Consistency can fail when a model is structurally overconfident in a wrong answer while also incurring O(k) inference cost versus the merge's O(1) cost.

Real-world applications (the paper's evaluations correspond to these settings, though it does not deploy any of them):

• Serving open-weight models in settings where a confidently wrong answer is worse than an uncertain one, such as question answering over knowledge-intensive domains like graduate-level science (GPQA, MMLU-Pro).
• Instruction-following and constrained-output tasks where format compliance matters (IFEval), where merged models retain substantially higher accuracy than the base model at intermediate λ.
• Creative and open-ended generation, where restoring calibration is associated with higher utility and distinctness on NoveltyBench.
• Human-behavior simulation and distributional prediction, where the optimal merged model scored 20.4 on SimBench versus 7.7 for PT and 18.2 for IT.

Industry relevance. The approach requires no training, no gradient computation, and no GPUs, making it unusually cheap to adopt in the open-weight ecosystem. It also comes with an explicit caveat: any merge reintroduces weights from a model that has not undergone full safety fine-tuning, so practitioners are cautioned to run rigorous safety testing on any chosen "sweet spot" model rather than deploying it on accuracy and calibration metrics alone. The method is inapplicable to proprietary API-only models because it requires full weight access.

Future Directions

• Hierarchical and finer-grained merging. The authors suggest applying different merge coefficients to different layers or modules, which they did not test, as a way to gain finer control over the trade-off.
• More sophisticated merge strategies. They did not perform an exhaustive hyperparameter search for more complex methods, such as varying densities in DARE-TIES.
• A formal theory of Pareto-superiority. The paper offers a geometric interpretation through Linear Mode Connectivity but explicitly does not provide a formal mathematical account of why merged models consistently exceed both parents in accuracy. Developing such a theory, possibly from loss-landscape geometry or connections to Bayesian model averaging, is flagged as important.
• Safety-oriented merge selection. Given that merged models can dilute safety guardrails relative to the fully aligned model, an open question is how to select sweet-spot merges that preserve refusal behavior and other safety properties while recovering calibration.

Target Audience

Researchers and engineers working on LLM post-training, alignment, calibration, and uncertainty quantification will find the core argument most relevant, as will practitioners deploying open-weight models who need reliability without paying for retraining. The paper is also useful for model-merging researchers, since it targets the PT-to-IT axis rather than merging distinct fine-tunes, and for safety researchers, because it quantifies how far merging can move a model back toward its base before safety properties erode. A working familiarity with concepts such as ECE, instruction tuning, and weight-space interpolation is assumed; the core idea is accessible, but the empirical results require comfort with benchmark and calibration metrics.

Authors’ abstract

The "alignment tax" of post-training is typically framed as a drop in task accuracy. We show it also involves a severe loss of calibration, making models overconfident, less reliable, and model outputs less diverse. We show that this trade-off can be navigated effectively via a simple post-hoc intervention: interpolating between a model's weights before and after alignment. Crucially, this is not a strict trade-off. We find that the process consistently reveals Pareto-optimal interpolations - models that improve accuracy beyond both parents while substantially recovering the calibration lost during alignment. Our work demonstrates that simple model merging provides a computationally efficient method for mitigating the full scope of the alignment tax, yielding models that are more capable and more reliable.