Exploring the Effects of Alignment on Numerical Bias in Large Language Models

AI summary

Overview

Research area: Natural Language Processing — specifically LLM-as-a-judge evaluation, model alignment (instruction tuning and preference tuning), and evaluation reliability.

Technical level: Intermediate. The paper assumes familiarity with LLM prompting, alignment terminology, correlation coefficients, and distribution statistics such as kurtosis.

Scope: The paper tests whether alignment causes "numerical bias" (the disproportionate overuse of specific numeric scores) in LLM evaluators, and evaluates three strategies for mitigating that bias.

What This Paper Is About

LLMs used as judges tend to hand out the same few scores over and over regardless of the input, which undermines the reliability of the evaluation. Because nearly all evaluator LLMs are aligned through instruction tuning and preference tuning, and prior work shows alignment reduces output diversity, the authors hypothesize that alignment is the source of the problem. They compare pre-alignment and post-alignment versions of the same models across three scoring tasks to test this, then try to reduce the bias.

Key Contributions

1. A controlled pre- vs. post-alignment comparison for numerical bias. The authors use four open model series in matched pairs (gemma-7b / gemma-7b-it, Mistral-7B-v0.1 / Mistral-7B-Instruct-v0.1, Meta-Llama-3-8B / Meta-Llama-3-8B-Instruct, Qwen2-7B / Qwen2-7B-Instruct) and measure bias with kurtosis across MTQE, GECQE, and LCP scoring tasks.

2. Evidence that alignment is a cause of numerical bias. Post-alignment models show higher kurtosis than their pre-alignment counterparts on most datasets, and generated scores concentrate heavily around particular values (notably 8 on a 0–9 scale) regardless of the input.

3. A quantified link between bias strength and evaluation accuracy. Across models and language pairs, kurtosis and Pearson correlation with gold scores show a strong negative correlation of −0.60 for both MTQE and GECQE.

4. Systematic evaluation of three mitigation strategies. Temperature scaling, distribution calibration, and score range adjustment are compared, with score range adjustment identified as the most effective and a reframing of the score range as a tunable hyperparameter rather than a fixed design choice.

Main Findings

• Alignment amplifies numerical bias. Pre-alignment models produce varied score distributions (attributed to weak instruction-following rather than genuine quality), while post-alignment models concentrate scores around specific values such as 8 on the 0–9 scale, and their distributions look strikingly similar across datasets even when the gold distributions differ.

• Greater bias means lower accuracy. Models and language pairs with higher kurtosis tend to have lower Pearson correlation with human scores, with a strong negative correlation of −0.60 for both MTQE and GECQE.

• Bias strength varies by model. Post-alignment Gemma shows the highest kurtosis for most language pairs, Mistral generally follows, and Llama and Qwen show lower kurtosis for most language pairs except En-De.

• Bias strength varies by language. For En-De, En-Zh, Et-En, Ro-En, and Ru-En, post-alignment models show highly skewed distributions. For Ne-En and Si-En, distributions differ across models with no excessive concentration; the authors suggest lower representation of Nepali and Sinhala in alignment data may explain this.

• Post-alignment models still win overall. Despite introducing bias, aligned models outperform their pre-alignment counterparts, suggesting the performance loss from bias does not outweigh the gains from better instruction following.

• Temperature scaling reduces bias but not necessarily improves accuracy. Higher temperatures mitigate bias, especially for strongly biased models, but the temperature maximizing correlation is 1.0 for Gemma and 0.7 for the other models, which matches their default settings. Excessive temperature decreases accuracy.

• Distribution calibration reduces kurtosis but has mixed performance effects. In MTQE, all models show reduced kurtosis after calibration (for example, Gemma's En-De kurtosis drops from 128.174 to 47.932 and Mistral's from 52.923 to 3.394). In GECQE, Mistral (8.485 to 0.106) and Llama (3.510 to 0.994) improve. Calibration slightly helps models whose original accuracy is relatively high (Llama and Qwen) but tends to degrade performance when original accuracy is low.

• Score range adjustment is the most promising strategy. Testing {1–5, 0–9, 1–100} at a fixed temperature of 0.7, the 1–5 scale achieves the lowest kurtosis for most tasks and models but does not always give the best performance. Ranges with the highest performance also tend to have relatively low kurtosis, indicating each model and task may have its own optimal range. For Gemma on MTQE, the 1–100 scale reaches r = 0.14, surpassing the best temperature setting and the calibration result.

• Input features correlate with bias. Using a "mode ratio" as an example-level bias indicator, MTQE results show that perplexity negatively correlates with the mode ratio for all models except Gemma, meaning more fluent sentences show stronger bias. In GECQE, word overlap negatively correlates with the mode ratio across all models, meaning more correction leads to stronger bias.

Methodology in Plain English

The researchers take four open LLM series where both a base (pre-alignment) and instruction-tuned (post-alignment) version are publicly available, so the only major difference between paired models is alignment. They feed each model the same evaluation prompts for three scoring tasks: machine translation quality estimation, grammatical error correction quality estimation, and lexical complexity prediction. Prompts follow a template based on GEMBA, with the score range injected as a variable.

For each input, the model is asked to output only a numeric score, with a maximum of 5 tokens to suppress non-numeric text. For the first experiment the temperature is fixed at 0.7 and the score range at 0–9. Each input is evaluated 10 times; non-numeric outputs are discarded, out-of-range values are clipped to the boundary, and the remaining scores are averaged.

Bias is measured with kurtosis of the score distribution, which captures local concentration around particular values better than variance or interquartile range would. Evaluation quality is measured with Pearson's correlation coefficient between model scores and human scores, with statistical significance tested at p < 0.01. Gold scores are rescaled to the model's evaluation range before computing kurtosis.

For the mitigation experiment, the authors vary temperature over {0.4, 0.7, 1.0, 1.3} with the range fixed at 0–9, then fix temperature at 0.7 and vary the range over {1–5, 0–9, 1–100}. For calibration, they apply a generative calibration method: the label marginal p(y) is estimated by sampling 1,000 times, the true distribution q(y) is modeled as a Beta distribution with parameters estimated by maximum likelihood, and the final score is a weighted average of sampled scores where each weight is q(y_i)/p(y_i) normalized. Experiments were run on a single Quadro RTX 8000.

Why This Matters

For research, the paper shifts numerical bias from a descriptive complaint to a testable consequence of alignment. It introduces kurtosis as a practical measurement tool, shows bias can be audited without gold data, and demonstrates that a design choice long treated as fixed — the score range — is actually a tunable variable.

Real-world applications:

• Building LLM-as-a-judge pipelines. Teams can select evaluator models by kurtosis alone when no gold scores exist, and tune the score range to improve reliability.
• Machine translation quality estimation. Systems that flag translations needing human post-editing depend on score distributions that reflect real quality differences; concentrated scores hide those differences.
• Grammatical error correction evaluation. The word-overlap finding indicates that heavily edited sentences are the most likely to receive biased scores, which matters for correction systems that make many edits.
• Dataset and benchmark construction. Knowing that high-resource languages in alignment data show stronger bias points to gaps in alignment data coverage for low-resource languages.

For industry, the practical takeaway is that aligned models remain the right choice as evaluators but should be audited for score concentration, and that prompt-level adjustments to the score range offer a cheap accuracy improvement compared with retraining or calibrating models.

Future Directions

• Identify the optimal score range systematically. The authors call score range adjustment heuristic and task-specific, and state that optimal score range selection is nontrivial and remains open.

• Move beyond symptom-level fixes. The paper argues that future work should seek more fundamental approaches that directly address the causes of alignment-induced numerical bias rather than prompt-level remedies.

• Study varying alignment strengths. The conclusion proposes exploring a range of alignment intensities to understand how bias scales with alignment.

• Extend to other evaluation formats. The limitations note the study covers only numeric scores; the effect of alignment on natural language labels and rankings is unresolved.

• Open up alignment data and methods. Because public models do not disclose their alignment methods and data, the authors cannot analyze which data properties or methodological choices drive the effect, and in-house alignment of large pretrained models is computationally expensive.

Target Audience

NLP researchers studying evaluation methodology, LLM alignment side effects, and model calibration; machine learning engineers building or maintaining automated evaluation pipelines; and applied practitioners in translation, writing assistance, or education technology who rely on LLM-generated quality scores and need to know when those scores can be trusted.

Authors’ abstract

"LLM-as-a-judge," which utilizes large language models (LLMs) as evaluators, has proven effective in many evaluation tasks. However, evaluator LLMs exhibit numerical bias, a phenomenon where certain evaluation scores are generated disproportionately often, leading reduced evaluation performance. This study investigates the cause of this bias. Given that most evaluator LLMs are aligned through instruction tuning and preference tuning, and that prior research suggests alignment reduces output diversity, we hypothesize that numerical bias arises from alignment. To test this, we compare outputs from pre- and post-alignment LLMs, and observe that alignment indeed increases numerical bias. We also explore mitigation strategies for post-alignment LLMs, including temperature scaling, distribution calibration, and score range adjustment. Among these, score range adjustment is most effective in reducing bias and improving performance, though still heuristic. Our findings highlight the need for further work on optimal score range selection and more robust mitigation strategies.