The Confidence Trap: Gender Bias and Predictive Certainty in LLMs

AI summary

Overview

• Research area: Natural Language Processing — fairness and uncertainty calibration in large language models, specifically gender bias in pronoun/coreference resolution.
• Technical level: Intermediate. The paper introduces a new metric that requires understanding of confidence calibration (ECE, reliability diagrams, Brier score), but it defines each metric from scratch.
• Scope: An empirical study of six open-weight LLMs across four gender/LGBTQ+ bias benchmarks, evaluating whether model confidence scores are calibrated and whether calibration error tracks human bias judgments.
• Venue metadata: arXiv:2601.07806v1 [cs.CL], 12 Jan 2026. Institutions: University of Tartu (Estonia) and Plaksha University (India).

What This Paper Is About

Large language models output probability scores alongside their predictions, and users often treat a high-confidence score as a signal that the prediction is trustworthy. This paper asks whether that assumption holds when the prediction involves gendered pronouns — that is, whether a model that says "80% confident" about a pronoun is actually right about 80% of the time, and whether that reliability differs between male and female pronouns. The goal is to determine whether standard calibration metrics are sufficient to expose fairness problems, and to propose a gender-aware metric that does.

Key Contributions

1. A fairness-aware calibration evaluation of six open-weight LLMs — GPT-J-6B, Falcon3-7B-base, Llama-3.1-8B, Gemma-2-9B, Qwen2.5-7B, and DeepSeek-R1-Distill-Llama-8B (DeepSeek-8B) — measuring predicted token confidence against human-annotated bias judgments.
2. A new metric, Gender-ECE (Gender-Aware Group-ECE), which computes ECE separately for male and female predicted labels and averages the two, exposing calibration disparities that aggregate ECE hides. It differs from MacroCE, which splits instances by correct/incorrect prediction rather than by gender, and is bin-wise rather than instance-wise.
3. Cross-benchmark evidence on GenderLex, WinoBias, Winogender, and WinoQueer, plus a post-hoc Beta calibration experiment showing that calibration repairs confidence estimates and raises accuracy but does not remove bias.
4. Public code release at https://github.com/ahmedssabir/GECE, along with an ablation on how sample size affects the stability of ECE estimates.

Main Findings

• Gemma-2-9B is the worst-calibrated model. On GenderLex it reaches an ECE of 0.327, the highest among the six models (GPT-J-6B is lowest at 0.076). It also shows the worst calibration on WinoBias (ECE 0.429) and Winogender (ECE 0.373), and the weakest human alignment on WinoBias (0.509).
• GPT-J-6B is the best-calibrated model overall. It has the lowest ECE on GenderLex (0.076), the lowest on Winogender (0.086), and among the best on WinoBias (0.157), with relatively small male–female Gender-ECE gaps (M 0.085, F 0.066 on GenderLex).
• Llama-3.1-8B shows the strongest alignment with human bias judgments (0.727 on GenderLex, the highest in Table 1), with nearly identical calibration across genders there (M 0.112, F 0.109).
• Falcon3-7B produces the most accurate probability estimates — lowest Brier score on GenderLex (0.356) — but with a wider male–female gap (M 0.081, F 0.217).
• Calibration is worse for female pronouns. Most models show higher ECE for female pronouns on WinoBias and GenderLex. Gemma-2-9B is extreme: WinoBias male ECE 0.067 versus female ECE 0.895; GenderLex male 0.056 versus female 0.901.
• Bigger models do not fix the disparity. Moving from Llama-3.1-8B to Llama-3.1-70B raises WinoBias ECE from 0.193 to 0.216 (+11.9%), with male ECE up 15.6% to 0.207 and female up 1.6% to 0.253. Gemma-2-9B to Gemma-2-27B lowers overall ECE by 14.7% to 0.366 and male ECE by 22.1% to 0.341, but female ECE rises 144.2% to 0.381.
• Gender-neutral wording increases miscalibration. On GenderLex, replacing occupation titles with "Someone" or "Person" raises error (Gemma-2-9B ECE 0.364 for "Someone" and 0.367 for "Person" versus 0.327 for the occupation condition; Gender-ECE 0.450 versus 0.267).
• The WinoQueer results reverse the ranking. Gemma-2-9B, the worst model on gender benchmarks, is the most calibrated on WinoQueer (Gay 0.026, Queer 0.182), while Qwen2.5-7B is worst overall (Trans 0.919, Lesbian 0.898, Gay 0.189) and DeepSeek-8B is poorly calibrated for Queer (0.910). All models show high miscalibration for Lesbian and Trans identities.
• Distillation degrades calibration. DeepSeek-8B, distilled from Llama-3.1-8B, shows higher gender calibration error and less human alignment than its base model; the authors note this effect diminishes in larger models such as DeepSeek-70B.
• Calibration error does not equal human bias. The paper's headline claim is that calibration error and human-annotated bias do not always correlate, and the degree of misalignment varies by model.
• Post-hoc Beta calibration helps. Splitting the data into 385 validation and 386 test instances, Beta calibration reduces ECE by roughly a factor of three and raises accuracy: GPT-J-6B 69.2% to 76.9%, Llama-3.1-8B 65.8% to 74.9%, DeepSeek-8B 63.5% to 69.9%, Gemma-2-9B 51.6% to 54.7%, Qwen2.5-7B 61.1% to 76.4%.
• Calibration is not bias mitigation. The authors state explicitly that Beta calibration improves confidence reliability without addressing the underlying sources of bias.
• ECE estimates are unstable at small sample sizes. Drawing 100 subsets without replacement from WinoBias (771 instances) with DeepSeek-8B, ECE standard deviation falls from 0.0381 at N=50 to 0.0076 at N=500, while the mean drifts from 0.2630 to 0.2401.

Methodology in Plain English

The team feeds each model template-based sentences containing an occupation and a pronoun, drawn from four benchmarks, and does a single deterministic forward pass with no decoding. They read the logits for the pronoun slot and convert them into a probability using a softmax over the vocabulary, so they can see how confident the model is about "her" versus "him".

Human annotators — three subjects plus one expert, deciding by majority vote — label each sentence with which gender the bias favors: 1 for male bias, 0 for female bias. The expert resolves cases where annotators were unfamiliar with rare occupations. Agreement was moderate on GenderLex (average pairwise Cohen's kappa = 0.51), which the authors attribute to rare occupation terms.

They then compute five calibration measures: Expected Calibration Error (ECE), which bins predictions by confidence and averages the gap between confidence and accuracy; reliability diagrams, which visualize that gap; Instance Calibration Error (ICE); Macro Calibration Error (MacroCE), which splits by correct versus incorrect predictions; and the Brier score, a squared-error scoring rule. Their new Gender-ECE repeats ECE separately for male-labeled and female-labeled predictions and averages the two, so a model that is well calibrated on men and badly calibrated on women no longer looks fine in aggregate.

For the fairness probe, they use WinoBias-syntax type-2 (1,542 sentence pairs, 771 unique sentences), replacing occupation titles with "person" to reduce ambiguity; Winogender (720 template sentences) for stereotype-free pronoun resolution; GenderLex (1,676 sentence pairs, 838 unique sentences, ChatGPT 3.5-turbo generated with human annotation and correction) for "last cloze" sentences where the pronoun comes at the end; and WinoQueer, from which they extract 1,488 of 45,540 sentences that end in a last-cloze structure. Finally, they apply Beta calibration on top of the WinoBias outputs to show what post-hoc correction can and cannot achieve. Experiments ran on an A40 GPU (48GB VRAM), with Llama-3.1-70B using 4 A100 GPUs (80GB each).

Why This Matters

Impact on research. The paper argues that aggregate calibration metrics can mask subgroup-specific miscalibration — a model can look adequately calibrated overall while being nearly perfectly calibrated on male pronouns and badly miscalibrated on female ones. Gender-ECE is offered as a cheap addition to existing fairness toolkits. The finding that more parameters improve male calibration while worsening female calibration (the +144.2% jump for Gemma-2-27B) challenges the assumption that scale fixes trustworthiness, and the WinoQueer results show that a model's gender-bias ranking does not transfer to other identity groups.

Real-world applications:

• AI-assisted resume screening and hiring — explicitly named by the authors as a trust-sensitive setting where overconfident gender-skewed predictions could steer candidate evaluation.
• Healthcare decision support — a named high-stakes domain where an overconfident, miscalibrated prediction can mislead a human reviewer.
• Legal decision-making — also named, where confidence scores may be taken as evidence of reliability.
• Model selection and procurement — the per-model comparison gives deployment teams a concrete basis for choosing between open-weight models when gender-sensitive text is involved.

Industry relevance. All six evaluated models are open-weight and widely deployed (GPT-J-6B, Falcon3-7B-base, Llama-3.1-8B, Gemma-2-9B, Qwen2.5-7B, DeepSeek-R1-Distill-Llama-8B), so the results apply directly to self-hosted deployments. The distillation result matters for teams that compress large models into smaller ones: calibration can degrade even when the parent model is acceptable. The Beta calibration experiment gives a practical, low-cost repair recipe — with the caveat that it fixes the confidence numbers, not the bias.

Future Directions

1. Post-processing bias mitigation. The authors note that calibration techniques are not bias mitigation and state that proposing post-processing mitigation methods is their planned next step.
2. Extending beyond gender. They plan to cover stereotypes related to nationality and disability, citing Nangia et al. 2020.
3. Cross-lingual bias transfer. The current framework is English-only with gendered pronouns; future work will study English as a pivot for training low-resource languages.
4. Stabilizing calibration estimates under small samples. The ablation shows ECE variance grows sharply at smaller N, and the authors call for combining complementary measures and ensuring each bin holds enough instances — an open problem for subgroup metrics like Gender-ECE, where splitting by gender halves the data available per bin.

Target Audience

Researchers and practitioners working on LLM fairness, uncertainty quantification, and trustworthy ML; engineers evaluating or deploying open-weight models in high-stakes settings such as hiring, healthcare, or legal review; and anyone studying coreference resolution or gender bias benchmarks (WinoBias, Winogender, GenderLex, WinoQueer). Readers need basic familiarity with classification probabilities — the calibration math is presented in full, so no prior exposure to ECE specifically is required.

Authors’ abstract

The increased use of Large Language Models (LLMs) in sensitive domains leads to growing interest in how their confidence scores correspond to fairness and bias. This study examines the alignment between LLM-predicted confidence and human-annotated bias judgments. Focusing on gender bias, the research investigates probability confidence calibration in contexts involving gendered pronoun resolution. The goal is to evaluate if calibration metrics based on predicted confidence scores effectively capture fairness-related disparities in LLMs. The results show that, among the six state-of-the-art models, Gemma-2 demonstrates the worst calibration according to the gender bias benchmark. The primary contribution of this work is a fairness-aware evaluation of LLMs' confidence calibration, offering guidance for ethical deployment. In addition, we introduce a new calibration metric, Gender-ECE, designed to measure gender disparities in resolution tasks.