Annotation-Efficient Universal Honesty Alignment

AI summary

Overview

Research area: Natural Language Processing / LLM alignment (honesty and confidence calibration). Technical level: Intermediate. The paper assumes familiarity with supervised fine-tuning, LoRA adaptation, calibration metrics such as AUROC, and self-consistency sampling, though the core idea is explained in accessible terms. Scope: The paper proposes EliCal, a two-stage, annotation-efficient training framework for teaching large language models to express calibrated confidence, and releases HonestyBench, a large-scale benchmark spanning ten free-form QA datasets.

What This Paper Is About

Large language models often sound equally confident whether they are right or wrong, and getting them to accurately report their own reliability normally requires expensive human-labeled correctness annotations at scale. This paper asks whether so many correctness labels are really necessary, and proposes to first teach the model to express the confidence signal it already carries internally (captured cheaply via self-consistency), then calibrate that expressed confidence using only a small set of correctness labels. The goal is universal honesty alignment: a single model whose stated confidence tracks its actual accuracy across many different question-answering tasks.

Key Contributions

1. EliCal, a two-stage framework. Elicitation-Then-Calibration first trains the model to express internal confidence using inexpensive self-consistency supervision over a large question set, then calibrates that confidence against correctness using a small annotated set. The authors frame this as a pretraining–finetuning paradigm tailored to honesty alignment.
2. HonestyBench, a large-scale honesty benchmark. It consolidates ten public free-form factual QA datasets, with over 560k training samples, roughly 38k in-domain evaluation samples and roughly 33k out-of-domain evaluation samples. Each model–question pair includes twenty sampled responses and one greedy response from three LLMs (Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, and Llama3-8B-Instruct), annotated for both correctness and self-consistency.
3. A large-scale study of the honesty alignment upper bound. The authors report the first training and validation of honesty alignment at this scale, showing performance saturates as annotated data grows and providing a reference upper bound across ten QA tasks.
4. Demonstration of annotation efficiency and generalization. EliCal reaches roughly 98% of the full-supervision upper bound with only 1k correctness annotations (about 0.18% of full supervision), and generalizes better than a calibration-only baseline on unseen MMLU tasks.

Main Findings

• Training-free methods are weakest, and self-consistency wins among them. On Qwen2.5-7B-Instruct, semantic self-consistency (Consis-Sem) achieved the highest average in-domain AUROC among training-free methods at 73.62, ahead of Consis-Lex (69.80), Verbal-10 (68.90), Verbal-0 (67.22), N-Prob (64.75) and Prob (55.48).
• Full supervision approaches an upper bound, far above training-free baselines. Trained on all available annotations, EliCal (560k) reached an average in-domain AUROC of 86.49 and Cal-Only (560k) reached 86.20 on Qwen2.5-7B-Instruct, outperforming the best training-free baseline by over 17%.
• EliCal is annotation-efficient. With only 1k correctness annotations, EliCal (1k) averaged 84.36 in-domain AUROC on Qwen2.5-7B-Instruct, versus 73.41 for Cal-Only (1k). The authors report this is about 98% of the Cal-Only performance trained on over 560k annotations. Cal-Only (1k) failed to beat the best training-free methods on several datasets, including NQ and HQ.
• EliCal generalizes better out of domain. On the OOD evaluation with Qwen2.5-7B-Instruct, EliCal (1k) averaged 84.47 AUROC compared with 77.32 for Cal-Only (1k); at full supervision the two converge (85.83 vs 85.75).
• The generalization advantage is clearest on MMLU. On the multiple-choice MMLU benchmark, which differs substantially from the free-form training questions, Cal-Only lags behind EliCal even with over 560k annotations.
• Models can be taught to express internal confidence. Eli-Only, which learns from Consis-Sem signals, performs on par with Consis-Sem while requiring no annotated data and no repeated sampling at inference, removing the cost of consistency checking during deployment.
• Results hold on a larger model. With Qwen2.5-32B-Instruct, EliCal (1k) averaged 84.39 in-domain AUROC versus 79.31 for Cal-Only (1k) and 74.57 for Consis-Sem; at full supervision EliCal (560k) reached 87.12 versus 86.95 for Cal-Only (560k).
• Binarized confidence is usable for decisions. Using an alignment metric where 20% of each test set selects a threshold and the remaining 80% is evaluated, EliCal significantly outperformed Cal-Only; in-domain they converge with abundant annotations, but on MMLU EliCal consistently leads.
• Ablations. Elicitation performance improves with more training data and approaches Consis-Sem; training only a linear head still beats Cal-Only under limited data but performs worse overall than the LoRA setup; and EliCal is stable across sample sizes k in {2, 5, 10, 20}, while Eli-Only shows only minor variation with k, suggesting that even k=2 provides a sufficiently informative signal.

Methodology in Plain English

The authors keep the underlying language model frozen and add LoRA modules to all linear layers plus a small linear head on the final layer that outputs a single confidence score. Training minimizes mean squared error between the predicted score and a target.

The training happens in two stages. In Stage 1 (Confidence Elicitation), the target for each question is the model's self-consistency confidence — roughly, how consistently the model produces answers semantically matching its most confident answer across repeated samples. This signal needs no human annotation and can be collected at scale, so the model learns to state its internal confidence in one shot instead of requiring many samples at inference. In Stage 2 (Confidence Calibration), the target switches to actual correctness, but only a small set of correctness-annotated QA pairs is used to adjust the already-elicited confidence so it matches real accuracy.

For evaluation, they measure honesty alignment with AUROC, which captures how well confidence separates correct from incorrect answers, plus ECE (reported in the appendix) and a binarized alignment metric. Ground-truth checking, semantic consistency checking, and correctness annotation for the benchmark are performed with Qwen2.5-32B-Instruct using fixed prompts. They compare against six training-free baselines (probability-based, self-consistency-based, and verbalized confidence, each in two variants) and two training-based baselines (Elicitation-Only and Calibration-Only), plus the recent Thermometer and DACA methods.

Why This Matters

Impact on research. The paper reframes honesty alignment as a two-stage learning problem and argues that correctness annotations mainly serve to calibrate confidence rather than to teach its expression. If confidence expression is learnable from cheap self-consistency signals, then large-scale annotation budgets can be redirected, and the released benchmark and models give the community a common testbed for studying the performance ceiling of universal honesty alignment rather than only in-domain settings.

Real-world applications:

• Deciding when a model should answer versus abstain or defer to a human.
• Triggering retrieval-augmented generation only when model confidence is low.
• Flagging low-confidence answers in factual QA, customer support, or knowledge-assistant deployments.
• Reducing the labeling cost of building reliable self-assessment into models for high-stakes domains.

Industry relevance. Correctness labeling at the scale the paper studies is expensive, so a method that reportedly matches near-optimal alignment using about 0.18% of full supervision is directly relevant to teams that need trustworthy deployment signals but cannot afford massive annotation pipelines. The fact that training uses LoRA rather than full fine-tuning also lowers the compute barrier to reproducing or adopting the approach.

Future Directions

• Extending beyond the current scope. The authors explicitly identify multi-turn interactions, multimodal settings, and a broader range of task types as important directions, since this work focuses on single-turn, text-only, free-form QA.
• Closing remaining gaps on harder task formats. EliCal still leads on MMLU, but the margin and the underlying causes of the format gap between free-form and multiple-choice questions remain open questions.
• Alternative supervision signals for elicitation. The paper compares self-consistency with semantic uncertainty, noting that semantic uncertainty requires costly clustering; whether cheaper or stronger internal signals could improve Stage 1 is unresolved.
• Understanding the elicitation–calibration division of labor. The ablation showing that EliCal remains stable across k values suggests the elicitation signal is robust, but how the optimal split between elicitation data and calibration annotations shifts with model size or task domain is not settled.

Target Audience

This paper is most useful to NLP and machine learning researchers working on LLM alignment, calibration, and uncertainty estimation, as well as AI safety researchers interested in honesty and knowledge-boundary recognition. Engineers and product teams deploying LLMs who need confidence-based routing, abstention, or retrieval triggers will find the annotation-efficiency and generalization results practically relevant. Readers without background in calibration metrics such as AUROC or in parameter-efficient fine-tuning will need some supplementary reading.

Authors’ abstract

Honesty alignment-the ability of large language models (LLMs) to recognize their knowledge boundaries and express calibrated confidence-is essential for trustworthy deployment. Existing methods either rely on training-free confidence estimation (e.g., token probabilities, self-consistency) or training-based calibration with correctness annotations. While effective, achieving universal honesty alignment with training-based calibration requires costly, large-scale labeling. To support annotation-efficient training, we introduce Elicitation-Then-Calibration (EliCal), a two-stage framework that first elicits internal confidence using inexpensive self-consistency supervision, then calibrates this confidence with a small set of correctness annotations. To support a large-scale study, we release HonestyBench, a benchmark covering ten free-form QA datasets with 560k training and 70k evaluation instances annotated with correctness and self-consistency signals. Experiments show that EliCal achieves near-optimal alignment with only 1k correctness annotations (0.18% of full supervision) and better alignment performance on unseen MMLU tasks than the calibration-only baseline, offering a scalable solution toward universal honesty alignment in LLMs.