ADVICE: Answer-Dependent Verbalized Confidence Estimation

AI summary

Overview

Research area: Natural language processing / large language model reliability — specifically uncertainty and confidence calibration through verbalized confidence.

Technical level: Intermediate. The paper combines fine-tuning methods, calibration metrics (ECE, NCE, Brier score, AUROC), and mechanistic probing techniques (Attention Rollout, Integrated Gradients), so familiarity with LLM evaluation basics helps.

Scope (1 sentence): The paper diagnoses why LLMs verbalize overconfidence — they largely ignore their own generated answer when stating confidence — and proposes a fine-tuning framework, ADVICE, that trains models to condition confidence on the answer.

What This Paper Is About

Large language models can state their confidence in natural language, but they tend to report high confidence regardless of whether their answer is actually correct. Prior work has focused on how to reduce this overconfidence rather than why it occurs. This paper identifies "answer-independence" — the failure to condition confidence on the model's own generated answer — as a primary cause, and builds a training framework called ADVICE (Answer-Dependent Verbalized Confidence Estimation) to correct it.

Key Contributions

1. Diagnosis of answer-independence. Through a distributional analysis of confidence conditioned on different answer candidates and through two attribution-based analyses (Attention Rollout and Integrated Gradients), the paper shows that verbalized confidence barely varies across different answers and that answer tokens receive little attribution.

2. The ADVICE framework. A lightweight fine-tuning method that trains the model on triplets of (question, correct answer, wrong answer) using four objectives: a language modeling loss (L_LM), a Jensen–Shannon divergence loss (L_JSD) that contrasts confidence distributions for correct versus wrong answers, a margin loss (L_Margin) that enforces the correct direction, and a sum loss (L_Sum) that enforces the confidence definition μ_correct + μ_wrong = 1.

3. Strong calibration and generalization. ADVICE outperforms Default, Prompting, and Self-Consistency baselines on the training-domain dataset (TriviaQA) and generalizes to out-of-distribution datasets (MMLU, LogiQA) and to three unseen verbalization formats. It outperforms ConfTuner in 19 of 24 cases (2 datasets × 4 metrics × 3 models) and yields orthogonal gains when combined with ConfTuner.

4. Causal validation. Post-hoc experiments — masking answer tokens, re-running Attention Rollout, and tracking Integrated Gradients attribution across training steps — show that the calibration gains come from increased answer dependence, not from improved task accuracy.

Main Findings

• Verbalized confidence is nearly answer-independent. In the Jensen–Shannon divergence analysis over confidence distributions conditioned on different answers (using the threshold τ = 0.1 defined in prior work as indicating similar distributions), JSD scores concentrate near zero with long right tails, indicating that confidence estimates change little across different answers. This was tested on TriviaQA, MMLU, and LogiQA with Gemma-2-9b-it and Llama-3.1-8B-Instruct; for TriviaQA, m = 30 sampled answers were used, and m = 4 for multiple-choice datasets based on pre-defined distractors.

• Attribution analyses agree. Attention Rollout scores for the Confidence-to-Answer direction (C→A) are significantly lower than for Answer-to-Question (A→Q) and Confidence-to-Question (C→Q). Integrated Gradients shows answer tokens are consistently under-weighted compared to tokens such as "user" and the BOS token.

• ADVICE improves calibration on the training domain. On TriviaQA with Llama-3.1-8B-Instruct, ADVICE achieves ECE 10.4 versus 16.9 for Default and 12.1 for Prompting, with AUROC 77.0. With Gemma-2-9b-it, ADVICE reaches ECE 6.2 and AUROC 77.4, versus Default ECE 21.9 and AUROC 52.7. ADVICE is comparable to ConfTuner (Llama-3.1 TriviaQA ECE 5.2; Gemma-2 ECE 5.7) rather than always better.

• Out-of-distribution generalization. On MMLU and LogiQA, ADVICE outperforms ConfTuner in 19 of 24 cases. For Gemma-2-9b-it, ADVICE achieves MMLU ECE 5.6 and LogiQA ECE 11.9, versus ConfTuner's 11.0 and 18.4. For Mistral-7B-Instruct-v0.3, ADVICE's LogiQA ECE is 35.6, which is worse than ConfTuner's 24.8, though the combined setting reaches 24.0.

• Orthogonal gains with ConfTuner. Combining ADVICE with ConfTuner often improves results further, for example Mistral-7B on TriviaQA (ECE 6.9, |NCE| 4.7, AUROC 76.4) and LogiQA ECE 24.0.

• Generalization across verbalization types. Trained only on ScoreLetter and ScoreNumber, ADVICE also improves performance on the unseen ScoreText, ScoreFloat, and ScorePercent formats. For Gemma-2-9b-it under ScorePercent, TriviaQA ECE drops from 26.9 (Default) to 6.7 (ADVICE).

• Best efficiency trade-off. In the ECE-versus-token-usage plot, ADVICE clusters in the lower-left (better calibration, fewer generated tokens), while Self-Consistency dramatically increases token usage without satisfactory performance and ConfTuner typically incurs higher token usage.

• Ablation shows each loss component matters. Using L_LM alone gives poor calibration; L_JSD and L_Margin work well together (directional separation), and L_Sum improves out-of-distribution calibration. The full ADVICE combination gives the best overall balance.

• Task performance is preserved. QA accuracy changes are negligible after fine-tuning (for example, TriviaQA with Llama-3.1-8B-Instruct under ScoreLetter: 75.2 Default vs. 78.1 ADVICE). This indicates ECE reductions come from calibration, not accuracy gains.

• Answer masking confirms the mechanism. When answer tokens are replaced with equal-length padding tokens, Default remains overconfident while ADVICE shifts probability mass toward low-confidence expressions (e.g., E, 0, and 1).

• Attribution shifts during training. Tracking the top-10 attributed tokens for Gemma-2-9b-it across training steps 0, 100, 200, 300, 400, and 500, the answer token ("_Exile") rises in rank, and Attention Rollout distributions for C→A improve relative to Default (t-test confirms statistical significance).

Methodology in Plain English

The researchers first checked whether confidence actually depends on the answer. They asked models to answer questions, generated many alternative answers per question (via top-p sampling), and measured how similar the resulting confidence distributions were across those answers. If confidence truly reflects the answer, distributions for different answers should diverge; instead they clustered together. They then looked inside the model using Attention Rollout (which traces how attention flows between the question, answer, and confidence portions of the prompt) and Integrated Gradients (which assigns importance to individual input tokens using gradients). Both pointed to the answer being largely ignored.

To fix this, they fine-tuned models on TriviaQA. They sampled 4,000 instances from the training split and kept only those where the model answered correctly under greedy decoding. For each question, they paired the correct answer with a wrong answer drawn from the model's stochastic decoding output, producing triplets (q, a_correct, a_wrong). Stochastic decoding tends to produce "hard negatives" — plausible but wrong answers — which sharpens the required distinction. Each instance was built in two confidence-format variants.

Training used four objectives. L_LM preserves general question-answering ability via negative log-likelihood on the correct answer. L_JSD pushes the confidence distributions for correct and wrong answers apart. L_Margin ensures the separation goes in the right direction (higher confidence for correct answers). L_Sum enforces that the expected confidence for correct and wrong answers sums to 1, matching the idea that confidence should approximate the probability of being correct. All loss weights (λ_LM, λ_JSD, λ_Margin, λ_Sum) were set to 1. Hyperparameters δ_JSD and δ_Margin control how strongly correct and wrong answers are separated. Evaluation used three open-weight models (Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Gemma-2-9b-it), three datasets (TriviaQA in-domain; MMLU and LogiQA out-of-distribution), and four calibration metrics (ECE, |NCE|, Brier score, AUROC), averaged over two seen verbalization types and three random seeds.

Why This Matters

The paper reframes overconfidence in verbalized confidence as a structural failure — the confidence statement is nearly disconnected from the answer it is supposed to describe — and demonstrates that explicitly training for answer dependence fixes a large part of the problem without hurting task performance.

Research impact: It provides a mechanism-level explanation for a widely reported phenomenon and offers a diagnostic procedure (answer-conditioned JSD distributions plus attribution probing) that other researchers can reuse to analyze confidence estimation methods.

Real-world applications:

• Medical and clinical decision support, where a model's stated confidence must track whether its specific recommendation is trustworthy.
• Legal and regulatory workflows, mentioned in the paper as high-stakes domains, where unreliable confidence signals hinder dependable use.
• Customer-facing question answering and search assistants, where calibrated confidence lets a system escalate to a human when uncertain.
• Multi-step or tool-augmented pipelines, where downstream components need a reliable confidence signal to decide whether to accept a model's answer or re-query.

Industry relevance: The method is lightweight, requires no modification at inference beyond standard fine-tuning, works with multiple confidence formats (letters, numbers, text, floats, percentages), generalizes to out-of-distribution data, and adds minimal token overhead compared to sampling-based approaches such as Self-Consistency. That combination makes it practical for production calibration where cost matters.

Future Directions

• Reducing data construction cost. ADVICE requires LLM-generated answers to form contrastive pairs, which adds data construction overhead; the authors consider the trade-off reasonable but flag it as a limitation.
• Extending beyond short-form QA and multiple-choice questions. Applying the approach to long-context understanding and complex reasoning tasks is identified as a valuable next step.
• Rigorous evaluation under high accuracy regimes. Because calibration is coupled with task accuracy — on SciQ, where models exceed 90% accuracy, the base model attains the best calibration (Table 7 in the Appendix) — the authors call for more careful evaluation practices in the literature.
• Combining with other calibration families. The observed orthogonal gains when combining ADVICE with ConfTuner raise the question of how far stacking complementary calibration methods can go.

Target Audience

Researchers and engineers working on LLM reliability, uncertainty quantification, and mechanistic interpretability; practitioners deploying LLMs in domains where stated confidence drives decisions (healthcare, law, customer support); and anyone comparing fine-tuning-based versus prompting-based or sampling-based confidence calibration approaches. Readers without a background in calibration metrics or probing methods will find the high-level argument accessible, but the experimental sections require intermediate familiarity.

Authors’ abstract

Recent progress in large language models (LLMs) has enabled them to communicate their confidence in natural language, improving transparency and reliability. However, this expressiveness is often accompanied by systematic overconfidence, whose underlying causes remain poorly understood. In this work, we analyze the dynamics of verbalized confidence estimation and identify answer-independence -- the failure to condition confidence on the model's own answer -- as a primary driver of this behavior. To address this, we introduce ADVICE (Answer-Dependent Verbalized Confidence Estimation), a fine-tuning framework that promotes answer-grounded confidence estimation. Extensive experiments show that ADVICE substantially improves confidence calibration, while exhibiting strong generalization to unseen settings without degrading task performance. We further demonstrate that these gains stem from enhanced answer dependence, shedding light on the origins of overconfidence and enabling trustworthy confidence verbalization.