Mapping from Meaning: Addressing the Miscalibration of Prompt-Sensitive Language Models

AI summary

Overview

• Research area: Natural language processing, uncertainty quantification, and LLM calibration. The paper sits at the intersection of model evaluation, prompt sensitivity research, and semantic uncertainty metrics.
• Technical level: Intermediate. The core intuition is accessible, but the paper assumes familiarity with uncertainty decomposition (epistemic vs. aleatoric), semantic entropy, and embedding-based metrics. The ergodic-theory motivation is optional reading for the main results.
• Scope: The paper diagnoses prompt sensitivity as a form of overfitting to token sequences, proposes paraphrase-based sampling to recover calibration, and introduces a variance-based uncertainty decomposition that quantifies how much of a model's uncertainty comes from phrasing versus sampling noise.

What This Paper Is About

Large language models often give different answers—and different confidence distributions—when asked the same question in different words, even when those wordings mean exactly the same thing. This means a model's stated uncertainty reflects the particular phrasing it saw rather than the underlying meaning of the question. The authors frame this as overfitting to token sequences, then show that averaging over many paraphrases recovers well-calibrated uncertainty without sacrificing accuracy, and introduce a metric that measures exactly how prompt-sensitive a model is.

Key Contributions

1. A semantic-invariant perturbation framework. The authors reformulate prompt sensitivity as a generalization error over "semantic concept space," and propose sampling answers across multiple paraphrases of the same question rather than repeatedly sampling a single phrasing. They justify the averaging procedure with an ergodic-theoretic argument (a Birkhoff ergodic theorem analogy) showing that sampling across token representations converges to the response distribution over meaning.

2. An embedding-variance uncertainty metric with additive decomposition. Unlike entropy-based metrics that require discretizing outputs into semantic classes, their metric embeds each generated response, computes the covariance matrix over embeddings, and takes its trace. This total variance decomposes additively into epistemic variance (differences across paraphrases) and aleatoric variance (spread within a single paraphrase).

3. A prompt sensitivity ratio (ρ_u). The ratio of epistemic to total uncertainty quantifies how much of a model's uncertainty is attributable to phrasing rather than to genuine sampling noise. The paper shows this ratio has a theoretical baseline of 1/n_s (where n_s is the number of samples per paraphrase) for a perfectly generalizing model, allowing direct measurement of departure from perfect generalization.

4. Empirical evidence that RLHF post-training induces prompt sensitivity. Comparing Llama 2-Base against Llama 2-Chat reveals that the base model is well-calibrated at individual paraphrases and barely benefits from perturbation averaging, while the RLHF-tuned chat model is substantially more prompt-sensitive and gains large calibration improvements from paraphrase sampling.

Main Findings

• Paraphrasing is the strongest perturbation method. Among meaning-preserving perturbations tested (paraphrasing, dummy tokens, system messages), paraphrasing produced the largest calibration gains. Sampling six different paraphrases once each outperformed sampling a single paraphrase six times.

• Calibration gains come without accuracy loss. The authors show that temperature-based perturbation improves calibration only by degrading accuracy—high temperature "smooths" distributions and makes models systematically underconfident. Semantic-invariant perturbations instead yield Pareto improvements, raising AUROC while holding accuracy steady.

• Embedding variance beats entropy-based and graph-based baselines. On TriviaQA with GPT-3.5, the total variance metric reached AUROC 86.4 at (n_p=3, n_s=4) versus 79.5 for entropy, 82.1 for semantic entropy, 83.4 for lexical similarity, and 85.7 for the eigenvalue-based affinity-graph metric. On Natural Questions at (6,2) it reached 72.0 versus 54.5 for entropy and 70.9 for the eigenvalue metric.

• More paraphrases shift calibration toward epistemic uncertainty. As the number of perturbations increases relative to samples per perturbation, the epistemic component contributes more to calibration and the aleatoric component contributes less. A sample from a new paraphrase carries more marginal information than an additional sample of the same paraphrase.

• Base models generalize better than RLHF-tuned models. Llama 2-Base calibration barely moved across (n_p, n_s) settings on TriviaQA (87.5 to 88.1) and NQ (77.8 to 78.3), while Llama 2-Chat improved markedly with more paraphrases (TriviaQA 79.2 to 83.6; NQ 68.1 to 73.1).

• ρ_u separates base and chat models. The prompt sensitivity ratio was consistently higher for Llama 2-Chat than Llama 2-Base at every (n_p, n_s) configuration, and both exceeded the 1/n_s theoretical baseline for a perfectly generalizing model. The ratio for chat models grows faster as samples per paraphrase increase, indicating genuine phrasing-driven variance rather than sampling noise.

Methodology in Plain English

The researchers take a set of QA questions (1,000 each from TriviaQA and Natural Questions) and, for each question, generate several paraphrases—different wordings that ask the same thing. For example, "Who was the British Prime Minister after Arthur Balfour?" becomes "Who succeeded Arthur Balfour as Prime Minister of Britain?" and similar variants. They then draw a fixed budget of samples from the LLM, split between the number of paraphrases (n_p) and the number of samples per paraphrase (n_s), and pool all responses into one aggregate answer distribution.

To measure uncertainty, they embed each generated answer into a vector space (using eigenvectors of a graph Laplacian over response similarities) and compute how spread out those embeddings are. This total spread splits cleanly into two pieces: the average spread within each paraphrase (aleatoric, or sampling noise) and the spread of the paraphrase-level averages around the overall mean (epistemic, or phrasing sensitivity).

They evaluate calibration by checking whether the uncertainty score can distinguish correct from incorrect answers—measured as AUROC, where GPT-3.5 Turbo serves as an automated grader. They compare against entropy, semantic entropy, lexical similarity, and an eigenvalue-based affinity-graph metric, and run everything on GPT-3.5, Llama 2-Base (7B), and Llama 2-Chat (7B), repeating each experiment five times.

Why This Matters

• Impact on research: The paper reframes prompt sensitivity not as a curiosity but as a measurable generalization failure, and provides a decomposable metric that makes the phrasing-vs-sampling distinction quantitative. It also complicates the assumption that RLHF improves models across the board, suggesting post-training can induce post-hoc overfitting to surface patterns.

• Real-world applications:

  • Uncertainty-aware question answering, where a system should signal low confidence only when it genuinely lacks knowledge, not because the user phrased a question unusually.
  • Medical, legal, and financial advisory tools, where miscalibrated confidence under paraphrase could systematically mislead users who phrase questions in unexpected ways.
  • Hallucination detection and abstention policies, since calibration quality directly determines when a model should say "I don't know."
  • Evaluation and auditing of deployed models, where the prompt sensitivity ratio could serve as a cheap diagnostic for whether a vendor's model is robust to input variation.

• Industry relevance: Any organization deploying RLHF-tuned models behind a user-facing interface faces paraphrase variance as a practical reliability problem. The perturbation framework requires only black-box API access, making it deployable without model weights, and the marginal-information finding gives a concrete sampling recipe: spread a fixed sample budget across paraphrases rather than concentrating it on one phrasing.

Future Directions

• Is prompt sensitivity structural or superficial? The authors explicitly ask whether the effect reflects a deeper change in how post-trained models encode and retrieve knowledge, or is an artifact of alignment-driven constraints on output distributions.
• Extending decomposition to white-box settings. The embedding variance metric uses externally computed embeddings; applying it with internal representations or attention weights could yield finer-grained attribution of uncertainty.
• Optimizing paraphrase selection. The paper generates paraphrases without evaluating their quality or diversity. Selecting paraphrases that maximally span the semantic concept space could improve calibration per sample.
• Broader task and modality coverage. Evaluation is limited to two factoid QA datasets and three models. Whether the findings hold for reasoning tasks, long-form generation, multilingual inputs, or vision-language models remains open.
• Reconciling with accuracy-preserving post-training. If RLHF induces prompt sensitivity, can alignment objectives be modified to preserve the base model's semantic generalization?

Target Audience

Researchers and practitioners in LLM evaluation, uncertainty quantification, and trustworthy AI will find the core contributions most useful—particularly those working on calibration, hallucination detection, or abstention mechanisms. Engineers deploying RLHF-tuned models in production benefit from the practical sampling recipe and the black-box prompt sensitivity diagnostic. Alignment researchers and those studying post-training effects will find the base-versus-chat comparison directly relevant. Readers without a background in uncertainty decomposition will need to work through Section 4 to follow the metric design, but the experimental findings in Section 5 are readable independently.

Authors’ abstract

An interesting behavior in large language models (LLMs) is prompt sensitivity. When provided with different but semantically equivalent versions of the same prompt, models may produce very different distributions of answers. This suggests that the uncertainty reflected in a model's output distribution for one prompt may not reflect the model's uncertainty about the meaning of the prompt. We model prompt sensitivity as a type of generalization error, and show that sampling across the semantic ``concept space'' with paraphrasing perturbations improves uncertainty calibration without compromising accuracy. Additionally, we introduce a new metric for uncertainty decomposition in black-box LLMs that improves upon entropy-based decomposition by modeling semantic continuities in natural language generation. We show that this decomposition metric can be used to quantify how much LLM uncertainty is attributed to prompt sensitivity. Our work introduces a new way to improve uncertainty calibration in prompt-sensitive language models, and provides evidence that some LLMs fail to exhibit consistent general reasoning about the meanings of their inputs.