Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistency

AI summary

Overview

Research area: Natural Language Processing / LLM truthfulness, uncertainty estimation, and knowledge robustness.

Technical level: Intermediate — the paper uses Bayesian-inspired formalization and requires familiarity with LLM evaluation concepts like self-consistency, but the core intuition is accessible.

Scope: This paper proposes a structural metric (Neighbor-Consistency Belief) and a stress-testing protocol to diagnose whether an LLM's correct answers reflect genuine, robust belief or brittle memorization, and introduces a training method to strengthen belief robustness.

What This Paper Is About

Existing ways of measuring whether an LLM "knows" a fact rely on point-wise confidence, such as how consistently the model answers the same question across repeated samples. The authors show that this is misleading: facts answered with perfect self-consistency can still collapse when the model is exposed to mild contextual pressure, like peer consensus or authoritative-sounding documents. The paper's goal is to define a structural measure of belief robustness, validate it through cognitive stress tests, and show it can be improved through training.

Key Contributions

1. Neighbor-Consistency Belief (NCB) — A new metric that estimates belief robustness by measuring whether a model stays coherent across a fact's conceptual neighborhood (entity prerequisites, logical implications, thematic associations), rather than just on the target question alone.

2. Cognitive stress-testing protocol — Two psychologically inspired evaluation setups: Peer Quantity (simulating Asch-style social pressure from multiple agents) and Source Credibility (testing authority bias via low/medium/high credibility misleading sources), each with Conflict and Misleading interference modes.

3. Neighbor-Enriched Dataset — A 2,000-sample, four-domain dataset (STEM, Arts & Culture, Social Sciences, Sports) where each target fact is embedded with ~7.84 verified neighbor facts and ~4.88 misleading neighbor facts, all human-verified through a multi-stage pipeline.

4. Structure-Aware Training (SAT) — A knowledge distillation-based training strategy that optimizes context-invariant belief structure, reducing long-tail knowledge brittleness by roughly 30% compared to answer-based and knowledge-based augmentation baselines.

Main Findings

• Self-consistency creates an illusion of confidence: In a pilot study, 995 questions answered correctly with perfect self-consistency (SC = 1.0) dropped to 33.8% accuracy once contextual interference was introduced — a 66-point collapse.

• NCB reliably predicts robustness: When stratifying the High Self-Consistency set by NCB score, High-NCB groups consistently show smaller accuracy drops than Low-NCB groups across all four models tested. For Qwen3-Thinking, the top-35% NCB group dropped only 11.3% versus 22.6% for the bottom group.

• Structured beliefs resist escalating interference: Under Peer Quantity Conflict, Low-NCB performance degraded from 76% to 60% as opposing voices accumulated, while High-NCB held roughly 90% to 80%.

• A single dissenting voice matters: Consistent with Asch's conformity experiments, the presence of one truth-teller among distractors substantially improved accuracy in both High and Low NCB groups.

• Chain-of-Thought is unstable; Reflection helps: CoT sometimes amplified degradation (e.g., Low-NCB-35% Qwen-2.5 Quantity Stress: drop increased from 25.7% to 31.6%). Multi-turn Reflection consistently reduced drop rates in nearly every setting.

• Reasoning models abstain rather than guess: Qwen3-Thinking selectively refuses to answer on Low-NCB samples, suggesting a conservative strategy when belief structure is missing.

• Scaling does not close the gap: Across the Qwen2.5 series (1.5B to 72B), High-NCB was always more robust than Low-NCB, with no clear trend of the gap narrowing with scale.

• SAT substantially reduces brittleness: On newly learned facts, SAT reached 93.0% base accuracy and 60.6% average stress-test performance, versus 33.4% for the best baseline (Knowledge-Based Augmentation) — roughly a 30% reduction in degradation — without harming generic capabilities (MMLU, GSM8K).

Methodology in Plain English

The researchers start from a cognitive-science intuition: humans hold knowledge as interconnected networks, not isolated facts. Accepting "Marcelo Viana was IMU Vice-President" should be reinforced by knowing about his broader academic career. They formalize this with a Bayesian-inspired argument: if a model consistently answers a target question and its related neighbor questions correctly, the posterior probability that its behavior stems from a structured (rather than unstructured/memorized) belief state is much higher.

To make this computable, they define NCB as the empirical correctness frequency on the target question multiplied by the geometric mean of correctness frequencies across neighbor questions (the geometric mean prevents the score from collapsing exponentially as neighbors are added).

They then build a dataset in which every fact is surrounded by a verified neighborhood and a plausible-but-wrong distractor with its own true supporting facts. They subject models to two interference protocols borrowed from psychology — social pressure from peer agents (Asch) and authority bias from high-credibility sources — and compare top-percentile versus bottom-percentile NCB groups, all drawn from the High Self-Consistency pool so that standard metrics would see them as equally "known."

Finally, they propose SAT: a frozen teacher model provides a reference answer distribution for the target fact, and a student is trained to match that distribution across both neighbor-containing and general/noisy contexts using KL divergence — forcing the model to produce the same answer regardless of surrounding noise.

Why This Matters

Impact on research: The paper challenges the widespread use of self-consistency as a proxy for knowledge reliability and argues that belief should be evaluated structurally. It also reframes context-invariance as a trainable property rather than an emergent one, with direct implications for knowledge editing, uncertainty quantification, and continual learning.

Real-world applications:

• Retrieval-augmented generation (RAG): Detecting when a model's answer is vulnerable to conflicting retrieved documents, enabling better grounding and refusal behavior.
• Multi-agent LLM systems: Predicting which facts will collapse when agent peers converge on incorrect answers.
• High-stakes domains (healthcare, law, science): Filtering out "confidently wrong" outputs before they reach decision-makers.
• Continual knowledge updating: Separating genuine knowledge revision from harmless contextual noise when incorporating new information.

Industry relevance: Any deployment that pairs LLMs with external context — search assistants, enterprise copilots, customer-facing agents — can use NCB as an audit signal. SAT offers a lightweight post-training recipe for reducing brittleness of newly injected facts, relevant to organizations fine-tuning models on proprietary knowledge.

Future Directions

• Richer relation types: The current framework covers only entity prerequisites, logical implications, and thematic associations. Causal chains, hierarchical taxonomies, and multi-hop relations remain unexplored and would require domain-specific resources.

• Dynamic and temporal knowledge: The paper deliberately restricts itself to time-invariant facts. Extending NCB to knowledge that genuinely changes over time (e.g., current office-holders) is an open problem, since it requires distinguishing legitimate updates from noise.

• Human alignment validation: The authors acknowledge that NCB has not been compared against human judgments of "genuine understanding." Future work could test whether high-NCB behavior aligns with what humans consider real comprehension.

• Scalability and efficiency: Constructing belief neighborhoods is computationally expensive at both training and inference time. Optimizing this pipeline is necessary before practical large-scale deployment.

• Agent-level consequences: Whether NCB scores predict downstream task success, decision reliability, and out-of-distribution detection in agentic settings remains to be tested.

Target Audience

This paper is most valuable to:

• LLM evaluation and reliability researchers working on uncertainty quantification, hallucination detection, and truthfulness benchmarks.
• Machine learning engineers deploying RAG pipelines, multi-agent systems, or fine-tuned models who need to assess whether learned knowledge will survive contact with noisy context.
• AI safety and alignment researchers interested in belief stability, sycophancy, and resistance to misleading authority or peer pressure.
• Cognitive scientists and NLP researchers exploring parallels between human belief structures and model internals.

Readers should be comfortable with standard NLP evaluation metrics and have some familiarity with Bayesian reasoning and knowledge distillation concepts to fully absorb the formalization and training sections.

Authors’ abstract

As Large Language Models (LLMs) are increasingly deployed in real-world settings, correctness alone is insufficient. Reliable deployment requires maintaining truthful beliefs under contextual perturbations. Existing evaluations largely rely on point-wise confidence like Self-Consistency, which can mask brittle belief. We show that even facts answered with perfect self-consistency can rapidly collapse under mild contextual interference. To address this gap, we propose Neighbor-Consistency Belief (NCB), a structural measure of belief robustness that evaluates response coherence across a conceptual neighborhood. To validate the efficiency of NCB, we introduce a new cognitive stress-testing protocol that probes outputs stability under contextual interference. Experiments across multiple LLMs show that the performance of high-NCB data is relatively more resistant to interference. Finally, we present Structure-Aware Training (SAT), which optimizes context-invariant belief structure and reduces long-tail knowledge brittleness by approximately 30%. Code is available at https://github.com/zjunlp/belief.