Hallucinate Less by Thinking More: Aspect-Based Causal Abstention for Large Language Models

AI summary

Overview

Research area: Natural Language Processing — hallucination mitigation and abstention (selective answering) in Large Language Models, using causal inference over internal parametric knowledge.

Technical level: Advanced. The paper builds on Structural Causal Models (Pearl 2009), the do-operator, back-door adjustment, the disjunctive cause criterion, and Augmented Inverse Probability Weighting (AIPW) estimation (Funk et al. 2011), combined with LLM agentic prompting and embedding-geometry analysis.

Scope: The paper proposes ABCA (Aspect-Based Causal Abstention), a two-stage framework that discovers interpretable "aspects" of a model's internal knowledge and estimates aspect-conditioned causal effects in order to abstain before generating an answer.

What This Paper Is About

LLMs frequently produce fluent but factually incorrect outputs, and the common safeguard — abstaining with phrases such as "I don't know" — is usually triggered only after generation via signals like output variation, confidence, or feedback. That timing means unreliable answers cannot be prevented in advance, and methods may either abstain unnecessarily when rare-but-correct knowledge exists, or fail to abstain when conflicting internal knowledge is hidden.

ABCA's goal is to move abstention earlier, to the pre-generation stage, by treating the model's parametric knowledge as multifaceted and organised along distinct aspects (disciplines, legal contexts, temporal frames, and so on). The paper illustrates the motivation with the question "Who is the bell ringer of Notre Dame?" — a TruthfulQA-style question with the ground-truth answer "I have no comment" — which GPT-4.5, Gemini Pro 2.5, and Claude Sonnet 4 confidently answer with "Quasimodo" because less dominant but relevant knowledge is ignored.

Key Contributions

1. ABCA framework. A framework that addresses the oversight of knowledge heterogeneity in existing post-hoc abstention methods by modelling how different aspects influence knowledge activation and decision reliability.

2. A causally principled abstention policy. A formal policy that distinguishes knowledge conflict (Type-1 Abstention), knowledge insufficiency (Type-2 Abstention), and knowledge consistency (Aggregation) through agent-aided exploration of parametric knowledge and aspect-conditioned causal inference.

3. Empirical validation on four datasets (TruthfulQA, KUQ, AVeriTeC, and AbstenQA/MMLU), reported as achieving state-of-the-art performance, enhancing answering ability without unnecessary abstention, and supporting interpretable abstention decisions.

4. Reusable artefacts. Code and appendix released at https://github.com/vnht/abca.

Main Findings

• State-of-the-art accuracy on three benchmarks. With GPT-4.1, ABCA surpasses CFMAD by 3.3 points on TruthfulQA, exceeds CausalAbstain by 2.7 points on KUQ, and gains 3.2 points on AVeriTeC.

• Large gains on abstention-specific metrics. ABCA reaches a U-Ac of 0.964 on TruthfulQA (versus 0.440 by CFMAD) and 0.876 on KUQ (versus 0.828 by LLM Collaboration), and consistently leads on U-F1.

• Balanced answering and abstaining. Methods such as CFMAD attain high A-Ac scores (for example 0.907 on TruthfulQA with GPT-4.1) but often underperform on abstention, while post-hoc detection methods like LLM Collaboration, Multilingual Feedback, and CausalAbstain offer limited accuracy gains over Zero-shot and Self-Consistency on answerable questions.

• Consistency across backbones. ABCA shows advantages across GPT-4.1, LLAMA 3.3 70B, and MISTRAL-NEMO 12B on the factual datasets, with the KUQ accuracy gain over CausalAbstain stable across models. On AbstenQA (MMLU academic questions requiring logical reasoning), ABCA performs competitively with leading methods.

• Dual-agent aspect discovery is validated. Rated on a [1–10] scale by GPT-o3 and Gemini-Pro against the validity criteria C_val, ABCA's aspects score (7.4, 8.7, 7.9) on TruthfulQA, (8.7, 8.1, 8.3) on KUQ, (8.5, 8.5, 8.2) on AVeriTeC, and (8.4, 8.3, 8.9) on AbstenQA, for dimensional consistency, temporal precedence, and factual grounding respectively. The 1-Agent and Lite configurations score lower on the same tuples. Error analysis in Appendix B.7 shows higher error rates align with lower validity scores.

• Higher generation diversity. NLI Diversity scores (Stasaski and Hearst 2022, scored with RoBERTa) show ABCA elicits more diverse chains of thought than Self-Consistency. With GPT-4.1: 0.65 (+0.26) on TruthfulQA, 0.62 (+0.24) on KUQ, 0.64 (+0.39) on AVeriTeC, 0.59 (+0.38) on AbstenQA. With LLAMA 3.3 70B: 0.48 (+0.34), 0.46 (+0.31), 0.47 (+0.24), 0.45 (+0.23).

• No universal golden set of aspects. Applying BERTopic to aspects for correct outputs, only 46%, 40%, 18%, and 41% of questions in TruthfulQA, KUQ, AVeriTeC, and AbstenQA respectively show over 70% topic overlap between GPT-4.1 and LLAMA, indicating different models rely on distinct but valid aspects.

• Better abstention informativeness. On a [1–100] scale evaluated by GPT-o3 and Gemini-Pro, ABCA scores 85.45 (All) and 85.41 (Abs) on TruthfulQA, 79.56/74.68 on KUQ, 86.45/84.23 on AVeriTeC, and 81.53/75.39 on AbstenQA — well above LLM Collaboration (for example 78.25/45.85 on TruthfulQA) and CausalAbstain (75.44/49.57 on TruthfulQA).

• Confusion between abstention types remains. Among correct abstention cases on AVeriTeC, 14.3% of claims involving conflicting evidence are identified as Type-2, while 18.7% of those related to insufficient evidence are labelled as Type-1.

• All ablated variants underperform. Removing aspect conditioning (No-X), using single-agent discovery (1-Agent), uniform aspect weights (Uniform-w), uniform effects (Uniform-τ), limiting iteration and sampling to T = K = N = 1 (Lite), pooling all CoTs (Collapsed-X), and fixing aspects to three languages (Fixed-X, English/French/German) all score below full ABCA with GPT-4.1.

Methodology in Plain English

The causal model. The authors extend the standard reasoning picture — query Q leads to chain-of-thought C, which leads to answer A — by adding an aspect variable X that activates distinct branches of the model's parametric memory. Unobserved confounders U (pre-training biases, memorised artefacts) can create spurious query–answer associations. Conditioning on X is shown to block all back-door paths, making the aspect-conditioned causal effect identifiable from observational data.

Validity guardrails. Aspect conditioning is only allowed to use variables satisfying criteria C_val: dimensional consistency (aspects operate on the same outcome scale so strata can be aggregated), temporal precedence (aspects precede Q, avoiding post-treatment bias), and factual grounding (aspects push toward evidence-based knowledge). Aggregation across aspects is governed by C_agg: structural invariance of the Q → C → A mechanism, prevalence validity (aspect-aware weights rather than equal contributions), and directional coherence (estimated effects must not conflict — the condition the abstention policy is designed to police).

Stage 1 — Aspect Discovery. A two-agent dialogue identifies X, its constituent aspects {x_i}, and weights {w_i}. The DAgent proposes conditioning dimensions; the CAgent prunes those violating temporal precedence or factual grounding. The top-ranked dimension becomes X, DAgent stratifies X into aspects, CAgent validates each, and the agents reconcile aspect-level weights until convergence.

Stage 2 — Aspect Resolution. For each aspect x_i, the framework generates K candidate CoTs, samples N answers, estimates the empirical mediator distribution p̂(c_j|x_i) and an outcome regression μ̂(c_j|x_i) (log-probability for categorical generations; normalised weighted geometric mean of log-probabilities for open-ended generations, to avoid length bias), and combines them in an AIPW estimator τ̂(x_i). AIPW is chosen because it remains consistent if either the mediator distribution or the outcome model is correctly specified — useful where assumptions in black-box settings cannot be verified.

Abstention policy. Each aspect yields a representative answer whose normalised vector representation e_i is weighted by a significance score α_i = w_i · τ̂(x_i). The causally weighted centroid c is computed and normalised; angular deviations θ_i = arccos(e_i · c) are aggregated into a Centroid Angular Deviation (CAD) score. A three-way gate follows: if CAD > θ_max, Type-1 Abstention (knowledge conflict); if 1 − (c · e_null) ≤ ρ_null, where e_null is a precomputed null-consensus embedding such as "I don't know" or "No data", Type-2 Abstention (knowledge insufficiency); otherwise Aggregation, which prioritises high-significance aspects and records high-deviation low-significance aspects as acknowledged caveats.

Evaluation. Comparisons are against Zero-shot, Self-Consistency, SelfCheckGPT, Multilingual Feedback, LLMs Collaboration, CFMAD, and the causal abstention method CausalAbstain, across GPT-4.1, LLAMA 3.3 70B, and MISTRAL-NEMO 12B, using the confusion-matrix formulation from Madhusudhan et al. 2025. Metrics are Acc, A-Ac, U-Ac, A-F1, and U-F1. The paper also reports an evaluation on AbstentionBench (Kirichenko et al. 2025, Meta), covering Answer Unknown, False Premise, Subjective, Underspecified Context, and Underspecified Intent. Dataset sizes and statistics are referenced to Appendix B.3 and are not reported in the content available here.

Why This Matters

Impact on research. The work reframes abstention as a pre-generation problem grounded in causal inference over internal knowledge, rather than a post-hoc filter over outputs. It extends the Q → C → A structural causal model with a conditioning variable X, and it supplies a principled reason — identifiability, validity criteria, collapsibility — for treating aspects as causal interventions rather than merely as prompt variations. It also distinguishes two failure modes that prior work tends to conflate: conflicting knowledge and insufficient knowledge.

Real-world applications.

• Fact-checking and claim verification. AVeriTeC-style claims can be routed to Supported, Refuted, Not Enough Evidence, or Conflicting Evidence outcomes, with an explanation of which aspects conflict.
• High-stakes question answering. Medical, legal, or regulatory queries where an abstention (for example, on Vitamin D and COVID-19 guidance) is preferable to a confident but unsupported answer.
• Enterprise and API-based deployments. The method is black-box and does not require access to model internals, so it can be applied to proprietary and closed-source systems.
• Auditing and compliance. Because aspects are interpretable and their causal effects are estimated, abstention decisions come with an auditable rationale rather than an opaque confidence score.

Industry relevance. Organisations deploying LLMs through vendor APIs face a choice between over-answering (hallucination risk) and over-abstaining (usability loss). ABCA reports improving both A-Ac and U-Ac simultaneously, and its complexity analysis (Appendix B.8) reports that it uses computational resources more efficiently than baselines under equivalent budgets — a practical consideration for any deployment with cost constraints.

Future Directions

• Finer-grained aspect representations. The authors explicitly name this as future work, motivated by the finding that only 46%, 40%, 18%, and 41% of questions across the four datasets show over 70% topic overlap between GPT-4.1 and LLAMA.

• Non-linear aggregation and abstention policies. Also named as future work; the current CAD-based centroid method is a linear aggregation subject to the collapsibility requirements of C_agg.

• Better separation of conflict from insufficiency. The reported 14.3% and 18.7% cross-type misclassifications on AVeriTeC point to overlap between the two abstention triggers when small variations in causal-effect estimates are read as genuine disagreement.

• Addressing spurious facts. The error analysis identifies spurious facts as the dominant failure mode, which the authors describe as a fundamental limitation in LLM knowledge that aspect conditioning alone does not resolve.

Target Audience

Researchers and engineers working on hallucination mitigation, selective prediction, and uncertainty quantification in LLMs; practitioners applying causal inference to model reasoning; and teams deploying black-box or proprietary LLMs in fact-checking, medical, legal, or other high-stakes question-answering settings where knowing when not to answer is as important as answering correctly. Readers without a background in structural causal models, the do-operator, and inverse probability weighting will need to consult Appendix A, as the methodology section assumes fluency in those concepts.

Authors’ abstract

Large Language Models (LLMs) often produce fluent but factually incorrect responses, a phenomenon known as hallucination. Abstention, where the model chooses not to answer and instead outputs phrases such as "I don't know", is a common safeguard. However, existing abstention methods typically rely on post-generation signals, such as generation variations or feedback, which limits their ability to prevent unreliable responses in advance. In this paper, we introduce Aspect-Based Causal Abstention (ABCA), a new framework that enables early abstention by analysing the internal diversity of LLM knowledge through causal inference. This diversity reflects the multifaceted nature of parametric knowledge acquired from various sources, representing diverse aspects such as disciplines, legal contexts, or temporal frames. ABCA estimates causal effects conditioned on these aspects to assess the reliability of knowledge relevant to a given query. Based on these estimates, we enable two types of abstention: Type-1, where aspect effects are inconsistent (knowledge conflict), and Type-2, where aspect effects consistently support abstention (knowledge insufficiency). Experiments on standard benchmarks demonstrate that ABCA improves abstention reliability, achieves state-of-the-art performance, and enhances the interpretability of abstention decisions.