Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say "I Don't Know"

AI summary

Authors: Dhruv Madhwal, Lyuxin David Zhang, Dan Roth, Tomer Wolfson, Vivek Gupta (University of Pennsylvania; Arizona State University; Oracle AI) arXiv: 2602.04853v2 [cs.CL]

Overview

Research area: Natural Language Processing — closed-book multi-hop question answering, LLM reliability, uncertainty estimation, and abstention.

Technical level: Intermediate. The paper uses standard prompting terminology and well-known QA benchmarks, so a reader comfortable with how LLMs are prompted and evaluated can follow it without deep background in calibration theory.

Scope: An empirical study of how three task-equivalent prompting regimes (Direct, Assistive, Incremental) affect both accuracy and cross-regime agreement across nine LLMs and six multi-hop QA datasets, and a training-free abstention method built on the disagreement signal those regimes produce.

What This Paper Is About

Large language models answering factual questions without retrieval often hallucinate confidently when they lack the necessary knowledge, and they are poor at recognizing their own limits. Question decomposition is normally promoted as a way to improve accuracy, but this paper instead asks whether decomposing a question changes how reliable a model is. The authors show that for frontier models decomposition stops helping accuracy, yet the disagreement (or agreement) between a direct answer and a decomposed answer becomes a strong, usable signal for deciding when the model should say "I don't know."

Key Contributions

1. A comprehensive empirical analysis of how factual accuracy and cross-prompt consistency jointly vary across LLM scales, evaluation tasks, and prompting methods in closed-book multi-hop QA, comparing Direct prompting against Assistive and Incremental decomposed prompting, spanning nine LLMs and six benchmarks (1,433 verified instances after filtering).

2. The Reliability Multiplier (RM) — a quantitative measure of how much more often correct Direct answers occur among cross-regime-consistent cases than among inconsistent ones — used to characterize how the diagnostic value of consistency scales with model size.

3. Disagreement-Based Abstention (DBA) — a simple, training-free abstention method that treats cross-prompt disagreement between Direct and decomposed prompting as a reliability signal for deciding when closed-book answers should be trusted, requiring no retrieval, fine-tuning, auxiliary classifiers, or confidence heads.

4. A controlled evaluation setup using a fixed, manually verified gold-standard decomposition expressed in a domain-specific language (DSL), which decouples planning from execution so that Direct, Assistive, and Incremental differ only in how they execute the same semantically equivalent plan.

Main Findings

• Accuracy and consistency both rise with model scale, but inconsistency never disappears. Larger models are more likely to answer correctly and to agree across regimes. Even so, cross-regime consistency on MuSiQue reaches only 59.7% for the most accurate models evaluated. The authors frame this as a failure of logical invariance: semantically equivalent queries should yield identical outputs regardless of prompting regime.

• Decomposition helps non-frontier models and plateaus for frontier models. For models at or below ~70B parameters, decomposition acts as a reasoning scaffold with double-digit accuracy gains — for example, Qwen-72B gains +26.8% on Bamboogle. For Gemini-2.5 and GPT-5.1, gains fall to near-parity or become negative, which the authors attribute to a "ceiling effect" where frontier models have internalized the necessary reasoning chains.

• Cross-regime agreement is a strong accuracy indicator. Accuracy and cross-regime consistency are tightly coupled, with Pearson correlation coefficients reaching up to r = 0.98. The Reliability Multiplier grows steadily with capacity, typically reaching between 2x and 10x in the 30B to 70B range and exceeding 50x for Gemini Pro in the authors' measurements. Smaller (~8B) models often show near-one or inverted RM values, which the authors describe as a floor effect where correct answers are too sparse for RM to be stable.

• DBA outperforms standard uncertainty baselines as an error detector. DBA beats the AYS baseline in F1 in 17 of the 18 evaluated model-dataset pairs. For GPT-5.1, DBA-A yields F1 improvements ranging from +18 to +28 points across Bamboogle, HotpotQA, and MuSiQue. For Qwen3-8B, F1 is typically at or above 0.80; for Llama-3.3-70B, DBA exceeds 0.80 F1 on most datasets. AUROC stays above random throughout, ranging from the mid-0.60s to the low-0.90s.

• The gains come mainly from recall, not from being more abstention-prone. Standard baselines like AYS often stay overconfident when the model hallucinates, producing low recall. On Bamboogle, recall for Llama-3.3-70B improves from 37 to 90. Across open-source models and harder datasets, DBA typically increases recall by +30 to +50 points compared to AYS.

• DBA has boundary conditions on frontier models. When base accuracy is high (e.g., GPT-5.1 on Mintaka), errors are rare and disagreement signals are sparse, letting AYS stay competitive. On complex datasets with lower accuracy (e.g., GPT-5.1 on CRAG), frontier models can produce stable but incorrect predictions across regimes, so DBA finds no disagreement to exploit.

• Ensembling addresses complementary failure modes. AYS detects stable-but-uncertain errors while DBA flags confident-but-fragile hallucinations. On GPT-5.1 with CRAG, AYS scores 66 F1 versus DBA-A at 53 F1 individually, but their union reaches 75 F1. On HotpotQA, ensembling boosts F1 to 73.

• DBA cannot fix errors that are consistent across regimes. In a manual error analysis of 100 random examples drawn from the six datasets, 67% of frontier-model disagreement cases fall into the "Both" category, where Direct and Assistive are both wrong. DBA detects "reasoning shortcuts" effectively but cannot correct errors rooted in missing parametric knowledge.

Methodology in Plain English

The authors hold the plan fixed and vary only the execution. For each question they build a gold-standard decomposition using a domain-specific language (DSL) that specifies variables, answer types, and sub-questions. Decompositions come from a two-stage pipeline using a strong LLM: for benchmarks with step-by-step annotations, the LLM translates those steps into DSL; for benchmarks without them, Gemini-2.5-Flash synthesizes a DSL program from the question. Every decomposition is then manually verified to remove planning errors as a confound.

Three prompting regimes execute that same plan differently:

• Direct — answer the question in one step with no explicit structure.
• Assistive — receive the full set of sub-questions up front and answer all of them in a single call.
• Incremental — execute the decomposition line by line, one model call per sub-question, filling in placeholders with previous answers and never showing the model the original top-level question or other steps.

Nine instruction-tuned models are evaluated: Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct, and Qwen3-8B at the ~8B scale; Qwen3-32B at medium scale; Llama-3.3-70B-Instruct and Qwen2.5-72B-Instruct at ~70B; plus GPT-5.1, Gemini-2.5-Pro, and Gemini-2.5-Flash. All models use greedy decoding (T = 0) except GPT-5.1, whose temperature is not exposed (API default with reasoning effort set to "medium"). Six multi-hop QA datasets are used: Bamboogle, FRAMES, MuSiQue, CRAG, HotpotQA, and Mintaka, filtered for multi-hop complexity, temporal independence, and semantic clarity, yielding 1,433 verified instances.

Grading uses an LLM-as-judge protocol (Gemini-2.5-Flash) rather than lexical metrics like EM or ROUGE, because surface-form matching is brittle. The judge normalizes units, aliases and abbreviations, enforces numeric tolerance and exact date matching, and penalizes explicit contradictions. Accuracy compares a regime's final answer to the gold reference; consistency compares two regimes' outputs for semantic equivalence independent of correctness.

For abstention, disagreement with the Direct answer triggers an "I don't know" response — this is DBA, instantiated as DBA-A (Assistive) or DBA-I (Incremental). Abstention is evaluated as an error-detection task where the positive class is an incorrect Direct prediction, reporting precision, recall, F1, and AUROC. Baselines are AYS ("Are You Sure?"), which adds one binary verification turn; IC-IDK, which appends an instruction permitting an explicit "I don't know" and is primed with K = 15 demonstrations (D = 4 instances where that model's Direct answer was verified incorrect, used as IDK demonstrations, and 11 where it answered correctly, used as standard answer demonstrations); and self-consistency, which samples multiple Direct responses at higher temperatures and takes a majority vote. Because heuristic-majority-vote self-consistency achieves high precision but critically low recall in these experiments, its detailed analysis is relegated to an appendix.

Why This Matters

The paper reframes question decomposition from a performance intervention into a diagnostic instrument. It shows that as models scale, the marginal value of decomposition shifts from expanding what can be answered to auditing confidence in what is known — and it does so with a method that needs no retrieval, no fine-tuning, and no self-reported confidence scores, which are often unfaithful. This matters for any deployment where a model must answer from internal knowledge alone.

Real-world applications:

• Privacy-restricted systems where external retrieval or verification is not permitted, and the model must decide whether to answer or defer.
• On-device applications where models run without access to a knowledge base or verification service.
• Time-critical decision-making pipelines where there is no opportunity to consult evidence before an answer is used.
• Hallucination detection and quality gating in pipelines that need to flag confident-but-incorrect outputs before they reach a downstream consumer.

Industry relevance: DBA is training-free and requires only additional inference calls, so it can be layered onto existing closed-book deployments. The cost profile is documented: Direct prompting uses 1 model call and 833 total tokens (1.00x); AYS uses 2 calls and 1,054 tokens (1.26x); IC-IDK uses 1 call and 963 tokens (1.16x); DBA-A uses 4 calls and 4,702 tokens (5.64x); DBA-I uses 6.4 calls and 4,912 tokens (5.89x); and Self-Consistency uses 8 calls and 5,420 tokens (6.51x). DBA-A generates 38% fewer output tokens than self-consistency. Moving from low-overhead methods like AYS to DBA-A raises total token usage from 1.26x to 5.64x relative to Direct prompting, so the reliability gain is a deliberate compute trade-off.

Future Directions

1. Handling consistent errors. DBA only detects errors that manifest as disagreement across regimes. When a model gives the same incorrect answer under Direct, Assistive, and Incremental prompting, DBA treats it as stable and offers no corrective signal — a case that accounts for 67% of frontier-model disagreement cases in the authors' manual analysis.

2. Reducing dependence on externally supplied decompositions. The main experiments rely on decompositions that are manually verified or produced by a strong teacher model, which makes the method less self-contained, especially for weaker models that struggle to generate good multi-hop decompositions themselves. The paper notes that a 70B-scale open model can generate usable decompositions in most manually audited cases.

3. Lowering the computational and latency overhead. DBA requires at least one decomposed execution in addition to the Direct call, and Incremental prompting adds one model invocation per hop. Finding cheaper ways to obtain the same disagreement signal is an open problem.

4. Better integration of complementary uncertainty signals. The ensemble results suggest AYS and DBA fail in different regimes, so principled combination strategies — rather than simple unions — may extend coverage across both stable-but-uncertain and confident-but-fragile errors.

Target Audience

Researchers and practitioners working on LLM hallucination detection, uncertainty estimation, selective prediction, and abstention; NLP researchers studying multi-hop QA, question decomposition, and prompt sensitivity; and applied engineers building closed-book QA systems that must decide when a model should decline to answer. It is also useful for evaluation-focused readers, since it offers a concrete, training-free protocol for probing model reliability without new supervision or external retrieval.

Authors’ abstract

Large language models often struggle to recognize their knowledge limits in closed-book question answering, leading to confident hallucinations. While decomposed prompting is typically used to improve accuracy, we investigate its impact on reliability. We evaluate three task-equivalent prompting regimes: Direct, Assistive, and Incremental, across different model scales and multi-hop QA benchmarks. We find that although accuracy gains from decomposition diminish in frontier models, disagreements between prompting regimes remain highly indicative of potential errors. Because factual knowledge is typically stable while hallucinations are stochastic, cross-regime agreement provides a precise signal of internal uncertainty. We leverage this signal to implement a training-free abstention policy that requires no retrieval or fine-tuning. Our results show that disagreement-based abstention outperforms standard uncertainty baselines as an error detector, improving both F1 and AUROC across settings. This demonstrates that decomposition-based prompting can serve as a practical diagnostic probe for model reliability in closed-book QA.