Rational Synthesizers or Heuristic Followers? Analyzing LLMs in RAG-based Question-Answering

AI summary

Overview

• Research area: Retrieval-Augmented Generation (RAG), LLM evidence aggregation, and model trustworthiness/robustness to contextual persuasion.
• Technical level: Intermediate. The paper is behavioral and empirical rather than mathematical, but assumes familiarity with RAG pipelines, token probabilities, greedy decoding, and Chain-of-Thought prompting.
• Scope: The paper introduces GroupQA, a dataset of 1,635 controversial binary questions paired with 15,058 evidence documents, and uses controlled interventions to characterize how LLMs aggregate groups of conflicting retrieved evidence. Single author: Atharv Naphade (Carnegie Mellon University), arXiv:2601.06189v1 [cs.AI], 08 Jan 2026.

What This Paper Is About

Prior work on conflicting evidence in RAG mostly studies clean one-versus-one document comparisons. Real retrieval, however, returns noisy top-k lists containing clusters of documents — several supporting one claim, several supporting another, plus redundancies. This paper asks why an LLM answers the way it does when given such groups: is it because the evidence is factually strong, because of a prior belief, or merely because a claim is repeated frequently? To answer this, the author builds GroupQA and runs controlled manipulations of evidence quantity, redundancy, ordering, and conflict.

Key Contributions

1. The GroupQA dataset: A benchmark of 1,635 controversial binary questions paired with 15,058 clustered retrieved documents varying in stance, built to support systematic evaluation of how LLMs respond to groups of contradictory information (average 9.21 paragraphs per question).
2. Persuasion evaluation methods: Simple intervention-based metrics — answer flip thresholds (X_min), belief plasticity (PL), and leave-one-out document influence — that characterize how evidence quantity, diversity, ordering, and conflict affect model decisions across scales.
3. Group dynamics characterization: Demonstration that group structure significantly influences model answers: repetition of a single piece of evidence can be more persuasive than distinct evidence sets, model scale trades off belief flexibility and stability, explicit conflict stabilizes decisions, and evidence order affects favorability.
4. An unfaithfulness result: Evidence that model explanations for group-based answers do not match the causally most important document, so self-reported attribution cannot be trusted as a diagnostic.

Main Findings

• Larger models are less plastic. Plasticity (PL), the mean absolute shift in probability assigned to "Yes" when documents are added, was 0.0074 for Llama-3.1-70B versus roughly 10x higher (0.075) for DeepSeek-R1-8B. Across 10 checkpoints from the Llama and Qwen families (1B to 70B parameters) on a random subset of 100 tasks, plasticity decayed with parameter count according to the power-law fit y = 0.180 · x^(−0.097), with family differences contributing to variance (R² = 0.472).
• Redundancy beats diversity (Illusory Truth Effect). When evidence opposed model priors, paraphrased variations of a single document flipped answers more often than distinct independent documents for every model tested: DeepSeek-R1-8B 76.5% vs. 67.6%, Gemini-2.5-FL 75.6% vs. 63.7%, Llama-3.1-70B-Instruct 69.8% vs. 62.9%, and Qwen3-32B 73.7% vs. 67.3%. In the low-evidence regime (1–2 documents), distinct evidence was strictly required to initiate belief revision, but redundant evidence scaled more aggressively as context grew.
• Aggregate flip rates vary widely across models. Over roughly 1,630 questions, Qwen3-32B had the lowest overall flip rate (170 flips, 10.4%) and Gemini-2.5-FL the highest (351 flips, 21.5%); Llama-3.1-70B-Instruct flipped 197 times (12.1%) and DeepSeek-R1-8B 279 times (17.1%).
• Decisiveness differs from flip frequency. Llama-3.1-70B was the most decisive: it flipped least often but, when it did yield to distinct counter-evidence, required the fewest documents (X_min = 1.27 for distinct evidence) versus 1.52 for DeepSeek-R1-8B. Flip thresholds were consistently higher for paraphrased evidence (1.67 for Llama-3.1-70B, 2.01 for DeepSeek-R1-8B, 2.24 for Gemini-2.5-FL, 1.95 for Qwen3-32B).
• Chain-of-Thought changes almost nothing. The absolute probability mass shift between standard and CoT prompting was < 0.5 ± 0.56% at 95% confidence across all model scales, suggesting reasoning traces act as post-hoc rationalizations rather than corrective inference.
• Verbally reported evidence importance is unfaithful. On a subset of 200 questions, Llama-3.1-70B's verbalized attribution matched the leave-one-out causal attribution on only 26% of questions.
• Primacy bias holds. Models were significantly less likely to flip belief when prior-confirming evidence appeared first (t = 0) rather than last; for Llama-3.1-70B this produced a 3.5% higher probability of prior retention compared to the inverse ordering.
• Explicit conflict stabilizes but does not eliminate the heuristics. Models detected knowledge conflicts over 89.8% of the time and correctly attributed document stance over 76.2% of the time. Introducing balanced conflicting context increased decision stability, reduced flip rates relative to prior-only settings, and made paraphrased documents show faster diminishing returns — though the attenuation was only consistent for Llama-3.1-70B; in smaller models paraphrased documents were stronger on average in certain scenarios.
• Priors lean slightly to "No" and evidence shifts little. Prior probability of "Yes" ranged from 0.551 (DeepSeek-R1-8B) to 0.681 (Llama-3.1-70B) across the four models, and the full-evidence posterior differed from the prior by only 0.074 ± 0.084 for Llama-3.1-70B, which agreed with the majority viewpoint in 69% of cases.

Methodology in Plain English

The author first generated realistic controversial but unambiguous binary questions using GPT-4o, stratified across 95 distinct topic categories (e.g., Bioengineering, Zoology, Historical Revisionism) and de-duplicated with cosine similarity filtering at a threshold above 0.92. A pool of 1,948 candidate questions was produced. Each question was converted into an affirmative and a negative assertion using a deterministic template (for example, "Do vaccines cause autism?" becomes "Vaccines cause autism" and "Vaccines do not cause autism"), and the top k = 10 documents were retrieved for each using the Google Search API to emulate a real retrieval engine.

Raw text was extracted with the Trafilatura library (retaining only documents with at least 200 characters of extractable text), and metadata such as source URL or publication date was deliberately excluded so models relied only on textual content. GPT-4o-mini and Gemini-2.5-Flash served as judges that had to agree on each document's stance (Affirmative, Negative, or Neutral) and qualitative strength (Strong, Medium, Weak); neutral documents were discarded. Manual verification of 100 random documents showed 99% stance agreement. Questions were kept only if at least one valid document existed for both sides, removing 313 questions (16.1% attrition). The most relevant paragraph per document was selected using all-MiniLM-L6-v2 embeddings and cosine similarity. The paper's Table 1 reports 1,883 questions processed with an 86.83% acceptance rate and 22,264 total documents scraped, alongside the final counts of 1,635 questions and 15,058 documents.

Experiments then manipulated the document groups. Model belief was measured as the normalized probability assigned to the "Yes" token before evidence (prior) and after evidence (posterior). The author measured answer flips under greedy decoding, defined a flip threshold X_min as the mean minimum number of documents needed to invert a decision, and contrasted distinct documents against GPT-4o-generated paraphrases of a single document. Attribution faithfulness was tested by asking the model which document would matter most if removed and comparing that answer to a leave-one-out perturbation ground truth. Finally, balanced contexts (one supporting plus one opposing document) were used to test conflict detection and ordering effects. Prompts used temperature 0.0 unless otherwise noted.

Why This Matters

The paper argues that current LLMs behave as heuristic aggregators rather than rational synthesizers: they track low-level cues such as token frequency and position instead of evaluating evidentiary independence. If that holds, a core assumption behind RAG — that models weigh conflicting evidence to synthesize a coherent truth — is not supported.

Real-world applications:

• RAG system design: De-duplicating retrieved evidence could mitigate the Illusory Truth Effect, and randomizing document order could reduce expected primacy bias.
• Deliberate retrieval of dissent: Because models become more resistant to redundancy when explicitly confronted with contradiction, intentionally retrieving dissenting viewpoints — which the paper notes typically does not happen unless forced — is offered as a direction for more rational synthesis.
• Adversarial content attacks: The findings imply RAG systems are vulnerable to "context stuffing," where a malicious actor need not supply high-quality evidence but simply needs to dominate the context window with redundant, paraphrased variations of a target claim.
• Safety and explainability: The attribution-unfaithfulness result, combined with the CoT finding, means users cannot rely on a model's stated reasoning to know which documents actually drove an answer.

Industry relevance: Any deployment of retrieval-augmented assistants, search-augmented chat, or agentic pipelines that assemble multi-document context is directly affected, particularly in contested domains such as health, policy, and consumer claims. The paper also notes publication of these vulnerabilities is intended to motivate defenses such as frequency-penalized attention.

Future Directions

• Mechanistic explanations: The paper explicitly states it characterizes behavioral and causal effects but does not identify their origin inside model internals; attention patterns, circuit-level explanations, and neuron-level attributions remain open.
• Beyond binary questions and English text: The limitations section flags that richer answer spaces, non-English sources, and domains with stronger factual consensus (such as mathematics or formal logic) may produce different belief dynamics.
• Mitigation strategies: The paper proposes de-duplication of evidence, randomized retrieval order, and forced retrieval of dissenting viewpoints as candidates, but does not evaluate them.
• Resolving the divergence across model sizes: The redundancy attenuation under conflicting contexts was consistent only for Llama-3.1-70B, with paraphrased documents stronger on average in certain scenarios for smaller models — a discrepancy the paper reports but does not explain.
• Reconciling reported collection figures: The paper reports 1,948 candidate questions with 313 removed (16.1% attrition) in the main text while Table 1(b) reports 1,883 questions processed and an 86.83% acceptance rate, an internal discrepancy left unaddressed.

Target Audience

AI and NLP researchers studying retrieval-augmented generation, hallucination mitigation, and knowledge conflicts; safety and trustworthiness researchers interested in sycophancy, contextual persuasion, and unfaithful explanations; and engineers building or evaluating production RAG pipelines who need to know whether document composition, redundancy, or ordering is steering system outputs. The paper is also relevant to policy and misinformation researchers, given its explicit dual-use discussion: GroupQA is released under a license prohibiting its use for factual knowledge training, and all instances are metadata-flagged for exclusion from future pre-training corpora.

Authors’ abstract

Retrieval-Augmented Generation (RAG) is the prevailing paradigm for grounding Large Language Models (LLMs), yet the mechanisms governing how models integrate groups of conflicting retrieved evidence remain opaque. Does an LLM answer a certain way because the evidence is factually strong, because of a prior belief, or merely because it is repeated frequently? To answer this, we introduce GroupQA, a curated dataset of 1,635 controversial questions paired with 15,058 diversely-sourced evidence documents, annotated for stance and qualitative strength. Through controlled experiments, we characterize group-level evidence aggregation dynamics: Paraphrasing an argument can be more persuasive than providing distinct independent support; Models favor evidence presented first rather than last, and Larger models are increasingly resistant to adapt to presented evidence. Additionally, we find that LLM explanations to group-based answers are unfaithful. Together, we show that LLMs behave consistently as vulnerable heuristic followers, with direct implications for improving RAG system design.