Do We Need an LLM to Know Why Sources Conflict? Decision Models for Knowledge-Conflict Typing in RAG

AI summary

Overview

Research area: Natural Language Processing — specifically knowledge-conflict typing in retrieval-augmented generation (RAG), and the use of "typed decision models" as cheap, fast classifiers in place of generative LLMs.

Technical level: Intermediate. The paper is readable without deep mathematical background, but it assumes familiarity with retrieval-augmented generation, classification metrics (accuracy, macro-F1, ECE, AUROC) and significance testing.

Scope (one sentence): The paper benchmarks all 11 decision models served by a single commercial inference gateway against Gemini LLMs on the 458-query Conflicts benchmark, testing accuracy, cost, latency, calibration, question format, confidence-gated cascades, robustness diagnostics and end-to-end answer quality.

What This Paper Is About

When a search-augmented model retrieves pages that disagree, the correct response depends on why they disagree: an underspecified question with several valid answers, a contested topic, stale information, or a genuinely false source each call for different behaviour. Cattan et al. (2025) built a five-label taxonomy for this and showed that telling a generator the conflict type improves responses by about 24 points — but predicting the type is hard even for frontier LLMs, with their best model, Gemini 2.5 Flash, reaching 65.3% accuracy on the published result.

This paper asks whether typed decision models — models that return probabilities over fixed options in a single forward pass instead of generating text — can serve as this classification layer quickly and cheaply, and whether their cheaper types preserve the downstream benefit.

Key Contributions

1. A census of decision models on conflict typing. All 11 decision models served by one commercial gateway (from 7 providers) are evaluated on Conflicts under the original paper's protocol — same input representation, same category definitions and examples copied verbatim, and Gemini rerun on identical inputs so every comparison is paired on the same queries.

2. A formulation study. The taxonomy is tested both as a single five-option choice question ("flat") and as a decomposition into yes/no questions combined in code ("tree"), including a revised tree (v2) fitted on one model's development-split answers.

3. Cost-quality cascades. The least-confident fraction of queries (fixed at 25% and 50%) is escalated from each decision model to Gemini 3.8 Flash, measuring how well each model's confidence ranks its own errors and how much of the LLM's accuracy can be recovered at a fraction of its cost.

4. Pre-specified diagnostics plus an end-to-end test. Three robustness hypotheses (dependence on publication dates, sensitivity to source order, detection of a single planted false source) are tested, and the predicted types are fed to a generator and checked by LLM judges to see whether the accuracy gain from gold types survives.

Main Findings

• Decision models are competitive with the paper's LLM at far lower cost. Asked directly, the 11 decision models reach 50.0–67.0% accuracy (median 62.6%) against 61.3% for Gemini 2.5 Flash with the paper's prompt. Ten of the 11 match or exceed it at 4–39× lower billed cost (2 were free) and 2.6–6.0× lower latency. Statistically, 8 are tied with it, 2 are significantly better (Liquid d1 Ultra at 67.0%, +5.7 points, p = 0.027; Perplexity Decider v1.2 at 66.7%, +5.4, p = 0.049), and 1 is significantly worse (ScaleDown Classify-1 at 50.0%, −11.3, p < 0.001).

• Cost and latency gap. Decision models cost $0.04–$0.41 per thousand queries against $1.66 for Gemini 2.5 Flash, and answer in 0.26–0.72 s against 1.55 s.

• The current LLM is still more accurate, but three models come close. Gemini 3.8 Flash at its lowest thinking level reaches 70.0%, significantly better than 8 of the 11 decision models. The other three (Liquid d1 Ultra, Perplexity Decider v1.2, OpenAI GPT-6 Luna Decisions) trail by 3.0 to 3.7 points without a significant difference.

• More thinking does not help. Gemini 3.8 Flash at its default thinking level scores 66.7%, which is −3.2 points against its low level (p = 0.019), and Gemini 2.5 Flash with thinking on scores 57.9% (p = 0.19).

• The published number did not fully reproduce. On all 458 queries, the rerun of Gemini 2.5 Flash scores 60.7% against the published 65.3%; the authors therefore compare all systems against their own rerun on identical inputs.

• The hardest classes split the field. On Outdated information, per-class F1 ranges from 27.5 to 72.7 (Gemini 3.8 Flash: 66.1). Misinformation has only 4 test instances and defeats every system: no decision model exceeds an F1 of 11.8 (Gemini 3.8 Flash: 25.0).

• Calibration varies more than accuracy. Expected calibration error of the flat question ranges from 0.031 to 0.188, and 7 of the 11 models are over-confident on average.

• Decomposing into yes/no questions fails for every model. Tree v1 is significantly less accurate than the flat question for 11 of 11 models. Jev 1.13 answers the compatible question "yes" almost as readily for opinion topics as for complementary ones (mean probability 0.69 vs. 0.81) and labels 89 of the 102 Conflicting opinions test queries as Complementary (F1 3.8). The repaired tree v2 is significantly better than flat for none of the models, tied for 7 and significantly worse for 4 — a repair fitted on one model does not transfer.

• Confidence-gated cascades work. Confidence separates correct from wrong answers with AUROC 0.64–0.76. Sending the least confident quarter of queries to Gemini 3.8 Flash gives cascades statistically tied with the LLM alone for 8 of the 11 models, at 25–37% of its cost, and beats random escalation for 10 of the 11, by up to 4.9 points. The exception is GPT-6 Luna Decisions (AUROC 0.65), whose 25% cascade equals random escalation (67.0% vs. 67.2%).

• Dates play no measurable role. Removing every date or adding a recency rank computed in code changes accuracy significantly in 0 of 22 tests; recall on Outdated moves little in either direction. The models appear to recognise outdated sources from content cues, not publication dates.

• Source-order sensitivity is model-specific. Shuffling results changes the predicted type on 5.4–42.1% of queries (Gemini 3.8 Flash: 9.4%). Four decision models flip significantly more often than the LLM, one less often, and six no differently; only 2 of 22 shuffle tests change accuracy significantly, both for GPT-6 Luna Decisions.

• A planted false source shows in the probabilities more than in the top label. In 96 modified No conflict queries, decision models label 7.3–66.7% of planted copies as Misinformation (Gemini 3.8 Flash: 52.1%), with Youden indices of +0.052 to +0.490 against +0.490 for the LLM (8 significantly worse, 3 tied). The Misinformation probability nevertheless separates planted from clean copies with AUROC 0.66–0.86, at least 0.75 for 10 of the 11.

• Cheaper types keep the downstream gain. With Gemini 2.5 Flash as generator and the primary judge, the original ordering reappears: no type 87.9%, types from Gemini 2.5 Flash 91.6% (+3.7 points, p = 0.032), gold types 95.3% (+7.4, p < 0.001). 10 of the 11 decision models are statistically tied with the Gemini 2.5 Flash predictor (−3.9 to +0.7 points); only ScaleDown Classify-1 is significantly worse (−3.9, p = 0.011). With a quarter of queries escalated, 11 of 11 cascades are tied (−1.0 to +0.7). Gains concentrate on Conflicting opinions (82.4% without a type to 97.1% with gold) and Outdated (72.7% to 87.3%), and no condition lowers answer recall (90.6–93.1%).

• The size of the gain depends on the judge. Under the second judge (Gemini 3.8 Flash), gold types still help (+5.7, p < 0.001) but predicted types do not (91.6%-style pipeline figure of 89.7% vs. 91.1% for no type, p = 0.47).

• Not everything is reported. Providers do not disclose model sizes or training data, so the differences between models cannot be explained, and the paper does not measure factual grounding, the third metric of Cattan et al. (2025).

Methodology in Plain English

The team took an existing benchmark, Conflicts — 458 search queries, each with roughly 9.2 Google search results and an expert-assigned conflict label from five categories — and froze a stratified 52-query development split (used only to choose settings) and a 406-query test split on which everything is reported.

Every system saw exactly the same input: each result as its URL, title, snippet, publication date when available, and a 512-token segment, kept in dataset order and numbered. The decision models received these as a "state" plus a typed question; the LLMs received them inside the original paper's prompt, copied word for word, with a code test that fails if a single word differs.

Two question formats were compared. In the flat format, one multiple-choice question offers the five categories, and the most probable option is the prediction. In the tree format, five yes/no questions (do the results differ; could all be correct; is it a contested topic; is it outdated; is some result incorrect) are asked in a single request and combined arithmetically into a five-way distribution. A revised tree reordered the questions after inspecting one model's development answers.

For cascades, the decision model answers every query and the quarter (or half) with the lowest top probability is re-typed by Gemini 3.8 Flash. Because the threshold is a quantile of the model's own confidences and uses no labels, the whole accuracy–cost curve can be computed on the test split without leakage; the 25% and 50% operating points were fixed before any cascade was computed.

Diagnostics were written down before being run: strip the dates, reorder the results twice with fixed seeds, and plant a false source by rewriting one result in 96 No conflict queries so it states a plausible wrong answer (written by GPT-4.1-mini so no classifier judges its own text). Answer changes were compared against an identical rerun, which sets the noise floor.

Finally, an end-to-end test generated 2,436 answers with Gemini 2.5 Flash — one with no type and one for each of the eleven predictor types — and two LLM judges, which do not see the search results, decided whether each answer showed the gold type's expected behaviour. The primary judge was validated on 100 answers labelled blind by two annotators, who agreed on 92% (Cohen's κ = 0.67) and matched the judge on 92%.

Comparisons are paired on identical queries, with 95% bootstrap intervals over 10,000 resamples and two-sided exact McNemar tests; with 406 queries the interval is about ±4.8 points wide, so differences under about five points are reported as ties.

Why This Matters

Impact on research. The paper reframes conflict typing as a deployment-engineering problem rather than a pure generation problem, and provides the first comparison of decision models as a class on a task that requires reading many documents and reasoning about their mutual relation. It also gives a reusable evaluation template — paired comparison against a rerun baseline, a measured noise floor, pre-specified diagnostics, and a label-free cascade threshold — that addresses known reproducibility problems in this area, including the fact that the published 65.3% did not fully reproduce (60.7% on the full 458 queries).

Real-world applications:

• Search-augmented assistants that must decide, per query,

Authors’ abstract

When the sources retrieved for a query disagree, the right response depends on why they disagree: a question with several valid answers, a contested topic, stale information and a false source each call for different behaviour. Cattan et al. (2025) showed that telling a generator the conflict type improves its responses by about 24 points, but that predicting the type is hard even for frontier LLMs. We study whether typed decision models, which return probabilities over fixed options in a single forward pass instead of generating text, can do this classification fast and cheaply. Using the CONFLICTS benchmark and the original paper's prompt verbatim, we evaluate all 11 decision models served by one commercial inference gateway, from 7 providers, against Gemini models on identical inputs. Asked directly, the decision models reach 50.0--67.0% accuracy; 10 of the 11 match or exceed Gemini 2.5 Flash, the paper's model (61.3%), at 4--39x lower billed cost (2 were free) and 2.6--6.0x lower latency. The current Gemini 3.8 Flash (70.0%) is significantly more accurate than 8 of them, but sending each decision model's least confident quarter of queries to it gives a cascade statistically tied with it for 8 of the 11, at 25--37% of its cost. End to end, answers generated with the types of 10 of the 11 follow the expected behaviour as often as with the LLM's types. Decomposing the taxonomy into yes/no questions lowers accuracy for every model, and a repair fitted on one model does not transfer. Pre-specified diagnostics show that dates play no measurable role and that sensitivity to source order varies widely between models. Decision models detect a planted false source less reliably than the LLM as hard labels, but their probabilities still separate planted from clean sources. We release code, prompts and all model outputs at https://github.com/Parv17k/decision-models-conflict-typing.