Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation

AI summary

Overview

Research area: Natural Language Processing — retrieval-augmented generation (RAG), KV-cache compression, and faithfulness evaluation.

Technical level: Intermediate. The paper assumes familiarity with transformer inference, key-value caching, and quantization, but its central argument (accuracy metrics hide grounding failures) is stated plainly enough for readers with general ML background.

Scope: A controlled audit of how offline KV-cache quantization (INT8 and INT4) affects both answer accuracy and evidence faithfulness in RAG, using Qwen2.5-7B-Instruct on two QA benchmarks.

What This Paper Is About

RAG systems can precompute and store the key-value caches of retrieved documents so they do not have to re-encode the same context on every query, and quantizing those stored caches reduces their disk footprint. Prior work only checks whether compression hurts task accuracy (Exact Match, F1). This paper asks a different question: after compression, are the model's answers still actually supported by the retrieved evidence, even when the answer string remains correct?

Key Contributions

1. A faithfulness-centered audit protocol for offline KV-cache quantization. The protocol caches the full retrieved context in a single causal prefill pass, avoiding positional-stitching confounds, so cache bit-width is the only variable changed across conditions.
2. Evidence that INT8 is near-lossless but INT4 silently degrades grounding. On examples where containment-EM is unchanged between BF16 and INT4, more than 90% of faithfulness flips are negative (McNemar p < 10^-20), meaning accuracy metrics are blind to the regression.
3. Demonstration that the harm is amplified by retrieval conditions. INT4 damage grows with retrieval depth (directional, p = 0.12, underpowered) and with distractor fraction on RGB (β = 0.22, significant at K = 3 and K = 5).
4. The claim that faithfulness, not accuracy alone, is the necessary audit signal for compressed RAG deployment, especially given that INT4 is the only condition producing degenerate outputs.

Main Findings

• INT8 is near-lossless across both metrics. Compared against the C1 BF16 cache baseline, INT8 stays close on containment-EM, token-F1, HHEM hallucination rate, NLI entailment, and the LLM judge across both benchmarks.
• INT4 hurts accuracy. Containment-EM and token-F1 both drop. Correct-to-wrong flips far outnumber wrong-to-correct flips on RGB (133 vs. 10) and HotpotQA (117 vs. 24).
• INT4 causes hidden faithfulness harm. On the accuracy-preserved subset (containment-EM unchanged between C1 and C3), the LLM judge records 231 worsenings vs. 24 improvements on RGB (p < 10^-38) and 173 vs. 31 on HotpotQA (p < 10^-30). HHEM corroborates the trend on RGB at all K and on HotpotQA at K = 5. The paper's summary table states this effect as McNemar p < 10^-20 on both datasets.
• Harm grows with retrieval depth. The INT4–BF16 hallucination gap rises monotonically from K = 1 to K = 5 on both benchmarks: on RGB from 0.09 to 0.26, and on HotpotQA from 0.05 to 0.25. The slope test uses only three retrieval depths and is underpowered (p = 0.12), so the authors treat it as directional.
• Distractor passages amplify the damage. On RGB, where per-example noise ratios are available, the INT4 faithfulness gap grows with distractor fraction independently of retrieval depth (β = 0.22, significant within-K at K = 3 and K = 5).
• HotpotQA refusal calibration breaks at low K. The refusal rate drops from 0.513 to 0.317 at K = 1 under INT4, which artificially inflates containment-EM and NLI entailment. The Table 1 summary describes this as a drop from 51% to 32%. Because of this, the LLM judge is the primary H1 signal on HotpotQA.
• INT4 uniquely produces degenerate output. Empty outputs or repetition loops reach 6% on RGB and 3% on HotpotQA at K = 5; BF16 and INT8 never degenerate.
• Storage savings are real but below nominal. At K = 5, INT8 and INT4 reduce cache size by roughly 1.9× and 3.6× versus BF16, below the nominal 2× and 4×. Per-block scale and zero-point overhead consumes approximately 11% of the INT4 cache footprint; per-token overhead consumes approximately 5% of the INT8 cache.
• The cache round-trip itself is clean. C1 (BF16) reproduced C0 (Oracle) within floating-point tolerance on all 50 held-out gate examples, so differences in C2/C3 are attributable to quantization bit-width.
• Degeneracy is semantic, not numerical. No non-finite logits appeared under C0, C1, or C2. Under C3 (INT4), non-finite logits occurred in under 0.1% of examples and were aborted rather than allowed to emit corrupted tokens.
• Faithfulness signals are complementary, not redundant. HHEM and NLI entailment agree only moderately on RGB (Pearson r = 0.44). The LLM judge reaches 92% agreement with human labels on RGB and 76% on HotpotQA, where refusal artifacts add noise. No single signal captures the full harm: H1 is supported by all three signals on RGB but only by the LLM judge on HotpotQA at low K.

Methodology in Plain English

The researchers held everything constant except how precisely the stored document cache was represented. For each query they concatenated a system prompt with the top-K retrieved chunks and ran a single prefill pass, producing one unified KV cache for the whole retrieved context. They deliberately avoided caching chunks separately and stitching them, because separately cached chunks are built at local token positions but consumed at global positions, and that positional mismatch can degrade quality on its own.

After prefill, the cache was quantized before storage and dequantized back to bfloat16 before decoding, matching an offline deployment where document caches are written to disk once and reused. INT8 used per-token asymmetric quantization. INT4 used group-wise quantization with groups of 64 channels, each group carrying its own scale and zero-point, to prevent large-magnitude channels from dominating the range and collapsing smaller values under INT4's 16 levels.

They compared four conditions with identical chunks and greedy decoding: C0 (Oracle) full-context generation with no cache round trip, C1 (BF16) an uncompressed cache stored and reloaded as the cache baseline, C2 (INT8), and C3 (INT4). C1–C2 isolates INT8's effect; C1–C3 isolates INT4's.

The model was Qwen2.5-7B-Instruct, chosen partly because its bfloat16-native precision avoids float16 dynamic-range overflow that can silently corrupt cached KV values. Two benchmarks were used: RGB (300 examples) with its own positive and distractor passages, and HotpotQA (300 examples, distractor split) requiring multi-hop evidence combination. Each example ran at K ∈ {1, 3, 5}, giving 300 × 3 × 4 = 3,600 generations per dataset. Retrieval used a FAISS index over bge-small-en-v1.5 embeddings, retrieving top-10 chunks and slicing to the first K.

Accuracy was measured by containment exact match (a prediction counts as correct if any normalized gold alias appears in the answer) and token-level F1. Faithfulness was measured by three independent signals: HHEM-2.1-Open hallucination rate at threshold 0.5 (lower is better), DeBERTa-v3-large NLI entailment (higher is better), and an LLM judge using Claude Haiku 4.5 (higher is better). Three hypotheses were tested against C1 BF16 as reference: H1, whether faithfulness degrades when accuracy is preserved (paired McNemar test on faithfulness label flips within the accuracy-preserved subset); H2, whether harm increases with retrieval depth (linear slope over the three K values); and H3, whether retrieval noise amplifies INT4 degradation (regressing the per-example INT4 faithfulness gap on distractor fraction within each K, evaluated only on RGB).

Why This Matters

Impact on research. The paper argues that the standard accuracy proxies used to validate KV-cache compression are structurally incapable of detecting the failure it reports. Prior compression work (KIVI, KVQuant, ZipCache, KVTuner) evaluates online cache compression through accuracy or perplexity; prior RAG faithfulness work (RAGChecker, RAGTruth, HELMET) does not examine cache compression; prior offline cache RAG work (TurboRAG, CacheBlend) does not examine faithfulness. This paper positions itself at the intersection of those three gaps, making faithfulness a required evaluation target rather than an optional one.

Real-world applications:

• Document QA and enterprise search assistants that precompute and store caches for large internal corpora and must justify aggressive compression to fit storage budgets.
• Retrieval-augmented customer support or compliance systems, where an answer that is correct but no longer grounded in the cited policy document is a liability rather than a success.
• Multi-hop research or legal assistants on HotpotQA-style workloads, where the paper shows the refusal mechanism itself degrades and the system guesses instead of declining.
• Long-context RAG deployments with high retrieval depth, where the hallucination gap grows from K = 1 to K = 5 and the storage benefit falls below the nominal ratio.

Industry relevance. The reported savings are meaningful but qualified: approximately 1.9× for INT8 and 3.6× for INT4 at K = 5, with INT4 unable to reach its nominal 4× because of per-block metadata overhead, and with INT4 additionally producing degenerate outputs on up to 6% of RGB generations. For teams weighing storage cost against reliability, the paper's practical claim is that the storage-versus-faithfulness trade-off must be measured directly, not inferred from EM or F1.

Future Directions

1. A denser retrieval-depth sweep. The authors note that a sweep over K ∈ {1, 2, 3, 4, 5, 7, 10} would likely yield a statistically significant slope for H2, which the current three-point test (p = 0.12) cannot establish.
2. Broader model and benchmark coverage. The limitations section notes the study covers one model family, two QA-style benchmarks, and three retrieval depths, and that the magnitude of faithfulness loss may differ for larger models, other architectures, denser retrieval settings, or production retrievers.
3. Resolving the HotpotQA refusal confound. The refusal-calibration artifact at low K undermines HHEM and NLI in that regime, so a more reliable faithfulness measurement for refusal-heavy settings is needed.
4. Faithfulness-preserving compression methods. Since the paper isolates the harm but does not propose a fix, an open question is whether selective, mixed-precision, or query-aware allocation schemes can retain INT4-level storage savings while keeping grounding intact.

Target Audience

Researchers and engineers working on efficient LLM inference, KV-cache compression, and RAG deployment who currently validate compressed systems using accuracy or perplexity alone. It is also relevant to evaluation researchers interested in hallucination detection and to practitioners deciding how aggressively to quantize stored document caches. Readers without background in transformer inference or quantization will find the framing accessible, but the method details assume intermediate familiarity.

Authors’ abstract

Retrieval-augmented generation systems can precompute and store key-value caches of retrieved documents to avoid re-encoding context at every query. Quantizing these caches further reduces storage, but no prior work asks whether compression damages faithfulness, whether responses remain grounded in the retrieved evidence. Faithfulness and accuracy are not equivalent: a model can produce a correct answer that is no longer supported by the context it was given. We evaluate Qwen2.5-7B-Instruct under INT8 and INT4 quantization on RGB and HotpotQA, measuring both accuracy and faithfulness with a hallucination detector, NLI entailment, and an LLM judge. INT8 is near-lossless across both metrics. INT4 reduces accuracy and, more critically, even among answers that remain factually correct, over 90% of faithfulness changes are negative, i.e., accuracy metrics are blind to this regression. The harm grows under noisy retrieval and with more retrieved chunks. Faithfulness must be audited before compressed caches are deployed.