Multi-Step Knowledge Interaction Analysis via Rank-2 Subspace Disentanglement

AI summary

Overview

Research area: Natural Language Processing / mechanistic interpretability of Large Language Models — specifically how models combine parametric knowledge (PK, knowledge stored in weights) and contextual knowledge (CK, knowledge from the input context) while generating Natural Language Explanations (NLEs).

Technical level: Advanced. The paper relies on linear algebra (projection subspaces, rank, orthonormal bases, singular values), activation patching, and probing methodology. The high-level results are accessible, but the framework requires comfort with representation-level analysis.

Scope in one sentence: The paper replaces the previous rank-1 view of PK–CK interaction with a rank-2 projection subspace and uses it to trace, step by step, how PK and CK contributions shift during NLE generation across four QA datasets and three open-weight LLMs.

What This Paper Is About

LLMs generate explanations by drawing on two knowledge sources: what they memorized during training (parametric knowledge, PK) and what the provided context says (contextual knowledge, CK). Prior work (Minder et al., 2025) modeled this interaction as a single binary choice inside a rank-1 subspace, and only looked at one generation step — usually the final answer — and only at cases where PK and CK conflict.

This paper argues that a rank-1 subspace cannot represent the fuller set of interactions (supportive, complementary, conflicting, irrelevant, and knowledge-suppression cases), and that the dynamics of PK–CK use unfold over an entire generated explanation rather than in one step. The goal is to build a rank-2 subspace that disentangles PK from CK and to use it for the first multi-step analysis of knowledge interaction during generation.

Key Contributions

1. A rank-2 projection subspace for PK–CK disentanglement. The authors generalize the rank-1 formulation of Minder et al. (2025) to a subspace spanned by two orthonormal directions, u_c (context) and u_p (parametric), and argue theoretically (Theorem A.1) that rank-1 fails the identifiability ("bijective mapping") requirement across interaction types. They also define normalized per-step contributions α_i^c and α_i^p in [0,1], with Δ_i = α_i^p − α_i^c tracking the balance at each generation step.

2. The first multi-step analysis of PK–CK interaction. Rather than scoring a single answer token, the method tracks individual PK and CK contributions at every token position of the generated NLE sequence.

3. An extended taxonomy of PK–CK interactions. Following Cheng et al. (2024), the paper moves beyond conflict-only analysis to five categories: supportive, complementary, conflicting, irrelevant, and knowledge suppression (noise). It also introduces a joint intent w_b = f(w_c, w_p) representing the model's intrinsic behavior when both sources are considered.

4. Causal validation through subspace intervention. Steering the answer token's hidden representation along the learned PK and CK axes reproduces the behavior induced by explicit intent instructions, and down-weighting the PK axis reduces hallucinated generation on RAGTruth.

Main Findings

• Rank-1 is insufficient; rank-2 is enough. Kernel density estimates of the subspace component for answer tokens overlap across knowledge interaction types and their means converge to 0 under rank-1 (Figure 2). Cumulative explained variance reaches 1.0 at rank-2 across all three models and four QA datasets (Figure 3), which the authors take as evidence that rank-2 is the minimum sufficient projection subspace.

• PK dominates in context-rich commonsense tasks. On OpenBookQA, activation patching from w_b → w_p (BOTH→PRI, selected layers L13–18) produces a larger probability gain than w_b → w_c (BOTH→CTX, layers L15–17) for Llama-3.1-8B-Instruct; similar results appear for gemma-2-9b-it and Mistral-7B-Instruct-v0.3.

• Answer-level alignment tracks dataset composition. At the final answer token, BaseFakepedia and MultihopFakepedia show higher CK contribution, while StrategyQA and OpenBookQA show higher PK contribution for Llama-3.1-8B-Instruct. Both Fakepedia datasets contain more conflicting examples than other interaction types, consistent with prior findings that models align with CK when context conflicts, and with PK when contexts are supportive.

• Generation dynamics shift over the sequence. Across all datasets, the model starts NLE generation with higher CK, then draws on both sources with a slight prioritization of PK. For longer NLEs, CK and PK compete with higher fluctuation — the authors attribute this to harder multi-hop examples with higher token uncertainty.

• Hallucinated spans align strongly with the PK axis. On RAGTruth and Dolly (AC), the gap between PK and CK is much higher for sequences containing human-annotated hallucinated spans than for others, matching prior observations of positive correlation between PK and hallucination (Sun et al., 2025). For RAGTruth with Llama-3.1-8B-Instruct, hallucinated examples show average CK 0.46 (c_i=1.8559) and PK 0.54 (p_i=2.2277), versus 0.49 (c_i=1.9399) and 0.51 (p_i=2.0927) for non-hallucinated examples.

• CoT shifts alignment, with one exception. Comparing standard versus Chain-of-Thought prompting, CoT maintains similar CK alignment on all datasets and reduces PK alignment on all datasets except OpenBookQA (Figure 8).

• Intervention reproduces intent behavior. Steering the answer-token representation with weights along the CK and PK axes (p_i > c_i, p_i < c_i, p_i = c_i) recovers the answers obtained with explicit intent prompts at high rates across all datasets and models (Table 1, e.g. StrategyQA Llama 0.910 / 0.940 / 0.935; OpenBookQA Llama 0.940 / 0.960 / 0.960).

• PK down-weighting reduces hallucinations. On 450 steered RAGTruth examples, hallucinated/total counts were 243/450 without steering, 202/450 with PK-downweight p_i = 2.1 (p < 0.0001), 221/450 with CK-upweight c_i = 1.94 (p = 0.048), and 193/450 with both (p < 0.0001). Sun et al. (2025) reports 172 wins, 179 ties, 99 losses, a reduction of 73/450, compared to this paper's 50/450 reduction; the authors note their result is statistically significant but smaller, and that the PK axis is neither a unique nor a complete causal mechanism for hallucination.

Methodology in Plain English

The researchers ask a model the same question in three ways: with no context (to elicit parametric knowledge, a(q, ϕ)), with the provided context (to elicit contextual knowledge, a(q, c)), and with an instruction to consider both (w_b). Comparing these answers lets them label each example with an interaction type (supportive, complementary, conflicting, irrelevant, or knowledge suppression).

To find where in the network this interaction lives, they use Patchscope, an activation-patching technique. They take two nearly identical prompts that differ only in the instruction ("follow the context" versus "follow your prior knowledge") and swap hidden-state activations from a source prompt into a target prompt at each layer. Layers whose patching makes the target behave like the source are marked as intent-sensitive. For the rank-2 setup, they patch from the "both knowledge sources" prompt to each single-source prompt, yielding two layer sets, L_{b→p} and L_{b→c}.

They then learn a projection matrix P spanned by two orthogonal unit directions, one for context and one for parametric knowledge, by minimizing two negative log-likelihood objectives sequentially while keeping the directions orthonormal. Once learned, any token's hidden representation can be split into a residual component plus a CK component and a PK component. Taking absolute inner products with each direction and normalizing gives per-token contributions α_i^c and α_i^p that sum to 1, so the difference Δ_i says which source dominates at each step.

For validation, they intervene: they rewrite the answer token's hidden state with chosen weights along the two axes and check whether the model produces the same answer as it would under the corresponding explicit instruction. They also regenerate responses under PK-downweighting and measure hallucination reduction with human/model-based judgments.

Why This Matters

Impact on research. The paper reframes PK–CK interaction as a multidimensional phenomenon rather than a single binary switch, and provides a geometric, token-level instrument for studying explanation faithfulness and hallucination. It offers a reusable alternative to rank-1 probing, backed by a theoretical identifiability argument and causal intervention evidence, and it links knowledge alignment to hallucination and to CoT behavior.

Real-world applications (as suggested by the paper's framing and results):

• Retrieval-augmented generation (RAG) systems that need to detect when a model is ignoring retrieved evidence in favor of memorized content.
• Hallucination mitigation by steering the parametric direction down, which this work shows reduces hallucinated spans on RAGTruth.
• Faithfulness auditing of generated explanations, since the method scores each token of the explanation rather than only the final answer.
• Context-sensitive knowledge control for domains where grounding in provided documents matters more than recall, such as claim verification or evidence-based QA.

Industry relevance. Practitioners deploying instruction-tuned open-weight models (Llama-3.1-8B, Gemma-2 9B, Mistral-v0.3 7B in this study) could use the learned directions as inference-time steering vectors without retraining weights, which the paper frames as a lightweight intervention. The stronger-than-expected PK reliance observed on a context-rich commonsense task is a direct warning for RAG pipelines that assume retrieved context will be used.

Future Directions

• Relaxing the orthogonality constraint. The authors restrict the subspace to two orthonormal directions to get a clean directional contribution, and note that non-orthogonal directions are possible but would complicate mapping PK–CK contributions onto them; they leave this for future work.
• Moving beyond rank-2. The paper limits itself to rank-2 based on the observed cumulative explained variance, and the provided text is truncated mid-sentence on the question of rank-3 or higher projections.
• Identifying additional hallucination mechanisms. The PK axis is described as neither unique nor complete as a causal explanation for hallucination; the authors suggest another projection subspace exists but should not be orthogonal to the PK direction.
• Context-sensitive knowledge control. The observed PK dominance on the commonsense task OpenBookQA is flagged explicitly as highlighting potential for future work on making models ground answers in context rather than recall.

Target Audience

Researchers and graduate students in NLP interpretability, knowledge-conflict analysis, and hallucination study will get the most from this paper, particularly those already familiar with Minder et al. (2025) and probing-based methods. It is also relevant to applied ML engineers building RAG or evidence-grounded QA systems who want a representation-level diagnostic for whether a model is using retrieved context. Readers without a background in linear algebra or activation patching should expect the method section to require extra effort, though the experimental findings and steering results are stated in accessible terms.

Authors’ abstract

Natural Language Explanations (NLEs) describe how Large Language Models (LLMs) make decisions by drawing on external Context Knowledge (CK) and Parametric Knowledge (PK). Understanding the interaction between these sources is key to assessing NLE grounding, yet these dynamics remain underexplored. Prior work has largely focused on i) single-step generation and ii) modeled PK--CK interaction as a binary choice within a rank-1 subspace. This approach overlooks richer interactions and how they unfold over longer generations, such as complementary or supportive knowledge. We propose a novel rank-2 projection subspace that disentangles PK and CK contributions more accurately and use it for the first multi-step analysis of knowledge interactions across longer NLE sequences. Experiments across four QA datasets and three open-weight LLMs demonstrate that rank-1 subspaces struggle to represent diverse interactions, whereas our rank-2 formulation captures them effectively, highlighting PK alignment for supportive interactions and CK alignment for conflicting ones. Our multi-step analysis reveals that hallucinated generations exhibit strong alignment with the PK direction, though context-faithful generations maintain a more balanced alignment between PK and CK.