Old Habits Die Hard: How Conversational History Geometrically Traps LLMs

AI summary

Overview

• Research area: Natural Language Processing / LLM interpretability — specifically how multi-turn conversational history shapes an LLM's subsequent behavior, studied jointly at the behavioral (black-box) and representational (white-box) level.
• Technical level: Intermediate. Readers need familiarity with Markov chains, transition matrices, hidden-state representations, orthonormal bases, and Procrustes-style rotation estimation. The paper is written for an interpretability-literate audience rather than a general one.
• One-sentence scope: The paper introduces the History-Echoes framework, which measures how the presence or absence of a phenomenon (hallucination, refusal, sycophancy) in one conversational turn biases the same phenomenon in later turns, and shows that this behavioral persistence corresponds to a measurable geometric separation — a "geometric trap" — in the model's latent space.

What This Paper Is About

Once an LLM hallucinates, refuses, or agrees sycophantically in one turn, does it keep doing so in the following turns — and if so, why? The authors build a framework that answers this from two sides at once: a probabilistic Markov-chain model of the conversation's observable behavior, and a geometric analysis of the model's hidden representations during those same conversations. They find that higher behavioral persistence (a higher transition-matrix trace) corresponds to a larger angular separation between the latent representations of "phenomenon present" and "phenomenon absent" states, which they call a geometric trap.

Key Contributions

1. A dual-perspective framework (History-Echoes). A probabilistic metric based on the trace of a 2×2 Markov transition matrix over binary phenomenon states, combined with a geometric metric based on hidden-state representations projected onto a phenomenon/non-phenomenon orthonormal basis.
2. Evidence that the two perspectives agree. A Spearman correlation of 0.78 (p < 0.0002) between trace and reference angle across three models and six datasets (the Spearman correlation exceeds 0.77 for individual models).
3. A phenomenon-level ordering of persistence. Refusal shows the strongest carryover effects, followed by sycophancy, with hallucination the weakest — a finding reproduced in both the probabilistic and geometric analyses.
4. Sensitivity and robustness analyses. Carryover effects dissolve under topically inconsistent conversations (D_inconsistent), decay across higher-order Markov history, are strongest in layers at 85% relative depth, survive when the intermediate question is held fixed, and reproduce probabilistically in the closed models GPT-5 and Claude-Opus-4.5.

Main Findings

• Behavioral persistence is real and large. Averaged across open models, the mean trace of the transition matrix is 1.31, well above the value of 1 that would indicate no history dependence. Per-dataset traces are: NaturalQA 1.13 ± 0.01, TriviaQA 1.12 ± 0.01, Sorry 1.57 ± 0.13, Do-not-answer 1.59 ± 0.11, S-pos 1.33 ± 0.12, S-neg 1.14 ± 0.05.

• Self-transition probabilities drive the trace. For example, P(s_{φ+} | s_{φ+}) is 0.90 ± 0.02 on Sorry and 0.89 ± 0.02 on Do-not-answer, while P(s_{φ−} | s_{φ−}) reaches 0.89 ± 0.01 on S-neg. On hallucination datasets the pattern is split: NaturalQA shows P(s_{φ+} | s_{φ+}) = 0.74 ± 0.02, whereas TriviaQA shows P(s_{φ−} | s_{φ−}) = 0.68 ± 0.01.

• Probabilistic inertia manifests geometrically. Higher trace corresponds to a larger reference angle θ_ref, the separation between the mean phenomenon-present and phenomenon-absent representations. The correlation holds at Spearman 0.78 with p < 0.0002 across all models and datasets.

• Persistence varies sharply by phenomenon. Reference angles (θ_ref) are largest for refusal datasets (up to 66.52 for LLaMA-3.1-8B on Sorry, 54.26 on Do-not-answer) and smallest for hallucination datasets (11.30 on NaturalQA, 13.12 on TriviaQA). Sycophancy sits in between.

• Transitions are incomplete. After normalization by θ_ref, self-loop rotations are near zero (θ_{φ+→φ+} averages 0.03 ± 0.03; θ_{φ−→φ−} ranges from 0.00 to 0.11), while cross-state rotations average 0.77 ± 0.17 for θ_{φ+→φ−} and mostly remain below 1.0. Hidden states do not fully rotate to the target phenomenon's basis, retaining a trace of their previous state.

• Strong rotation ratios have a prior-driven explanation. The only a-priori phenomenon probability near zero was S-neg at 0.13, indicating a strong drift toward the φ− state; the Sorry dataset also approaches a ratio of 1.0 but has a phenomenon prior of 0.73, which the authors say the geometric analysis may not fully capture.

• Topic inconsistency dissolves the trap. Under D_inconsistent, θ_ref values stay comparable to the consistent setting, but their relationship with the trace weakens to only a marginal increase — geometric separation survives while carryover effects do not.

• First-order history dominates, but longer history matters. The marginal effect Δ_k drops sharply between k = 1 and k = 2, 3, yet remains positive at k = 2, 3, meaning earlier turns still influence the current response.

• Mid-to-late layers carry the signal. Extracting hidden states at 30%, 50%, 85%, and 100% relative depth yields correlations above 0.60 at every depth, with the strongest alignment at 85%.

• Carryover survives when the question is held fixed. Restricting the source dataset from 5,000 to 200 examples and conditioning on the previous state, refusal and sycophancy datasets still show higher carryover than hallucination on LLaMA-3.1-Instruct.

• Closed models show the same probabilistic trends. GPT-5 traces: NaturalQA 1.14, TriviaQA 1.02, S-pos 1.23, S-neg 1.05. Claude-Opus-4.5 traces: 1.10, 1.04, 1.40, 0.99. Refusal datasets were excluded for these models because of API-level filtering.

Methodology in Plain English

The authors first build conversations that look like plausible multi-turn chats. Starting from existing question–answer datasets, they embed each question–answer pair with Qwen3-Embedding-0.6B and order the examples greedily by cosine similarity, so consecutive questions stay on similar topics. This ordered pool is called D_consistent. Randomly shuffling the same pool produces D_inconsistent, where consecutive questions are unrelated. For each model–dataset pair they generate 100 conversations of 20 alternating user–model turns with greedy decoding, and label each answer as phenomenon-present or phenomenon-absent using string matching.

The probabilistic side treats each conversation as a two-state Markov chain and simply counts transitions: how often the model stays in the same state versus switches. The sum of the two "stay" probabilities — the trace of the 2×2 transition matrix — is the carryover score. If the history did not matter, both rows of the matrix would be identical and the trace would be exactly 1.

The geometric side requires access to hidden states. From the residual stream they take the representation of the first answer token, split the collected states into a basis set and a held-out analysis set, and compute the mean hidden state for the phenomenon and non-phenomenon classes. Those two means define a two-dimensional orthonormal basis via Gram-Schmidt. Every held-out state is then projected into this 2D basis, turning each answer into a point (and each transition into a small rotation). The authors use the orthogonal Procrustes closed-form solution to estimate the rotation angle for each transition type, and compare those rotation angles to θ_ref, the static angular separation between the two class means.

To check the labeling, they manually reviewed 50 examples per phenomenon across all three open models, a total of 900 random examples, finding an average misclassification rate of 6.5%.

Why This Matters

The paper reframes a familiar behavioral observation — LLMs get stuck in behavioral ruts — as a measurable structural property of the latent space, and shows the behavioral and representational measures track each other closely. That gives interpretability research a bridge between black-box statistics and white-box geometry, and it suggests that properties of closed models can be inferred probabilistically without any access to weights.

Real-world applications:

• Multi-turn safety monitoring: the transition-matrix trace is a cheap, model-agnostic statistic that could flag deployments where a model is prone to spiraling into repeated refusals or repeated hallucinations.
• Red-teaming and jailbreak defense: the finding that topically incoherent context dissolves carryover effects parallels existing adversarial strategies that use unrelated tokens to break behavioral persistence.
• Model comparison and release gating: the trace and θ_ref provide a per-phenomenon consistency score that can be compared across model families, including API-only models.
• Evaluating representation-based interventions: because the strongest signal appears at 85% relative depth, the geometric analysis points to where steering or probing interventions are likely to be effective.

Industry relevance: teams deploying long-running assistants, agents, or customer-facing chatbots care about exactly this failure mode — an early mistake or an early refusal echoing through an entire session. The framework is lightweight on the behavioral side (no weight access needed) and diagnostic on the representational side, which makes it usable both as a monitoring metric and as an internal audit tool.

Future Directions

• Granular hallucination subtypes. The authors note that hallucination is treated as a single broad label covering plausible-but-incorrect and non-plausible outputs, and suggest that separating subtypes could clarify how specific failure modes map to the latent space.
• Explaining mismatches between the two views. The Sorry dataset's transition ratios approach 1.0 despite a phenomenon prior of 0.73, hinting at nuances the geometric analysis may not fully capture.
• Using geometry to infer internals of closed models. The probabilistic similarity of GPT-5 and Claude-Opus-4.5 to open models is presented only as an initial indication; direct verification of internal geometric traps is not possible without weight access.
• Turning the trap off deliberately. Whether the incoherence effect that dissolves carryover can be exploited constructively — for instance in prompts or decoding strategies that intentionally disrupt persistence — is left open.

Target Audience

Interpretability and LLM-safety researchers who work with hidden-state analysis, mechanistic interpretability, or multi-turn evaluation; NLP practitioners building and auditing conversational agents who want a cheap behavioral metric for conversational drift; and graduate students looking for a worked example of connecting a black-box stochastic model of behavior to a white-box geometric account of the same phenomenon.

Authors’ abstract

How does the conversational past of large language models (LLMs) influence their future performance? Recent work suggests that LLMs are affected by their conversational history in unexpected ways. For instance, hallucinations in prior interactions may influence subsequent model responses. In this work, we introduce History-Echoes, a framework that investigates how conversational history biases subsequent generations. The framework explores this bias from two perspectives: probabilistically, we model conversations as Markov chains to quantify state consistency; geometrically, we measure the consistency of consecutive hidden representations. Across three model families and six datasets spanning diverse phenomena, our analysis reveals a strong correlation between the two perspectives. By bridging these perspectives, we demonstrate that behavioral persistence manifests as a geometric trap, where gaps in the latent space confine the model's trajectory. Code available at https://github.com/technion-cs-nlp/OldHabitsDieHard.