When LLMs Forget What They Believe: Characterizing State Integrity Collapse in Long-Horizon Dialogues

AI summary

Overview

Research area: Natural Language Processing — multi-turn instruction following, dialogue-state tracking, and LLM reliability evaluation.

Technical level: Intermediate (readers should be comfortable with benchmark design, binary judged metrics, and basic statistics such as Wilson confidence intervals and Cohen's/Fleiss' kappa).

Scope: The paper defines and measures "State Integrity Collapse" — the failure mode in which a model can restate a persistent constraint but stops obeying it as a dialogue grows — and introduces StateTrack-Bench, a 125-probe-per-model diagnostic benchmark across 7 model snapshots, 5 constraint categories, and 5 dialogue depths.

What This Paper Is About

Deployed assistants are expected to keep honoring standing rules — a word limit, a persona, a confidentiality boundary, a patient-specific medical restriction — across many turns. The authors observe that a model may repeat such a rule perfectly when asked and then violate it one turn later, so a single-turn instruction-following score says little about long-horizon reliability. The paper's goal is to separate this "recall but not compliance" failure from ordinary forgetting, measure how often it occurs across models and constraint types, and test a simple intervention for it.

Key Contributions

1. A formal definition and metric for State Integrity Collapse (SIC). The authors distinguish SIC from forgetting and operationalize it with a forked audit computed per instance, splitting failures into SIC+ (behavioral drift: the constraint is recalled but not applied) and SIC− (declarative fade: the constraint is applied but cannot be stated). Because the comparison is made per instance before averaging, opposite dissociations cannot cancel out.

2. StateTrack-Bench, a controlled dialogue suite with three phases (setup, distractor, test) covering five constraint categories — numerical, format, role/persona, factual-memory, and logical-rule — evaluated at turn depths 3, 5, 7, 10 and 12. The core benchmark yields 125 scenario-depth probes per model (25 per category) and 875 primary scored outputs across all models.

3. A "strict joint retention" scoring policy in which a response earns credit only if it satisfies both the immediate user request and the persistent constraint, plus supporting audits: split task-versus-constraint scoring, signed SIC components, a Gemini-excluded depth study, dynamic update/revocation, a two-constraint scope check, and prompt-style sensitivity analysis.

4. Structured State Anchoring (SSA), presented explicitly as a diagnostic intervention rather than a novel algorithm, which recovers up to 24.4 points on immutable constraints, together with a dynamic-state audit showing that a stale SSA ledger enforces a superseded rule in 74.3% of cases while updating the ledger reduces stale enforcement to 0.0%.

Main Findings

• Retention spans 62.4% to 100.0% across models. Best strict joint retention is DeepSeek-v4-Flash at 100.0%, followed by GPT-5.4-mini (98.4%), DeepSeek-v4-Pro (96.0%), Qwen3.6-Plus (95.2%), GPT-5.4 (94.4%), and Claude-Sonnet-4-6 (90.4%). Gemini-3.1-Pro is the outlier at 62.4%, dropping 10.0 points from turn 3 (64.0) to turn 12 (54.0).

• Failures are overwhelmingly behavioral drift, not forgetting. Across the full test set, 88.6% of the 79 observed failures are integrity failures (Wilson 95% CI: 79.7–93.9), with only 9 counted as memory failures. Excluding Gemini-3.1-Pro, the diagnosis holds at 84.4% of the remaining 32 failures (Wilson 95% CI: 68.2–93.1), while the all-probe SIC+ rate falls from 8.0% to 3.6%.

• Symbolic constraints are more brittle than semantic ones. In the prose category averages, factual-memory constraints score highest (93.7%) and logical-rule constraints lowest (88.6%), with numerical and format both at 91.4% and role/persona at 89.7%. The category-mean row of Table 4 lists a somewhat different profile (numerical 87.7, format 85.5, logic 83.5, factual 95.4, persona 95.2), and Section 5.2 describes symbolic categories as clustering 8–12 points below semantic ones.

• Aggregate scores hide turn-level instability. Turn-wise averages are 90.9, 91.9, 90.7, 91.4 and 90.0 at turns 3, 5, 7, 10 and 12 — no smooth monotonic collapse. Gemini-3.1-Pro shows a non-monotonic profile (64.0, 63.0, 63.0, 68.0, 54.0), while DeepSeek-v4-Flash is flat at 100.0 across all five depths.

• The dissociation persists at greater depth. A 216-probe confirmatory analysis over the other six models, 12 symbolic scenarios and depths 12, 30 and 50 yields compliance of 94.4%, 95.8% and 90.3%, with depth-50 SIC+ at 9.7%. Five of the six models show at least one recalled-but-violated case. For numerical constraints at depth 50, recall remains 100.0% while compliance falls to 79.2%, giving SIC+ = 20.8%.

• Reinstatement helps but does not erase the collapse. Among models with observed failures, full recovery after reinstating the constraint ranges from 53.2% (Gemini-3.1-Pro, 47 collapsed scenarios) to 100.0% (GPT-5.4-mini and DeepSeek-v4-Pro), with a no-recovery tail persisting for the weakest model (19.1%).

• Results are robust to decoding and prompt reformulation. Re-evaluation at τ ∈ {0.0, 0.3, 0.7} produced a maximum shift of 2.1 points with model ordering preserved. Rank order is stable: Kendall's τ = 0.81 (p < 0.01) between early- and late-turn orderings and τ = 0.90 (p < 0.005) between symbolic and semantic category orderings.

• Judge and distractor quality checks pass. Inter-judge agreement across three architecturally distinct judges on a 200-output subset was Fleiss' κ = 0.89, with no significant self-preference bias (McNemar's test, p > 0.15 for all model–judge pairs). Human majority vote agreed with the automated judge on 93.3% of 120 re-scored outputs. Mean Jaccard similarity between distractor turns and constraint statements was 0.040, and mean naturalness rating was 4.2 (σ = 0.6) on a 1–5 scale.

Methodology in Plain English

Each benchmark item is a scripted conversation in three stages. In the setup turn the user states a rule that is meant to persist (for example, "recommend only Spanish restaurants for the rest of this search"). In the distractor turns the user asks related but different questions that never restate the rule, so the model cannot succeed by simply pattern-matching a repeated instruction. In the test turn the model must answer a new request while still honoring the original rule. Distractors were generated from a template system that enforces topical relevance, no lexical overlap with the rule (verified as Jaccard < 0.05), and rising complexity, so later turns create natural pressure to drop the constraint without any adversarial intent.

Scoring is deliberately strict. A single automated judge (GPT-5.4 at temperature 0, with the full dialogue history, the constraint specification, and the model's response) returns a binary label, and credit is given only when both the immediate request and the standing rule are satisfied. Numerical and format cases can often be checked deterministically; semantic cases rely on the fixed judge prompt. Agreement with human annotators was checked on stratified samples, with Cohen's κ of 0.87 for role/persona and 0.82 for logical-rule labels between two annotators, and κ of 0.91 and 0.84 respectively between the human consensus and the automated judge.

To tell forgetting apart from drift, the authors replay each failed case on a copy of the dialogue prefix, using two non-interacting branches: a compliance branch that asks the next task question, and a recall branch that asks the model to restate the rule. The recall probe is never inserted into the scored compliance trajectory. A case where recall succeeds but compliance fails is counted as an integrity failure; a case where compliance succeeds but recall fails is a declarative fade. Comparing per instance before averaging prevents the two directions from cancelling.

Finally, the authors test a mitigation, Structured State Anchoring, which decomposes into prepending the active state, verifying against a concrete checkable predicate derived from the constraint, and measuring the residual enforcement gap. They also run a dynamic-state audit with 7 models and 5 scenarios to see what happens when the anchored state becomes stale.

Why This Matters

Impact on research. The paper argues that instruction-following evaluation is too weak a proxy for multi-turn reliability, and that retrieval-style fixes (longer context, memory summarization, RAG) address the wrong failure: if 88.6% of observed failures involve a rule the model can still state, then recovering the instruction text does not guarantee that it governs the next answer. This reframes persistent-state adherence as a state-application problem rather than a context-access problem, complementing long-context position-effect work.

Real-world applications:

• Clinical and legal assistants bound by standing constraints, such as avoiding medications contraindicated for a patient's condition or preserving a confidentiality rule, where a model that can restate the rule but violates it in the next response produces a failure invisible to standard evaluation.
• Customer-facing agents with fixed formatting or word-limit requirements, where exact structural commitments are the first thing users notice breaking.
• Persona and brand-voice systems, where the reported 71% of role/persona failures attributed to "affective override" under emotionally charged messages means the assistant quietly reverts to a generic voice.
• Compliance monitoring pipelines, where the finding that symbolic constraints fail silently argues for prioritizing them in production monitoring rather than relying on aggregate quality scores.

Industry relevance. Model selection by single-turn benchmark scores can be misleading: recency and scale did not predict multi-turn reliability here, and one snapshot lost 10 points between turn 3 and turn 12. The prepending-only SSA variant is described as a practical safeguard at roughly 13% token overhead — but only when the active-state registry is correct, since a stale ledger enforced a superseded rule in 74.3% of audited cases.

Future Directions

• Larger-scale benchmarks. The 25-probe-per-category size was intentionally modest and limits fine-grained within-category inference; the authors call for hundreds of scenarios per category to enable precise effect estimation, and note that a 40-scenario subset was used for depths 20–50 and a 30-scenario pilot for Chinese and Japanese.
• Beyond LLM judges. Despite κ = 0.89 inter-judge agreement, the authors flag reliance on LLM-based evaluation and call for larger-scale human evaluation plus investigation of whether judge biases correlate with specific constraint types or dialogue structures.
• Multilingual and typologically diverse coverage. Languages with richer morphology (Finnish, Turkish) or different discourse structures (Arabic) remain untested, and the existing cross-lingual pilot is described as too small to support language-universal claims.
• Dynamic and adversarial state management. The benchmark uses one immutable constraint per dialogue under cooperative conditions; open questions include multi-constraint interference and conflicting priorities (the two-constraint audit reached 76.5% joint success over 34 valid outputs), user-initiated revocation, adversarial social engineering, and multi-session settings where context is summarized or truncated.

Target Audience

Researchers and engineers working on LLM evaluation, multi-turn agent reliability, and instruction-following benchmarks; practitioners who deploy persistent assistants with standing constraints (safety, medical, legal, formatting, persona); and teams building memory or state-management layers who need to know whether a recalled rule actually governs generation. The paper is written for readers with intermediate familiarity with benchmark methodology and judged metrics; the appendices assume some interest in statistical validation and prompt design.

Authors’ abstract

Large language models can repeat an instruction while failing to keep obeying it. We study this recall-compliance dissociation in long-horizon dialogue and define State Integrity Collapse: a model preserves a stated constraint in memory but stops applying it to later turns. We introduce StateTrack-Bench, a controlled suite of conversations covering numerical, format, role/persona, factual-memory, and logical-rule constraints. Across seven provider-pinned model snapshots and five dialogue-depth probes, strict joint retention ranges from 62.4% to 100.0%, with degradation concentrated in symbolic constraints and in one especially brittle model. A forked recall audit shows that 88.6% of observed failures are behavioral-drift failures rather than ordinary forgetting; excluding the brittle outlier preserves this diagnosis (84.4% of failures). Extended evaluations at depths up to 50 turns confirm that the dissociation persists beyond the core benchmark, while Structured State Anchoring recovers up to 24.4 points on immutable constraints. The results show that reliability cannot be inferred from single-turn instruction following alone; evaluation must track whether remembered constraints remain operational as dialogue state grows. Code and artifacts are available at: https://github.com/xzzvsxd/StateTrack.