When Users Change Their Minds: Measuring and Repairing Intent Drift in LLM Agents

AI summary

Overview

Research area: Multi-turn LLM agents — specifically intent tracking and state maintenance when user requirements change mid-conversation, with a new benchmark plus a mitigation harness.

Technical level: Intermediate. The framing is conceptual (active intent as a set of goals edited by Add/Delete/Replace), but the evaluation infrastructure — executable graders, paired statistics, distillation experiments — is research-grade.

Scope (one sentence): The paper defines, measures, and partially repairs "intent drift," the failure mode in which superseded parts of a user's intent still influence an agent's final answer or tool action.

What This Paper Is About

LLM agents often receive a task incrementally: a user asks for something, later revises a parameter, withdraws a constraint, and finally asks the agent to act on whatever remains. The agent must work out which parts of the request are still active. When obsolete instructions keep leaking into the answer, that is intent drift. The paper builds IntentFlux, an executable benchmark that converts verifiable tasks into dialogues with controlled intent changes while keeping the original graders, and StateForge, a harness that maintains an explicit active-intent state before generation.

Key Contributions

1. IntentFlux, a benchmark for intent drift. It converts executable source tasks into controlled multi-turn dialogues where intent changes through Add, Delete, and Replace edits, while preserving each source task's original grader. Stale information is introduced as Variants (plausible values later replaced) and Decoys (plausible constraints later withdrawn), with decoys retained only when obeying them changes the source grader's verdict.
2. A controllable intent-drift stress condition. The calibration uses a monotone construction budget of 1, 3, and 5 for easy, medium, and hard strata, producing a monotonic decline in score.
3. Evidence that history management is insufficient. Two external harnesses — Deep Agents (rolling summarization) and OpenHarness (two-phase compaction) — score 0.354 and 0.392 on General-Test, versus 0.367 for bare multi-turn execution, so compression alone does not reliably recover lost performance.
4. Two transfer paths for state maintenance. On-policy distillation (OPD) trains a smaller tracker sub-role while the base agent stays frozen, and thinking distillation internalizes state-folding behavior into a standalone agent.

Main Findings

• Drift degrades performance monotonically with stale-information load. On a calibration pool of 627 source tasks, mean task score falls from 0.476 (easy) to 0.384 (hard), a 19.3% relative reduction. Realized variants rise from 0.95 to 2.24 and realized decoys from 0.61 to 2.50 across the strata; CAR falls from 0.381 to 0.298.

• Every tested model shows a positive intent-drift gap. Across eight models on General-Test (n = 125 per condition), hard-condition IDG ranges from 0.344 for GLM 5.2 to 0.448 for Qwen3.6 Plus. Clean CAR values range from 0.576 (GLM 5.2) to 0.640 (Qwen3.6 Plus); hard drift CAR ranges from 0.184 (Claude Opus 4.7) to 0.264 (Gemini 3.5 Flash). DeepSeek V4 Pro is the only model whose medium point estimate is slightly below its easy estimate.

• Dialogue length alone does not explain the loss. A turn-matched no-drift control on General-Test scored 0.734, close to the 0.778 single-turn condition and well above the corresponding drift score of 0.469 — a 0.265 gap. The calibration trajectories also lengthened from 13.2 turns on average (easy) to 37.1 (hard).

• Responses differ sharply by task family. With Gemini 3.5 Flash, from easy to hard: LiveCodeBench fell 0.309, GSM8K 0.184, HumanEval 0.178, Spider 0.121, ToTTo 0.042, and SummHay 0.003, while BFCL rose 0.048. An equal-family macro average also declined monotonically, from .497 to .433 to .385.

• StateForge partially repairs intent drift. On General-Test, StateForge raises mean score from 0.367 (bare multi-turn) to 0.467, an improvement of 0.100 (95% CI [0.030, 0.171]), with CAR of 0.368 versus 0.264 for bare multi-turn.

• Exact state information helps but does not close the gap. Replacing the estimated state with the ground-truth final intent raises mean score to 0.549, still 0.229 below the clean single-turn score of 0.778. State-estimation error is therefore material but explains only part of the gap.

• The tracking role does not need base-agent scale. A 9B tracker is statistically indistinguishable from the 122B reference, while 2B and 4B trackers are significantly below it. One OPD iteration raises a 2B tracker from 0.266 to 0.445, compared with 0.428 for a 122B tracker under the same protocol.

• State folding can be internalized into the agent. Thinking distillation raises a standalone 35B agent from 0.296 to 0.461 on the General-Test drift condition without an external harness, though it also modestly lowers the clean-condition score from .745 to .718.

• The ablation does not isolate the explicit state representation. In a component-matched re-run at n = 125, paired differences between each recap baseline and the full harness have 95% confidence intervals that include zero. The oracle condition, by contrast, shows a positive paired difference.

• Interactive transfer shows a partial-credit effect. On Interactive-Test (n = 100 per condition), the no-restate drift condition scored 0.594 versus 0.697 for clean — a 0.103 difference — while the default closing-restatement condition scored 0.720. CAR stayed at 0.190 across clean, default drift, and no-restate drift, and was 0.310 for the ground-truth-oracle intent condition.

• Observed failures are not only explicit retention of deleted content. Trace inspection found that in drift cases containing a Delete edit, agents often omit the deleted goal anyway, while in BFCL a common Replace failure binds a new value to the wrong function-call slot.

Methodology in Plain English

The researchers start from tasks that already have an automatic, verifiable grader — code problems tested against public and hidden tests, math problems graded on the extracted numeric answer, SQL run against Spider databases, data-to-text scored with sacreBLEU, summaries graded by a SummHay LLM grader, and tool actions checked with a BFCL AST check. VitaBench supplies interactive environments (delivery, in-store, OTA, cross-domain) with domain-specific evaluators.

Each source task is decomposed into ordered atomic shards turned into target goals. Because the starting task and its grader are known, the authors can add a scripted sequence of intent changes and know exactly what the final intent should be. A user simulator — DeepSeek-V4-Flash, which also serves as rubric judge — realizes those edits as natural dialogue, maintaining a queue per active goal plus a separate decoy queue. After the final edit, the simulator adds only a neutral request for the final answer; it never restates the final intent.

Performance is measured two ways: mean score (case-micro mean over normalized task-native scores) and CAR, the current-ground-truth-intent alignment rate, defined as the fraction of cases receiving full task-native credit. IDG, the intent-drift gap, is the case-paired clean-minus-drift difference in CAR. Clean and drift conditions share the source case, final intent, grader, and tested model. Key comparisons use case-paired bootstrap confidence intervals.

StateForge works by inserting a separately maintained active-state estimate between the cached dialogue history and the current user input. A tracker folds each new edit into the state, deleting or replacing intent items and also invalidating conclusions derived from them, while distinguishing directly stated intent from derived conclusions. A separate format tracker maintains output conventions, and a final gate can trigger regeneration. A quality-control audit recovered the annotated final intent for 95% of General-Test; an audit of 100 sampled edit-phase simulator utterances found 96% fidelity and 97% coherence.

Why This Matters

Impact on research. The paper isolates a failure mode that existing multi-turn benchmarks measure only in aggregate. By making stale intent grader-consequential and by reporting a turn-matched no-drift control, it gives the field a measurable, reproducible target distinct from generic long-dialogue degradation. The distinction between history management (what to retain) and state folding (what is still valid) is a useful framing for future agent-memory work.

Real-world applications:

• Customer service and support agents where a user changes an order, address, or refund condition mid-conversation and the agent must not act on the discarded version.
• Coding assistants where a developer switches an implementation language or revises a requirement after an initial plan has already been formed.
• Agentic tool use and booking workflows (the paper's delivery, in-store, and OTA VitaBench settings), where a withdrawn constraint could otherwise still trigger the wrong tool call or function argument.
• Enterprise workflow automation, where a deleted compliance or formatting constraint must stop propagating into downstream deliverables.

Industry relevance. The scale results matter commercially: a 9B tracker is statistically indistinguishable from a 122B reference, and OPD lifts a 2B tracker from 0.266 to 0.445 while the base generator stays frozen. That suggests intent-state maintenance can be deployed as a cheap modular component, or, via thinking distillation, absorbed into a 35B agent with no external harness.

Future Directions

1. Decouple the causal factors. The 1/3/5 budget jointly increases superseded values, withdrawn items, and dialogue length, so the calibration supports only a joint stale-information effect. The authors explicitly note that separate marginal effects of variants, decoys, message order, and edit wording remain unidentified.
2. Isolate what actually drives the StateForge gain. The component-matched ablation at n = 125 found paired differences between recap baselines and the full harness with confidence intervals including zero, so the explicit state representation is not established as the dominant source of the improvement.
3. Establish scaling behavior for both transfer paths. The paper states that the scaling behavior of the modular tracker path and the full-agent thinking-distillation path remains open, and warns that the distilled 35B agent has not been shown to learn all state-folding operations or to generalize to every update regime.
4. Broaden the harness and interactive comparisons. The external-harness comparison and Interactive-Test each use a single tested model (Qwen3.5-122B), and the harness comparison replays trajectories rather than running live. The authors also flag that a benchmark can inadvertently supply the state it intends to test, since the diagnostic restatement condition erased the measured interactive partial-credit gap.

Target Audience

Researchers and engineers working on multi-turn LLM agents, conversational memory, and agent evaluation — particularly those building or auditing benchmarks for tool use and instruction following. It is also relevant to practitioners who need agents to track changing user requirements in production, and to anyone studying distillation of state-tracking behavior into smaller models. Readers should be comfortable with benchmark design, paired evaluation statistics, and the distinction between full-credit and partial-credit task scoring.

Authors’ abstract

LLM agents often operate over multi-turn interactions in which user intent changes before execution. We study intent drift: the failure mode in which superseded parts of the user's intent continue to influence the final answer or tool action. We introduce IntentFlux, an executable benchmark that converts verifiable tasks into dialogues with controlled intent changes while preserving their original graders. In a 627-case calibration, mean task score falls from 0.476 to 0.384 as dialogues contain more superseded and withdrawn information. Across eight models, the rate of fully correct solutions is significantly lower when the same final task must be recovered from an evolving dialogue rather than given directly in a single turn. We further introduce StateForge, which explicitly maintains the active requirements before generation. On General-Test, it improves mean task score from 0.367 to 0.467. Providing the ground-truth final state improves performance further but still does not recover single-turn performance, indicating that state-estimation errors explain only part of the gap. These results establish intent drift as a measurable multi-turn failure mode and explicit state maintenance as a partial mitigation.