Grounding Agent Memory in Contextual Intent

AI summary

Overview

• Research area: Natural Language Processing — specifically agentic memory and long-horizon LLM reasoning.
• Technical level: Intermediate to Advanced. The paper assumes familiarity with retrieval-augmented generation, context windows, and memory-augmented agent architectures.
• Scope: The paper proposes STITCH, an agentic memory system that indexes each step of a task trajectory with a structured "contextual intent" cue and retrieves history by matching intent, plus CAME-Bench, a new benchmark for context-aware retrieval in long, goal-oriented trajectories.

What This Paper Is About

Large language models deployed on long-horizon, goal-oriented tasks must remember things across many steps, but the same entities and facts keep recurring under different hidden goals and constraints. Existing memory systems retrieve evidence that is semantically similar yet contextually wrong — for example, pulling a hotel price from the wrong day of a trip. The paper's goal is to build a memory system whose retrieval is grounded in the latent intent behind each step, and to build a benchmark that actually tests this ability rather than allowing shortcuts.

Key Contributions

1. STITCH (Structured Intent Tracking in Contextual History): an intent-aware, domain-agnostic agentic memory system that induces "contextual intent" online and uses intent compatibility to filter and prioritize retrieved history.
2. Contextual intent as a structured retrieval cue: three components induced without a fixed ontology — thematic scope (the current latent goal / segment label), event type (the action category), and key entity types (which attributes matter in context).
3. CAME-Bench (Context-aware Agent Memory Evaluation Benchmark): a multi-domain benchmark for context-aware retrieval in long, goal-oriented trajectories, with length-stratified subsets and four question types.
4. Empirical validation: STITCH outperforms strong long-context and structured-memory baselines on CAME-Bench, with state-of-the-art results on LongMemEval and competitive results on LoCoMo.

Main Findings

• Largest reported improvement on long trajectories: On CAME-Bench, STITCH outperforms the best baseline by 11.6% (20.5% relative) on the Medium subset and 35.6% (100% relative) on the Large subset. The abstract states STITCH outperforms the strongest baseline by 35.6% across CAME-Bench and LongMemEval, with the largest gains as trajectory length increases.
• Parity on the small subset: STITCH reaches parity with the strongest baseline (gpt-5-mini) on the Small subset of CAME-Bench, while scaling substantially better as lengths grow. Baseline methods degrade sharply as trajectory length increases by factors of 6 (Medium) and 17 (Large).
• Full CAME-Bench scores: STITCH reports 0.810 precision / 0.844 F1 on Small (N=144), 0.665 / 0.682 on Medium (N=168), and 0.616 / 0.592 on Large (N=61). It also reports 0.860 accuracy on LongMemEval O (N=50), 0.860 on LongMemEval S (N=50), 0.800 on LongMemEval M (N=15), and 0.703 accuracy on LoCoMo.
• Thematic scope is the dominant component: Removing thematic scope drops CAME-Bench F1 from 0.844 to 0.463 (Small), 0.682 to 0.257 (Medium), and 0.592 to 0.213 (Large) — by far the largest ablation drop among the four components tested.
• Removing coreference resolution also hurts substantially: F1 falls to 0.578 (Small), 0.489 (Medium), and 0.404 (Large) without the coreference module.
• Event type granularity trade-off: Removing event type lowers Small F1 to 0.753, Medium to 0.527, and Large to 0.273. The authors note event labels help fine-grained lookup but introduce instability on Type 4 (Information Synthesis), because the induced labels are often highly specific (e.g., distinguishing "Check Rates" from "Book Hotel").
• Removing key entity types: F1 falls to 0.735 (Small), 0.511 (Medium), and 0.458 (Large).
• Distinct baseline failure modes: Long-context LLMs stay competitive on the small subset but degrade heavily on the Large subset (described as the "lost-in-the-middle" phenomenon). Structured agentic memory baselines handle simple factual recall adequately but fail to distinguish semantically similar facts that differ by context.
• Label selection is a structural prior, not an exact fit: Question-time labels almost never yield zero overlap with candidate turns. The most frequent failure mode is selecting labels plausible for the broader trajectory but unsupported by the question alone (78.4%), followed by a mismatch in level of abstraction (71.8%).
• Segmentation granularity matters in both directions: With a default window of 50 turns, stress tests at 10 turns (over-segmentation) and 100 turns (under-segmentation) reveal complementary failures — under-segmentation most harms Type 2 (Context-Aware Factual Recall), while over-segmentation most reduces Type 3 (Context-Aware Multi-Hop Reasoning).
• Gains are not backbone-specific: With gpt-4o-mini, STITCH outperforms both the full-context baseline and SeCom on all three partitions; with gpt-4.1-mini, the full-context baseline stays strong on Small but STITCH scales better and surpasses it on Medium and Large.

Methodology in Plain English

The authors start from a cognitive-science idea (Event Structure Theory) that people understand long experiences by grouping them into goal segments and by recognizing recurring kinds of actions. STITCH mirrors this in three online-induced cues:

1. Thematic scope: an LLM looks at the current step, a recent history buffer, and the previous scope, and decides whether the underlying goal has shifted. If not, the step inherits the existing scope label (e.g., "Day 2 Itinerary"). A compressed summary of the current scope is kept online to avoid context overflow.
2. Event type: the system seeds a vocabulary of action labels from the first 50 steps, then for each new step retrieves the 5 most semantically similar labels and asks an LLM to pick the best fit, adding a new label if nothing fits. Every 50 steps it consolidates near-duplicate labels.
3. Key entity types: the system extracts the classes of detail needed (e.g., Price vs. Rating, Metric vs. Hyperparameter) into a growing, periodically consolidated label space.

A coreference module rewrites ambiguous references (e.g., "Book it") using structurally aligned prior steps that share the same thematic scope or compatible event types, resolving them before storage. Each step is stored as a memory snippet combining the raw step, its intent tuple, and a generated canonical summary.

At inference, a query is mapped into a filter configuration over the same label space. Snippets are ranked first by label density — how many intent constraints they satisfy — with semantic similarity used only as a tie-breaker. The top 40 snippets are retrieved within a 4,096-token budget.

For the benchmark, the authors decouple symbolic planning from natural-language generation in a four-stage pipeline: closed-world environment construction, symbolic storyboard planning, storyboard-conditional trajectory generation, and pragmatic refinement (referential remodeling and turn segmentation). This yields automatically annotated questions in two domains — Travel Planning and Debate — across four question types and three length strata, scored with answer-set macro-averaged precision, recall, and F1.

Why This Matters

Impact on research: The paper argues that retrieval quality in long-horizon agents is bottlenecked by cue quality, not just storage capacity. It offers a structured alternative to pure embedding similarity and to graph- or summary-based memory, and it introduces a benchmark that exposes a failure mode the authors say existing evaluations obscure — context-aware retrieval in interleaved, non-turn-taking trajectories.

Real-world applications:

• Personal travel planning agents: distinguishing a hotel's price or rating for Day 1 versus Day 2, and tracking accepted and rejected candidates across turns.
• Deep research and analysis workflows: keeping separate "Model Optimization" phases distinct when the same metrics and hyperparameters recur.
• Multi-agent debate and policy argumentation: retrieving the right evidence snippet for the right rebuttal amid repeated entity mentions.
• Tool-augmented autonomous assistants: maintaining state across deferred and interleaved requests that are resolved only after several intervening steps.

Industry relevance: Any product that runs LLM agents over sessions longer than a context window — customer support, coding assistants, enterprise research tools — faces the same context-mismatch retrieval problem. A domain-agnostic cue schema that requires no fixed ontology is attractive for deployment across heterogeneous task domains, and the authors frame the explicit intent structure as more auditable than black-box embeddings.

Future Directions

• Hierarchical label spaces: The paper's own error analysis shows that flat labels force premature commitment at the wrong level of abstraction; the authors suggest hierarchical structures could support both fine-grained filtering and coarse-grained synthesis.
• Deferring label refinement to query time: Instead of irrevocable one-shot label selection at question time, future work could refine labels when the query arrives.
• Reducing ingestion overhead: STITCH incurs multiple LLM calls per step to build intent tuples, and the buffered update strategy (N=50) adds latency before novel event types are formalized. The authors point to lightweight structural predictors as a path forward.
• Broadening the taxonomy's coverage: The induced event taxonomy is currently optimized for fine-grained factual lookup, so broad synthesis tasks still require aggregating across multiple event types. Whether broader benchmark coverage changes the picture is not reported.

Target Audience

Researchers and engineers working on LLM agent memory, long-context reasoning, and retrieval-augmented generation will get the most from this paper. It is also relevant to practitioners building goal-oriented assistants that must track state across long, interleaved trajectories, and to benchmark designers interested in how evaluation structure (turn-taking, topic partitioning) can mask genuine context-tracking difficulty. Readers should be comfortable with retrieval pipelines, embedding-based ranking, and the mechanics of context windows.

Authors’ abstract

Deploying large language models in long-horizon, goal-oriented interactions remains challenging because similar entities and facts recur under different latent goals and constraints, causing memory systems to retrieve context-mismatched evidence. We propose STITCH (Structured Intent Tracking in Contextual History), an agentic memory system that indexes each trajectory step with a structured retrieval cue, contextual intent, and retrieves history by matching the current step's intent. Contextual intent provides compact signals that disambiguate repeated mentions and reduce interference: (1) the current latent goal defining a thematic segment, (2) the action type, and (3) the salient entity types anchoring which attributes matter. During inference, STITCH filters and prioritizes memory snippets by intent compatibility, suppressing semantically similar but context-incompatible history. For evaluation, we introduce CAME-Bench, a benchmark for context-aware retrieval in realistic, dynamic, goal-oriented trajectories. Across CAME-Bench and LongMemEval, STITCH achieves state-of-the-art performance, outperforming the strongest baseline by 35.6%, with the largest gains as trajectory length increases. Our analysis shows that intent indexing substantially reduces retrieval noise, supporting intent-aware memory for robust long-horizon reasoning.