HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory

AI summary

Overview

Research area: Long-term memory for LLM agents, user modeling, and personalization evaluation.

Technical level: Advanced — assumes familiarity with retrieval-augmented generation, memory architectures, and benchmark design, though the core question is stated plainly.

Scope: A benchmark paper introducing HABIT-Bench, a four-domain evaluation suite (114 users, 37,152 sessions, 3,328 probes) that tests whether agent memory systems can infer user habits from weak, distributed evidence and apply them with correct scope, exceptions, and timing.

What This Paper Is About

LLM agents that interact with users over long periods see repeated choices that hint at habits, but any single observation is only partial evidence: a habit may apply only in some contexts, may have exceptions, may have changed over time, or may reflect an assistant's suggestion the user never accepted. The paper argues that storing past interactions is not the same as being able to infer a behavioral default and decide when it justifies an action. HABIT-Bench is built to measure exactly that gap, using synthetic multi-hundred-session user histories and four-choice decision probes.

Key Contributions

1. A capability definition. The authors formulate "habit memory" as a joint problem: inferring behavioral defaults from incomplete cross-session evidence and determining when the evidence supports applying them, accounting for scope, exceptions, temporal validity, and user endorsement.

2. A constructed benchmark. HABIT-Bench spans four domains (Food, Finance, Software, Travel) built from task-oriented dialogue seeds (CookDial, MultiDoGO, Taskmaster-2) plus controlled habit graphs, totalling 114 users, 37,152 sessions, and 3,328 probes, with private annotations linking each probe to applicable habit states and supporting sessions.

3. Human validation of decision recoverability. Two independent reviewers plus an adjudicator inspected samples (200 Food, 340 Finance, 311 Software, 618 Travel probes) to check that a uniquely supported answer exists given the annotated evidence, with reported A/B agreement of 99.50%, 100.00%, 99.68%, and 92.84% respectively.

4. A controlled comparison with diagnostic controls. Eight memory methods, five retrieval baselines, and two controls are run under a shared answer protocol with Qwen3-235B-A22B and GLM-5.3-Flash, plus oracle controls that isolate evidence access from decision-making, and a cost/sensitivity analysis. The evaluation covers 308 configurations.

Main Findings

• Habit induction stays near one-third accuracy. Across the eight memory methods, four-domain macro accuracy ranges from 27.01% to 33.06% with Qwen3-235B-A22B and from 30.42% to 33.61% with GLM-5.3-Flash, against a 25% uniform-random baseline. The best memory method reaches 33.06% (A-MEM, Qwen) and 33.61% (A-MEM, GLM).

• Memory pipelines do not reliably beat simple retrieval. Best memory macro accuracy (33.06%) is close to BM25's 32.50% with Qwen, and 33.61% versus 33.21% for dense retrieval with GLM. The GLM Full Memory compressed-history control reaches 41.98%, exceeding all eight memory methods.

• Benefits are strongly domain-dependent. Under Qwen3-235B-A22B the best memory method improves over No Memory by 20.32 percentage points on Food and 13.07 points on Travel. In Finance, all eight methods fall below the 23.54% No Memory point estimate; in Software the largest gain is 1.03 percentage points. With GLM, the best memory-method accuracies are 28.29% on Finance and 25.44% on Software, versus 54.62% on Travel.

• Supplying the right evidence is not enough. Oracle Evidence reaches only 39.26–58.25% under Qwen3-235B-A22B (and 29.82–53.33% under Qwen3-8B), while Oracle Habit State reaches 98.41–99.26% (92.54–98.31% with Qwen3-8B). Access to annotated observations still leaves substantial decision errors.

• Evidence access and correct answering are separate. For six source-attributed Qwen3-32B methods, Hit@5 ranges from 37.77% to 41.44% and annotated-evidence recall from 11.17% to 13.89%; joint answer-plus-evidence accuracy is 12.92–16.20%, and Clean Grounded@5 is 5.49–8.20% on its eligible subset.

• Larger models do not uniformly help. A-MEM rises from 29.41% with Qwen3-4B to 33.06% with 235B-A22B, but LightMem reaches 28.14% at 14B, falls to 25.04% at 32B, and recovers to 27.01% at 235B. The authors note these comparisons also change the memory-stage model and reader protocol.

• Retrieval tuning and chunk size are not the bottleneck. In a Qwen3-8B study over 32 users and 954 probes, accuracy ranges from 24.83% to 31.71%; raising top-k from 2 to 20 improves A-MEM from 29.23% to 31.21% and Letta from 28.53% to 31.71%, while larger ingestion chunks do not consistently help.

• Memory construction dominates cost. A-MEM construction sums to 54.59 hours and 107.80 million LLM tokens per-user across the cohort, versus 13.24 minutes and 3.39 million tokens for queries.

Methodology in Plain English

The team could not use real people's multi-year histories, so they built lifelines: synthetic users whose long interaction histories are generated by pairing real task-oriented dialogue seeds with hand-designed "habit graphs." A habit graph specifies a user's default behavior, the contexts in which it applies, its boundary cases, one-off exceptions, when it changes, and whether the user actually endorsed it. Relevant sessions are scattered through hundreds of ordinary sessions so that no single observation settles the question.

Each test item is a four-choice question: a new request plus four candidate actions, all of which satisfy the request's hard constraints but differ in how they use the habit. One is correct; others apply an outdated version, overextend a default beyond its scope, or treat an unaccepted assistant suggestion as a decision. Answers are scored by exact match, avoiding an LLM judge.

Every memory system processes each user's sessions in chronological order, isolated per user, and can only see history up to a probe's visibility cutoff. At query time it retrieves context under its own budget, and the same base model answers for all methods in a given setting, so differences come from the memory pipeline. Beyond standard evaluation, the authors add oracle controls (feeding gold evidence, or the gold habit state) to separate "could not find it" from "could not decide," and they measure retrieval hits, evidence recall, build time, and token usage.

Why This Matters

Impact on research. The paper reframes memory evaluation away from "can you recall this fact" toward "does the evidence justify this action." The oracle results are the sharpest point: even with the right evidence handed over, accuracy stays well below 60% in most domains, which suggests the bottleneck is decision-making over reconstructed habits, not storage. The finding that elaborate pipelines do not consistently beat BM25 or compressed history is a direct challenge to memory-architecture research.

Real-world applications:

• Personal assistants that need to know when a user's usual choice still applies (e.g., a standard travel booking) versus when a recent exception overrides it.
• Financial or enterprise workflow agents that must reconstruct a user-confirmed procedure, track when it changed, and avoid resuming a superseded process.
• Customer support and technical help systems where similar errors occur in different environments and earlier unfinished cases must be continued correctly.
• Recommendation and personalization systems that must withhold personalization when evidence is weak or a suggestion was never accepted.

Industry relevance. The cost data matters for deployment: memory construction consumed 54.59 hours and 107.80 million tokens for A-MEM on a 32-user cohort, against 13.24 minutes for queries. If that upfront investment yields roughly retrieval-level accuracy on habit-driven decisions, teams have a concrete reason to question expensive memory pipelines in favor of simpler retrieval or compressed history.

Future Directions

• Representations that preserve relationships. The authors call for memory that stores a default together with its supporting observations and validity conditions, rather than a compact account of what happened.
• Better decision procedures over retrieved evidence. Since Oracle Evidence leaves large errors, work is needed on how an answerer resolves competing, partially applicable evidence at a target time.
• Isolating what causes the Finance and Software failures. The paper notes that domain comparisons cannot identify which combined requirement (multi-habit composition, temporal resolution, or distinguishing commitments from suggestions) drives the errors, and proposes capability slices as a partial remedy.
• Scale and protocol-controlled ablations. The model-size comparison changes both the memory-stage and answer models and uses an earlier reader protocol for smaller Qwen runs, so a clean scale-only study remains open.

Target Audience

Researchers and engineers building long-term memory or personalization for LLM agents; benchmark designers interested in human-validated, oracle-controlled evaluation; and practitioners deciding whether an elaborate memory pipeline is worth its build cost. Readers without a background in retrieval or agent memory will find the framing accessible but the experimental tables dense.

Authors’ abstract

In long-term interactions with large language model (LLM) agents, users reveal habits through their choices and actions. These habits capture implicit user information that agents must infer across conversations. However, individual observations may provide only partial evidence about a habit or when it applies. Retaining past interactions alone does not ensure that agents can infer habits and apply them appropriately. We introduce HABIT-Bench to evaluate how well agent memory systems infer habits from weak evidence and apply them to later requests. We construct synthetic user histories from task-oriented dialogue data and controlled habit graphs. The histories span hundreds of sessions across food, finance, software, and travel. Four-choice tasks assess agents' ability to infer user habits and apply them appropriately. They include temporary exceptions, changes in habits, and suggestions that users have not accepted. Sampled tasks undergo independent human review. We evaluate eight memory methods with Qwen3-235B-A22B and GLM-5.3-Flash. The best-performing memory method achieves 33.61\% macro accuracy across the four domains. Performance remains limited even when annotated evidence is supplied directly. These results expose a persistent gap between access to past interactions and the ability to infer and appropriately use user habits.