HiVe-Mem: Interleaving Horizontal Memory Expansion and Vertical Evidence Composition for Multimodal Agents

AI summary

Overview

Research area: Multimodal agent memory — long-horizon retrieval and evidence selection over interaction histories that mix text and images. Published on arXiv (2611.12872v1, cs.CL) by authors from Nanyang Technological University, Tsinghua University, Sun Yat-sen University, and Aumovio Singapore.

Technical level: Intermediate. Readers should be comfortable with retrieval, graph-based indexing, and reinforcement learning basics; the paper's own framing is intuitive, but the selector training (PPO) and the anchor-graph construction assume some background.

Scope (1 sentence): The paper proposes HiVe-Mem, a two-part memory framework that widens which past episodes get retrieved (via a graph of textual and visual anchors) and then decides, per retrieved episode, which evidence formats to actually show the model — reporting gains on three multimodal memory benchmarks with roughly 90% lower token consumption than the most efficient competing multimodal baselines.

What This Paper Is About

Multimodal agents accumulate long histories of text and images, and existing memory systems typically retrieve past entries by semantic similarity over compressed summaries, then hand the model a fixed representation of whatever was retrieved. That design can miss relevant episodes that are only loosely connected to the query, and it cannot match the level of detail to what each query actually needs. The goal of HiVe-Mem is to broaden recall beyond direct semantic matches while adaptively choosing, for each retrieved memory, which evidence forms (summary, caption, grounded visual view, raw image, raw interaction text) to expose.

Key Contributions

1. A joint retrieval-and-utilization framework. HiVe-Mem is presented as improving both which memories are retrieved and how each retrieved memory is presented, rather than fixing one stage and optimizing the other.
2. Horizontal Memory Expansion. Memory episodes are organized into an undirected affinity graph built from VLM-generated textual anchors and visual anchors, with edges weighted by IDF. Retrieval starts from semantic seeds and expands to one-hop neighbors scored by the specificity and query-relevance of the anchors they share with the seeds.
3. Vertical Evidence Composition. Each memory episode retains five evidence forms (r_i raw text, s_i compact summary, c_i caption, o_i raw image, g_i grounded visual view). A lightweight MLP selector takes the query and summary embeddings and outputs binary inclusion decisions per evidence form, trained with PPO on a reward combining answer quality and normalized token cost.
4. Empirical evaluation across three benchmarks and two base models, including a transfer setting where the selector trained on Qwen3-VL-4B-Instruct is applied to GPT-5-mini without retraining, plus ablations, budget sweeps, latency measurement, and per-query-type analysis.

Main Findings

• Consistent average gains over multimodal baselines. With Qwen3-VL-4B-Instruct, HiVe-Mem reaches average F1/LJ of 32.32/57.96; the best multimodal baseline averages in the table are M2A at 28.42/43.89 (F1) and MMA at 28.19/48.35 (LJ). With GPT-5-mini, HiVe-Mem reaches 35.94/64.69, versus M2A at 34.04/57.93 and MemVerse at 32.95/60.71.
• Per-benchmark results. Qwen3-VL-4B-Instruct: Mem-Gallery 49.44 F1 / 71.55 LJ at 0.491M tokens; WorldMemArena 23.66 / 68.18 at 0.991M; H2HMem 23.87 / 34.17 at 0.492M. GPT-5-mini: Mem-Gallery 53.31 / 72.45 at 0.367M; WorldMemArena 24.80 / 79.32 at 1.021M; H2HMem 29.71 / 42.29 at 0.856M.
• Large token savings. Average consumption is 0.658M tokens (Qwen) and 0.748M (GPT-5-mini), against 7.812M and 7.350M for the most efficient competing multimodal baselines — reductions of 91.6% and 89.8%. For contrast, multimodal baselines such as M2A and MMA consume up to 33.034M tokens on Mem-Gallery.
• The selector transfers across base models. The GPT-5-mini results use the selector trained only on Qwen3-VL-4B-Instruct, with no retraining.
• Removing vertical composition hurts most. Replacing adaptive composition with summary-only representations drops average LJ from 57.96 to 44.47. Removing raw text drops it to 49.68; removing raw images to 51.02.
• Horizontal expansion and IDF weighting both matter. Disabling horizontal expansion lowers LJ to 56.51; removing visual anchors from the graph gives 57.30; removing IDF weighting gives 56.55.
• Smaller evidence-form ablations. Removing compact summary gives 56.86, image caption 56.12, and grounded visual view 57.19 — each below the full 57.96 but above the summary-only setting.
• Budget control works as intended. Increasing λ shifts the policy toward smaller evidence sets: raw-history selection drops substantially, and raw images are selected relatively rarely yet account for an increasing share of consumed tokens because of their per-selection cost. Varying λ traces distinct quality-versus-token operating points.
• Low runtime overhead. Time to first token on Mem-Gallery is 1.03 seconds, slightly above the simplest retrieval baselines but substantially below other multimodal systems that perform agent coordination, memory decomposition, or evidence processing at answer time.
• Hyperparameter robustness. Performance is stable across a wide range of graph degree K_edge, with the best result at the default K_edge = 4. With K_ret + K_exp = 7 fixed, LJ peaks around a 5:2 split, is similar at 4:3, and degrades when expansion takes too large a share of the budget.
• Evidence selection tracks query type. Text-dominant queries almost never select grounded visual views or raw images, while visual-dominant and cross-modal queries select visual evidence more often. Even for visual-dominant queries, raw images are selected for only 35.2% of retrieved memories; compact summaries are frequently selected across all three query types.
• Grounded views are cheaper than raw images. The paper reports that grounded visual views reduce token consumption by approximately 20–40% compared with the corresponding raw images, depending on benchmark.

Methodology in Plain English

Building memories. Interaction history is split into contiguous episodes of roughly 256 tokens, extended to the nearest natural conversational boundary rather than cut mid-dialogue. A vision-language model processes each episode once to produce a compact summary, a set of textual anchors (entities, events, topics), and a set of visual anchors (objects, scenes, activities, attributes). Each memory also keeps raw text, a caption, the raw image, and a "grounded visual view" — a crop of the context-relevant object, produced by having a VLM predict the target object's spatial coordinates and then cropping that region.

Widening retrieval (horizontal). Memories become nodes in an undirected graph. Two episodes are linked when their anchor sets overlap, including cross-modal cases where a concept appears as text in one episode and as a visual anchor in another. Overlapping anchors are weighted by inverse document frequency so that distinctive cues count more than ubiquitous ones, and each node keeps at most K_edge edges. At query time, K_ret seeds are found by comparing the query to summaries; their one-hop neighbors are scored by summing, over shared anchors, the anchor's IDF weight times its embedding similarity to the query. The top K_exp neighbors join the retrieved set.

Choosing what to show (vertical). For each retrieved memory, a small MLP takes the concatenated query and summary embeddings and predicts one inclusion probability per evidence form in a single forward pass. Sampling those probabilities yields the binary composition used for that memory. Because the decision is discrete and its value depends on the final answer, the selector is trained with PPO against a reward of answer quality minus λ times a normalized token-cost term; cost is normalized with sliding-window 5th/95th percentile clipping and rescaled by the ratio of reward to cost standard deviations. Setting λ = 0 by default, larger λ trades answer quality for cheaper evidence.

Evaluation. Three benchmarks (Mem-Gallery, WorldMemArena, H2HMem) with token-level F1, an LLM-as-a-judge score using GPT-4o-mini, and total token consumption. Baselines include text-centric systems (LightMem, SimpleMem, BudgetMem) and multimodal systems (M2A, MIRIX, MMA, MemVerse), all under a consistent Top-7 retrieval setting; data is split 6:2:2 and retrieval embeddings come from Qwen3-VL-Embedding-2B.

Why This Matters

Research impact. The paper argues that multimodal memory research has invested heavily in how memories are built and stored while leaving how they are used at inference time comparatively unexamined. Its two-axis framing — coverage versus granularity — gives a reusable decomposition, and the finding that summary-only memory collapses LJ by more than 13 points (57.96 to 44.47) is a direct challenge to the common practice of retrieving compressed abstractions. The ~90% token reduction at equal or better accuracy also reframes efficiency as a property of evidence selection, not just of memory compression.

Real-world applications:

• Assistive and companion agents that must recall earlier conversations and previously shown photos or screenshots over weeks of interaction.
• Customer-support and service copilots where a ticket history mixes chat logs with uploaded images and only some queries need the original visual detail.
• Robotics and embodied assistants replaying past observations where a cropped, task-relevant view suffices instead of the full frame.
• Personal knowledge and lifelogging tools that answer questions over large archives of annotated screenshots or camera captures under tight context budgets.

Industry relevance. The cost profile is the headline for deployment: memory systems that retrieve aggressively can dominate serving cost, and here the choice of what to include is learned directly against a token budget rather than hand-tuned. The λ knob gives operators an explicit quality-versus-cost dial without rebuilding the memory store, and the 1.03-second time to first token suggests the adaptivity does not come at the price of interactive latency.

Future Directions

• Whether the evidence space should grow. The authors state explicitly that the five evidence forms are a default instantiation, not a fixed space, and that the composition formulation can accommodate additional textual, visual, or derived representations — which ones are worth adding is left open.
• Multi-hop or deeper graph expansion. Expansion is currently limited to one-hop neighbors of the semantic seeds; the sensitivity results show performance falling when too few seeds remain, raising the question of whether deeper traversal with better seed diversity could help.
• End-to-end or joint training. The selector is trained by RL while the memory construction, anchors, and retriever are fixed; whether co-training these components improves the reward trade-off is untested.
• Broader transfer and benchmark coverage. Transfer is demonstrated only from Qwen3-VL-4B-Instruct to GPT-5-mini on three benchmarks; generalization to other model families, longer histories, and other domains is not reported.

Target Audience

Researchers and engineers working on LLM/VLM agent memory, long-context retrieval, and multimodal RAG will get the most from this paper, particularly those who already run a retrieval pipeline and are deciding how much retrieved context to actually feed the model. It is also useful for practitioners optimizing inference cost in production agents, and for students looking for a clean example of combining graph-based retrieval with RL-trained, query-adaptive context selection. Readers without background in retrieval or policy-gradient methods will need to consult the cited prior work for the PPO details, which are placed in the appendix.

Authors’ abstract

As multimodal agents accumulate increasingly long interaction histories, effectively retrieving and utilizing past information becomes critical for long-term interaction. Existing memory systems typically rely on semantic retrieval over compressed memories and expose predefined representations, which may miss relevant memories and overlook heterogeneous evidence needs across queries. We introduce \textbf{\hivemem}, a multimodal memory framework that interleaves…