MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents

AI summary

Overview

Research area: LLM agent memory systems, multimodal memory curation, reinforcement learning for agent policies, and LLM/VLM routing.

Technical level: Advanced (assumes familiarity with reinforcement learning, GRPO-style policy optimization, retrieval-augmented agents, and multimodal models).

Scope: This paper introduces MemPilot, an RL-trained orchestrator that decides at query time whether to retrieve from a query-agnostic memory bank or delegate query-specific curation of raw multimodal history to heterogeneous LLMs and VLMs, under controllable performance–cost–latency preferences.

What This Paper Is About

Most agent memory systems build memory before knowing what future queries will ask, which wastes preprocessing effort and discards details that later turn out to matter. Some recent work moves memory processing to query time, but typically commits to a fixed operation or a fixed processing pipeline, leaving flexible control over quality, cost, and latency underexplored. MemPilot's goal is to let a learned policy choose, per query, how much evidence to pull, how to curate it, which model does the curation, and whether to include images, all while trading off answer quality against monetary cost and latency.

Key Contributions

1. A preference-aware runtime memory orchestration framework. MemPilot jointly controls memory access (retrieval from a query-agnostic bank versus query-specific curation of raw history), heterogeneous model delegation, and visual evidence usage, without fixed processing pipelines, predefined budget tiers, or a fixed processing backend.

2. Two adaptations for multi-objective policy training. The authors adapt objective-wise advantage decoupling (normalizing each objective's advantage separately before preference-weighted aggregation) and introduce prefix-based marginal utility estimation, which probes the rollout after each stage to assign fine-grained credit to individual curation steps.

3. A dual-view memory representation. MemPilot keeps a query-agnostic memory bank (interchangeable with off-the-shelf agent memory systems) alongside the raw multimodal history, segmented into fixed-length token chunks, so the orchestrator can trade efficient access against detail preservation.

4. Empirical demonstration across five multimodal agent-memory benchmarks. Results show favorable performance–cost–latency trade-offs across preferences, wider frontiers than trade-off-aware baselines, and behavioral shifts in model selection and visual access as preferences change.

Main Findings

• Controllable operating points rather than a single regime. On the five benchmarks (Mem-Gallery, WorldMemArena, H2HMem, MemEye, MemLens), MemPilot-Perf prioritizes quality, MemPilot-Bal offers an intermediate quality–cost point, and MemPilot-Cost reduces resource use with greater quality sacrifice. For example, with the Qwen3-VL-4B-Instruct answer model, MemPilot-Bal reaches F1 57.26 and L-J 64.27 on Mem-Gallery at cost 1.4e-2, while MemPilot-Cost drops to F1 32.67 and L-J 39.09 at cost 8.2e-3.

• Best LLM-judge scores across benchmarks. MemPilot-Perf achieves the best L-J score across all five benchmarks under both answer models (Qwen3-VL-4B-Instruct and Qwen3.5-9B).

• Runtime adaptation alone does not yield control. GAM, AgeMem, and Mem-T adapt memory processing at inference time but largely expose a fixed operating regime rather than an explicit quality–cost balance, whereas MemPilot spans multiple operating points.

• Multimodal memory can be expensive without adaptive allocation. MIRIX, M2A, and MemVerse often incur substantially higher inference cost without consistently better answer quality; selective curation and delegation let MemPilot achieve stronger performance at much lower cost in most settings, including the out-of-distribution MemEye and MemLens sets.

• Broader frontiers than trade-off-aware baselines. When sweeping preferences, MemPilot forms a stronger performance–cost frontier (Figure 2a) and a similarly broad performance–latency frontier (Figure 2b), largely enveloping LightMem, BudgetMem, and OmniSimpleMem. Cost and latency capture complementary constraints: MemVerse shows competitive quality at relatively low latency but substantially higher monetary cost.

• Coordinated reallocation of computation. As the preference shifts toward lower cost or lower latency, the policy reduces the number of policy steps, the frequency of Curate actions, image access, and policy output tokens, while evidence count also changes adaptively. Model selection shifts from stronger VLMs/LLMs under higher-cost settings toward cheaper model groups under lower-cost settings.

• Every core component matters (ablation, Table 2). Removing LLM/VLM delegation drops the average Judge score from 37.40 to 14.58; removing marginal utility gives 14.82; removing objective-wise advantage decoupling (w/o GDPO) gives 21.94; removing retrieval–instruction decoupling gives 15.35. All ablated variants cost less (e.g., 5.4e-3 to 5.9e-3 versus 1.1e-2), but the quality losses show that simply reducing computation does not yield a good trade-off.

• Compatibility with off-the-shelf memory banks. Swapping the default bank (LLMLingua-2 at a 0.4 compression ratio) without retraining yields separated frontiers: the default bank occupies the lower-cost regime, SimpleMem gives an intermediate profile, and A-MEM incurs higher cost but reaches stronger performance.

Methodology in Plain English

MemPilot keeps two views of the interaction history: a compressed, query-agnostic memory bank plus the raw multimodal history split into 256-token chunks. At answering time, an orchestrator policy (Qwen3-4B-Instruct by default, trained with VeRL) takes a series of steps. At each step it picks one of two action types: Retrieve from the compressed bank, or Curate the raw history. For a Curate step it additionally specifies a retrieval query, how many evidence items to pull (up to five per step), a curation instruction, which external model to call from a pool of LLMs and VLMs, and whether raw images are included. The orchestrator itself stays text-only and never processes image pixels directly; selected external models return textual observations, and after at most a bounded number of steps the policy stops and answers.

Training uses GRPO with three separate rewards: answer quality (e.g., F1), negative monetary cost, and negative latency. Cost is computed from token-level prices for both orchestrator and external calls (orchestrator proxy prices of 0.05 USD per million input tokens and 0.25 USD per million output tokens for Qwen3-4B-Instruct), and latency uses a deterministic proxy estimator for stability. Instead of combining rewards before normalization, each objective is normalized within the rollout group and then aggregated with preference weights (λ_Q fixed at 1). To distinguish useful curation stages from useless ones, the authors truncate rollouts after each stage, generate an answer, and use the resulting quality gain (the marginal utility) as an extra advantage signal for that stage's policy tokens, exploiting KV-cache reuse to keep the overhead low. The same data protocol is applied to all applicable baselines, using a 7/1/2 train/validation/test split with Mem-Gallery, WorldMemArena, and H2HMem merged for training and MemEye and MemLens held out for out-of-distribution evaluation.

Why This Matters

Impact on research. The paper argues for moving agent memory beyond performance-only optimization toward resource-aware runtime adaptation. It reframes memory construction as a preference-conditioned sequential decision problem and shows that fine-grained control dimensions (evidence count, curation instruction, backend model, visual access) can be learned rather than hard-coded into budget tiers.

Real-world applications:

• Long-horizon personal assistants that must recall past conversations and screenshots while operating under latency targets.
• Customer-support agents that need to decide per ticket whether a cheap cached memory lookup suffices or an expensive multimodal review of the raw history is warranted.
• Embodied or GUI agents where visual evidence is costly to process and should only be pulled in when it changes the answer.
• Cost-sensitive deployments that need a single system to serve both quality-critical and budget-critical queries without retraining a separate pipeline.

Industry relevance. The framework is designed to sit on top of existing memory systems (demonstrated with SimpleMem and A-MEM memory banks) and to route across external commercial models via OpenRouter, making it directly relevant to teams balancing inference spend, user-facing latency, and answer quality in production agent stacks.

Future Directions

• Scaling the model pool and pricing dynamics. The heterogeneous pool is accessed through OpenRouter with a configurable model set; how the learned policy behaves as the pool, prices, or providers change is not reported.
• Beyond the latency proxy. The paper uses a deterministic latency proxy rather than direct wall-clock measurement to ensure training stability; whether proxy-optimized policies transfer to real serving latency is unresolved.
• Extending beyond multimodal memory benchmarks. Evaluation covers five multimodal agent-memory benchmarks, with MemEye and MemLens held out; generalization to other domains (tool use, code, long-form documents) is not reported.
• Stronger credit assignment. Prefix-based marginal utility addresses single-stage contributions; whether more sophisticated stage-level or token-level credit assignment further improves multi-step orchestration remains open.

Target Audience

Researchers and engineers working on LLM agent memory, retrieval-augmented generation, multimodal agents, and LLM routing, as well as practitioners who need to tune agent systems along cost and latency budgets rather than optimizing accuracy alone. Readers should be comfortable with reinforcement learning for language models and with the mechanics of agent memory pipelines.

Authors’ abstract

Memory has become integral to the LLM agent ecosystem, supporting information retention and reuse across interactions. However, most existing agent memory systems construct memory in a query-agnostic manner, which can incur unnecessary preprocessing cost and discard details that later prove essential. Recent studies have begun shifting memory processing toward runtime adaptation, but typically specialize in particular operations or fixed processing schemes, leaving flexible control over performance, cost, and latency largely underexplored. To address this challenge, we present \textbf{MemPilot}, a flexible framework that orchestrates on-demand memory curation under different performance--cost--latency preferences. Specifically, we optimize a multi-step LLM policy via reinforcement learning to iteratively choose between retrieving from query-agnostic memory and delegating query-specific curation of raw multimodal history to heterogeneous LLMs and VLMs. The policy jointly controls evidence amount, curation instructions, model selection, and visual access, enabling fine-grained allocation of runtime computation. To optimize this policy under competing objectives, we adapt objective-wise advantage decoupling by separately estimating each objective's advantage before aggregation. Moreover, we introduce prefix-based marginal utility estimation for fine-grained credit assignment across multi-step rollouts. Experiments on five multimodal agent-memory benchmarks demonstrate favorable performance--cost--latency trade-offs across optimization preferences, with preference sweeps yielding broader frontiers than existing trade-off-aware baselines.