MobileMem: Learning from a Year of Mobile Experiences

AI summary

Overview

Research area: Artificial intelligence agents with long-term memory, specifically on-device memory for personal mobile assistants, and the benchmarks used to evaluate them.

Technical level: Advanced. The paper assumes familiarity with LLM agents, knowledge graphs, memory architectures, multimodal data synthesis, and retrieval-style memory systems.

Scope: The paper introduces MobileMem, a benchmark suite and data-synthesis framework for studying long-term, heterogeneous, multimodal, and continually evolving memory built entirely on mobile devices.

What This Paper Is About

Existing agent-memory research assumes cloud storage, centralized databases, and infinite capacity, but the primary deployment target for next-generation assistants is mobile devices (smartphones, AR glasses, wearables, in-car assistants), which generate personal, multimodal, cross-app data under tight storage, power, latency, and privacy constraints. Existing long-term memory benchmarks do not capture this setting, because mobile experiences are heterogeneous, multimodal, temporally continuous, and deeply personal. The paper's goal is to define what on-device memory should look like, provide a synthesis pipeline that builds year-scale evaluation data from user prior knowledge, and instantiate two benchmark scenarios (text-only MobileMem and multimodal MobileMem-Omni) that test multi-hop and temporal reasoning, knowledge updating, implicit preference inference, and abstention.

Key Contributions

1. A benchmark suite for on-device long-term memory. MobileMem (textual) and MobileMem-Omni (multimodal) are built around a year-scale collection of mobile experiences and cover the full lifecycle of user experiences through structured personal knowledge graphs representing entities, relations, and temporal dynamics across modalities and applications.

2. KEME, a knowledge-guided experience synthesis engine. KEME (Knowledge-guided Experience synthesis for evolving MEzmory) constructs coherent, temporally consistent long-horizon trajectories from user prior knowledge by alternating top-down temporal planning with bottom-up experience evolution, and it derives multi-hop question-answer pairs from those trajectories rather than statically assembling them.

3. A composite on-device memory architecture proposal. The paper argues that mobile memory should not be a single monolithic store but two collaborating components: a system-level memory layer acting as a shared "blackboard" (analogous to blackboard architecture) that accumulates cross-application knowledge, and application-specific memories acting as cognitive offloading / knowledge sources that filter and expose only meaningful events through a standardized protocol.

4. Two instantiated benchmark scenarios with distinct observation channels. MobileMem covers apps tightly integrated with the system-level assistant via predefined templates (records forwarded as structured memory events), while MobileMem-Omni covers apps that are not directly connected, where users share screenshots of important interactions, adding visual reasoning and cross-modal retrieval challenges.

Main Findings

• Cloud-based memory assumptions do not transfer to mobile. The paper identifies five challenges specific to on-device memory: memory explosion from daily events, images, long videos, and continuous voice; fragmentation across modalities and apps without a unified semantic structure; continuous evolution of interests and habits requiring adaptation without catastrophic forgetting; severe constraints on storage, power, latency, and compute; and privacy risks that necessitate local-first processing.

• The observability gap is the central technical problem. Much mobile activity occurs in third-party apps (browsing, reading, shopping, finance) and is only partially observable to the memory layer. The paper argues that screenshot-based perception used by systems such as MRIXI and MineContext treats apps as passive observation targets, introducing redundancy, privacy concerns, and processing overhead.

• Applications already function as specialized memory systems. The paper's conceptual finding is that apps already store interaction histories and expose retrieval (analogous to lexical retrieval such as BM25) and consolidation behaviors (for example, finance apps aggregating transactions into monthly income and expenditure summaries), which motivates a composite rather than monolithic memory design.

• MobileMem's textual scenario is grounded in real but limited user data. MobileMem defines a user profile schema and recruits two volunteers whose profiles are collected through interviews, supplemented with trajectories from the volunteers' OPPO smartphones. Collection records only app usage statistics and associated metadata, explicitly avoiding raw interaction content, and includes contextual signals such as geographic location, Bluetooth connectivity, and network status. Seven app types are covered: Bill, Voice Recorder, Screen Memo, Document, Note, Calendar, and To-Do List.

• Sessions are segmented by an explicit temporal rule. Messages are grouped into user-app interaction sessions according to timestamps; when the time interval between two adjacent messages exceeds two days, the preceding ungrouped messages are segmented into one session.

• MobileMem-Omni expands scale and modality. It recruits eight volunteers plus eight additional virtual user profiles generated by prompting LLMs with the real profiles as demonstrations, and enriches prior knowledge with a personal relationship graph whose nodes are people with textual descriptions, edges are social relationships, and whose frontal portrait images are generated with a text-to-image model.

• Synthesized MobileMem-Omni trajectories are extremely long. The paper states they can exceed two million tokens, and that the bottom-up feedback loop is omitted in MobileMem-Omni to reduce synthesis cost, while retaining an implicit form of persona evolution by conditioning the planner on all preceding events.

• Visual memory points are synthesized through three distinct procedures. Mobile app screenshots are rendered by inserting structured field values into predefined HTML templates; camera photos of the user with other people are synthesized by retrieving participant portraits from the personal knowledge graph together with the memory-point description; other images such as food or scenery are synthesized directly from the textual description. The dialogue itself is generated only from the textual memory points, not from the images.

• Evaluation is end-to-end. Each instance is a tuple of trajectory and question-answer set, the memory layer updates state as actions arrive, and the answer is generated from the final memory state and judged against reference answers via an LLM-as-a-Judge protocol.

• No quantitative benchmark results appear in the provided content. The supplied text is truncated within Section 3.2.3 (Question-Answer Pair Synthesis), before the experimental setup and results sections. Accuracy figures, baseline comparisons, model names, and dataset sizes beyond those listed above are not reported in this content.

Methodology in Plain English

The authors start from the assumption that a realistic mobile memory benchmark cannot be bought or scraped, because long-horizon personal and multimodal trajectories are expensive to collect and raise privacy issues. So they synthesize them from a smaller amount of real user prior knowledge.

First, they build user prior knowledge. For MobileMem they interview two volunteers, collect anonymized app-usage metadata from their phones, and use public persona datasets (PersonaHub, OPeRA) and existing open-source datasets (UltraChat) as additional sources. For MobileMem-Omni they interview eight volunteers, generate eight virtual profiles from those, and construct a personal relationship graph with portraits.

Second, KEME turns that prior knowledge into a year-scale trajectory. A knowledge-guided planner splits the time horizon into coarse life events, then recursively expands events into finer sub-events or concrete sessions. Knowledge anchors (real, already-happened sessions, such as the collected user-app sessions) are grounded onto compatible event nodes whose time intervals fully contain the session intervals; if none fits, the graph is revised. Grounded content becomes a compatibility context that constrains later synthesis so generated interactions do not contradict anchored facts. As events are expanded, an experience-driven reviser adds, removes, or adjusts future events and dependency edges, forming an expansion-refinement loop so the trajectory grows rather than being assembled statically. A realizer produces the actual sessions: anchored sessions are adopted or merged verbatim, otherwise natural human-assistant dialogues are synthesized. Personas are modeled as dimension-wise attributes with version histories and message-level evidence links, so sessions progressively reveal and update the persona.

Third, question-answer pairs are synthesized bottom-up. The pipeline treats the profile and event structure as trees, generates simple QA pairs at leaf nodes (profile dimensions and concrete sessions), then uses child pairs as building blocks at internal nodes to create harder questions requiring reasoning across multiple dimensions or sessions, occasionally extending the question taxonomy. MobileMem-Omni instead uses a fixed taxonomy and shallow expansion depth, generating pairs directly from a session's memory points and its event.

Finally, evaluation is end-to-end: the memory system consumes the whole trajectory and answers each question from its final memory state.

Why This Matters

Impact on research. The paper reframes agent memory from cloud-centric information retrieval toward on-device, experience-based memory, and it argues that building accurate, secure, and durable on-device personal memory will determine who dominates the next generation of intelligent services. It also supplies a reusable synthesis framework: because KEME defines a generic interface for knowledge anchors, it can transform fragmented off-the-shelf data into long-horizon multi-session trajectories, which the authors note addresses the scarcity of multi-session data for training and evaluating agents with cross-session reasoning.

Real-world applications.

• Mobile system-level assistants such as OPPO's Breeno or Apple's Siri, which need cross-application user knowledge and personalized assistance.
• Application-specific memory features, for example note-taking, document editing, voice memo, calendar, and to-do apps forwarding structured memory events to a shared assistant memory.
• Screenshot-driven assistants that infer context from user-shared screenshots when apps are not directly connected to the memory layer.
• Wearable and edge form factors with tight resource budgets, including AR glasses, smart earbuds, wearables, and in-car agents.

Industry relevance. The paper is co-authored with OPPO and OpenKG, and frames on-device memory as infrastructure: just as cloud computing and databases constitute today's digital infrastructure, on-device memory is positioned as a new critical factor of production. The standardized protocol between the system-level memory and application memories also carries a commercial dimension, since users can specify which applications may contribute to system-level memory while applications decide what is safe to expose.

Future Directions

• Moving from the two simplified benchmark scenarios to the full composite architecture, in which system-level and application-specific memories collaborate bidirectionally through the standardized protocol rather than apps being templated integrations or screenshot sources.
• Handling the resource constraints the paper identifies (storage, power, latency, compute) as first-class evaluation criteria, since the presented scenarios evaluate reasoning quality rather than deployment cost.
• Testing adaptation and consolidation over time, including identifying stale information and incorporating new experiences without catastrophic forgetting, which the paper lists as a core challenge but which the described scenarios cover mainly through the synthesized task categories.
• Reducing the synthesis cost of the multimodal setting, since MobileMem-Omni omits the bottom-up feedback loop because trajectories can exceed two million tokens; a cheaper loop would restore explicit persona revision there.
• Quantifying how well memory systems actually perform: the provided content stops before the results section, so accuracy across multi-hop reasoning, temporal reasoning, knowledge updating, implicit preference inference, and abstention remains unreported in this excerpt.

Target Audience

Researchers and engineers working on LLM agents, long-term and personal memory systems, on-device or edge AI, and mobile assistant products. It is also relevant to benchmark designers interested in privacy-preserving data synthesis, and to practitioners at device manufacturers and application vendors deciding how much personal data should be exposed to a system-level assistant. Readers need prior familiarity with agent memory architectures, multimodal pipelines, and evaluation methodology; the paper is not written as an introductory survey.

Authors’ abstract

The next generation of AI agents is increasingly moving beyond systems that answer isolated questions toward persistent personal assistants that can understand, remember, and continuously learn from users' experiences. Such assistants require long-term memory to accumulate and leverage user-specific experiences over time, yet existing benchmarks remain inadequate for realistic mobile settings, where experiences are heterogeneous, multimodal, evolving, and deeply personal. We introduce MobileMem, a benchmark and framework for studying on-device long-term memory, grounded in a year-scale collection of mobile experiences. MobileMem employs a knowledge-grounded synthesis pipeline to construct coherent and temporally consistent long-horizon trajectories from user-app sessions. It provides complementary text and multimodal settings covering multi-hop and temporal reasoning, knowledge updating, and implicit preference inference. Specifically, MobileMem enables agents to remember the past, understand the present, and adapt to the future. By modeling experiences rather than isolated facts, MobileMem moves memory beyond information retrieval toward experiential intelligence for continuous personal learning.