Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents

AI summary

Overview

Research area: Personal AI agents for Graphical User Interfaces (GUI) — specifically how an agent should reuse a user's own interaction history to act on their behalf. Sits at the intersection of computer vision (screen understanding), GUI/phone automation agents, personalization, and memory retrieval for frozen (non-fine-tuned) vision-language models.

Technical level: Advanced. The paper assumes familiarity with vision-language policy models, retrieval-augmented prompting, attention/latent-space embeddings, and precision/recall-style evaluation of binary decisions.

Scope in one sentence: The paper argues that retrieving history relevant to a task is not the same as knowing which piece of that history applies to the current screen or moment, and proposes a training-free framework, ExpActivator, that selects (activates) the matching historical action or routine at decision time.

What This Paper Is About

Personal GUI agents are supposed to interpret vague instructions like "order my usual coffee" and to offer routines before being asked, both by drawing on the user's recorded interaction history. Existing systems retrieve history that is related to the task and paste it into the agent's context, implicitly assuming that relevance at the task level means usefulness at every decision. The paper shows this assumption fails: retrieved history helps mostly with the first action of an episode and then stops paying off, and it gives weak signal about whether a proactive suggestion is warranted at all. The goal is to turn personalization from a one-time retrieval problem into a decision-time selection problem — supplying only the past action or routine that fits the current situation.

Key Contributions

1. A diagnosis of task-level retrieval. The authors demonstrate empirically that task-level relevance does not imply decision-level applicability: retrieved history strongly improves the opening decision of an episode but yields little sustained benefit over the remaining ~90% of steps, and offers weak guidance on when a routine is due.

2. Experience activation at two granularities. They introduce ExpActivator, a fully training-free framework with step-level activation (each new screen is matched to historical states inside the frozen GUI backbone's latent space, and the corresponding action is passed to the policy as a reference) and task-level activation (a recurring intent is activated only when current time and scenario provide enough support, otherwise the agent abstains).

3. Empirical evidence across four frozen GUI backbones. Step-level activation improves personalized execution on every backbone while using roughly one-fifth the history tokens of baselines, and task-level activation produces the only selective proactive suggestion among the compared methods.

4. Controlled ablations isolating why it works. Fixed-history comparisons show that selecting one applicable action from a trajectory beats appending that same trajectory whole, that the match must be recomputed at every observation, and that time, scenario, and frequency normalization each play a distinct role in suggestion.

Main Findings

• History is spent at the first decision. On Qwen3-VL-8B, the strongest retrieval baselines raise opening-step success from 28.3% (no history) to 62.3%, and ExpActivator to 70.1%. Over the remaining 7,140 actions, none of the four baselines is distinguishable from the no-history condition, all staying within 5% of its success rate, whereas ExpActivator improves those same actions by 45% relative to no history.

• Consistent gains across backbones. ExpActivator achieves the best action-type accuracy, step success rate (SSR), and cumulative success rate (CSR) on Qwen3-VL-8B, MAI-UI-8B, Qwen3.5-9B, and GUI-Owl-1.5-8B. Relative to the same model without history it improves SSR by 44% on average and roughly doubles CSR; the abstract reports a 28% average improvement in within-trajectory step success.

• Roughly 20% higher SSR than the strongest baseline. Averaged over blocks it beats the best memory baseline by about 20% in SSR (37% on Qwen3-VL-8B), despite every method reading exactly the same history. Every open backbone surpasses the proprietary QwenVL-Max in CSR, reaching about 1.6 times its score on average.

• Far cheaper prompts. ExpActivator adds 144.6 history tokens per step versus 417.8 for HIM-Agent (the most economical baseline) and around 730–844 for Retrieve, LLM-UM, and Recent — nearly five times fewer than the average baseline and nearly three times fewer than HIM-Agent, while scoring highest.

• Proactive suggestion becomes selective. Because 68% of evaluation states are positive, always-suggesting already reaches 81.1% F1, so the authors use balanced accuracy (BA) and Matthews correlation coefficient (MCC). ExpActivator reaches about 2.3 times the MCC of the strongest baseline on average, improves BA by 24% over the best baseline, and attains the highest precision on every backbone (84% on average). On MAI-UI, three baselines behave exactly like the always-suggest rule (BA 50.0, MCC 0.0).

• Selecting beats appending. With the single top-ranked trajectory held fixed, supplying it whole gives 18.5% SSR — on par with no history — while selecting one action from it raises SSR to 23.8%, a 29% relative improvement.

• The match must be refreshed every step. Holding the eight retrieved trajectories fixed, matching against the previous screen lowers SSR by 15% relative, and refreshing only every fourth step lowers it by 13%; agreement between the selected reference and the recorded action drops 38% relative.

• References move across trajectories. On every backbone, about 92% of episodes draw references from at least two past trajectories (3.4 on average), and the reference switches trajectory on 55% of consecutive steps — still nearly half the steps after the twelfth.

• Each contextual signal does distinct work. Removing time weighting drops BA about 5% relative; removing the scenario condition drops BA 11% relative and raises false alarms from 37% to 62%; frequency normalization leaves trigger decisions unchanged but improves the semantic quality of emitted suggestions by 4% relative.

• Gains hold for users with little history. Grouping users into quartiles by history size, ExpActivator beats the strongest baseline in every quartile by more than 27% relative SSR, and by 37% for the users with the fewest records.

Methodology in Plain English

The approach has two halves, both operating on models that are never fine-tuned.

For carrying out an instruction (step-level activation): the system first narrows the user's history to the eight trajectories whose stated goals are most similar to the current instruction, using a multilingual sentence encoder. It strips only the leading app-opening phrase so the comparison is about the task, not the app name. Then, at every single step, it takes the current screenshot, encodes it with the frozen GUI backbone's own visual features (split into four centered quadrants, with an extra global block for MAI-UI, and centered/normalized against calibration images), and compares it against every recorded screen in those eight trajectories. The best-matching historical screen's recorded action is handed to the frozen policy as a reference alongside the current screen. Crucially, that reference is a suggestion, not a command — the policy still looks at the real screen and can adopt, adapt, or reject it — and the match is recomputed after every observation, so the reference can hop between different past trips and trajectories.

For offering a suggestion unprompted (task-level activation): given only a time and a scenario, the system groups the user's past goals into routine neighborhoods by embedding similarity (threshold 0.94). A routine only becomes eligible if it has enough total occurrences (at least 12) and enough occurrences in the present scenario (at least 4). Among eligible routines it ranks by a contextual score that sums, over same-scenario occurrences, a Gaussian weight on how close their hour is to the current hour (bandwidth 1 hour), divided by the routine's total support so that a routine matching on every occurrence outranks one matching on a few. If no routine clears the thresholds, the agent says nothing. Otherwise the most representative same-scenario goal is passed to the frozen backbone as a reference for phrasing.

Evaluation uses AndroidIntent: 82 users, 135 applications, 13 scenario categories; the chronological first 80% of records (15,966) form each user's history, and the held-out remainder gives 775 ambiguous episodes (7,915 steps) for execution and 315 states (215 positive, 100 negative) for suggestion. Comparisons are paired and step-by-step against Base (no history), Recent, Retrieve, LLM-UM, and HIM-Agent, with 20,000 paired bootstrap draws over users.

Why This Matters

The paper reframes personalization as a selection problem rather than a retrieval problem, and shows the reframing is worth a large, consistent accuracy gain at a fraction of the prompt cost. Because it requires no training, it can be layered onto existing frozen GUI agents — an important practical property when backbones are large and frequently replaced.

Real-world applications:

• Phone assistants acting on vague requests such as "order my usual" — resolving which app, item, and store from the user's own past screens rather than guessing.
• Proactive routine reminders — offering the commute check, the exercise log, or the alarm only at the times and places where the user actually does them, and staying silent otherwise (the paper's case study shows weather at 08:05, plant identification at 15:37, exercise at 20:26, and an alarm at 23:57, with silence before dawn and shortly after 1 p.m.).
• On-device accessibility or automation for repeated multi-step workflows where a user repeats the same screen sequence across apps.
• Enterprise/customer-support agents replaying established procedures for a specific operator or account, where the right next click depends on the current screen, not on the ticket title.

Industry relevance: the one-fifth token cost directly reduces inference expense and latency for always-on agents, and the abstention mechanism is the difference between a useful assistant and one that interrupts constantly — a key product concern for any proactive feature.

Future Directions

• Extending activation beyond screens to other context. The method uses screens, time, and a coarse scenario label; richer signals (location, device state, notifications, calendar) are not explored and could sharpen both granularities.
• Scaling and stress-testing the history store. The benchmark histories hold at most 414 records per user; how activation degrades with years of history, near-duplicate screens, or cross-app UI drift is not reported.
• Learning the matching rather than calibrating it. All thresholds (candidate count 8, similarity threshold 0.94, support minimums 12 and 4, bandwidth, mass threshold) are fixed by calibration on held-out records; whether these transfer across user populations, locales, or backbone families without recalibration is an open question.
• Richer evaluation of proactive behavior. The paper shows MCC/BA matter because 68% of states are positive, but the suggestion set is only 315 states; larger and more temporally realistic proactive benchmarks, including user feedback on unwanted suggestions, remain future work.

Target Audience

Researchers and engineers working on GUI/phone automation agents, personalized assistants, and memory or retrieval mechanisms for frozen vision-language models. It is also valuable for product teams designing proactive assistant features, since the abstention results (false alarms rising from 37% to 62% without the scenario condition) directly inform when an assistant should stay quiet. Readers need comfort with embedding-space matching and standard classification metrics; the paper is less suited to beginners seeking an introduction to GUI agents.

Authors’ abstract

Personal Graphical User Interface (GUI) agents rely on interaction history to infer what a user wants from ambiguous instructions and to anticipate recurring routines. Existing approaches retrieve task-relevant history and append it to the policy's context, implicitly assuming that experience relevant to a task remains useful for each decision within it. We find that this help is largely spent at the first decision: retrieved history strongly improves the opening step of an episode, yet provides little sustained benefit over the remaining 90\% of steps, and offers weak guidance on whether a proactive suggestion is warranted. A relevant record may tell the agent where to begin, but not which past action applies to the current screen or whether a routine is due now. The underlying issue is that relevance does not imply applicability}: relevance is determined at the task level, whereas applicability depends on the situation at decision time. We therefore recast personalization as experience activation and introduce ExpActivator, a training-free framework that activates only the experience applicable to the current situation. During execution, ExpActivator matches each new screen to historical states in the frozen GUI backbone's latent space and supplies the corresponding action as a reference. Before execution, it activates a recurring intent only when the current time and scenario provide sufficient support, and otherwise abstains. Across four GUI backbones, ExpActivator improves within-trajectory step success by 28\% on average, achieves the best personalized execution on every backbone while using about one-fifth as many history tokens, and reaches approximately 2.3$\times$ the Matthews correlation coefficient of the strongest proactive baseline. Experience pays where it is activated, not where it is appended.