iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents

AI summary

Overview

Research area: Computer-use agents (CUAs) for mobile devices, with a focus on personalization — specifically how agents decide which missing task details to infer from a user's history and which to ask the user about. The work sits at the intersection of NLP/LLM agent evaluation, human-computer interaction, and mobile GUI automation.

Technical level: Intermediate. The paper is readable without deep technical background, but familiarity with agent benchmarks, LLM judging, and metrics like recall/precision helps.

Scope (one sentence): The paper introduces iPrefBench, an executable iOS benchmark of 130 underspecified mobile tasks built on a single 3,051-event user activity history and an interactive simulated user, and uses it to evaluate the infer-and-ask capabilities of seven frontier agents.

What This Paper Is About

Computer-use agents are good at executing tasks that are spelled out completely, but real users rarely spell things out — they say "book me my usual restaurant for tomorrow" and expect the agent to figure out the restaurant from their past behavior while asking about anything only they could know. The problem is that no existing benchmark tests both halves of this skill at once: personalization benchmarks give an agent a user history but no one to ask, and interactive benchmarks give a simulated user but no history against which a question's necessity can be judged. iPrefBench fills that gap by annotating each omitted detail with a single, controlled source — either recoverable from history ("inferable") or knowable only from the user ("ask-only") — so that both correct personalization and appropriate questioning can be measured.

Key Contributions

1. A benchmark with controlled information provenance. iPrefBench contains 130 underspecified tasks across 26 mock iOS apps, grounded in a 3,051-event user activity history for one coherent persona (Jordan Avery). Every omitted user-dependent detail is annotated by construction as either inferable from history or ask-only.

2. A dual evaluation framework. The benchmark separates personalized task completion (task-specific rubrics split into action-completion criteria and personalization criteria, with personalization further split by resolution source) from user-agent interaction (recall and precision over ask-only information, inspired by Grice's maxims).

3. An extended action space and simulated user. The environment adds an ask_user action to standard mobile GUI actions, backed by an LLM user simulator grounded in a hidden 13-dimension user profile that answers only what is asked.

4. Three perturbation categories for harder personalization. Tasks include instruction perturbations (11), preference perturbations (10), and environment perturbations (13), testing whether agents let explicit instructions override history, track shifting preferences, and handle unavailable preferred options.

Main Findings

• Execution far outpaces personalization. Across the seven evaluated agents, action-completion scores range from 77.5% to 91.6%, but full task success drops to 26.2%–47.7%. Claude Opus 5, the strongest agent, completes 91.6% of required actions and 69.4% of user-dependent decisions, succeeding on only 47.7% of tasks.

• Ask-only information is the weak point. Personalization on ask-only details is consistently lower than on inferable ones for every agent, by as much as 46 points for Gemini 3.7 Flash.

• Agents ask too little. Tasks contain 0.89 ask-only requirements on average, yet agents issue only 0.08 to 0.35 ask_user calls per task, with recall topping out at 44.3% in the paper's reported figures.

• What they do ask is often unnecessary. Interaction precision ranges between 60% and 77%, meaning a substantial share of questions target inferable or irrelevant information, or repeat earlier requests.

• Conflicts with history hurt personalization, but obstacles prompt questions. On instruction- and preference-perturbation subsets, personalization falls below overall averages while action completion holds roughly steady. On environment perturbations, agents issue 2.3× as many questions and improve ask-only personalization for every agent — suggesting they react to visible obstacles more than to uncertainty they must notice themselves.

• Silent guessing and silent defaults dominate ask-only failures. Manual labeling of failed Claude Sonnet 5 and Qwen3.8-27B trajectories yields four error types: Silent Guess and Silent Default for ask-only information, Missed Inference and Wrong Inference for inferable information. Qwen3.8-27B's reasoning traces show it often raises "should I ask?" and then talks itself into a guess.

• Forcing a clarification phase helps coverage but not judgment. Adding a pre-execution clarification phase raises Claude Sonnet 5's success rate from 39.2% to 53.8%, ask-only personalization from 41.5% to 82.0%, and recall from 25.7% to 85.0% — but precision falls from 60.4% to 42.5% and ask_user calls jump from 0.24 to 1.37 per task. Qwen3.8-27B improves from 26.2% to 35.4% success under the same intervention.

• More history helps inference but not questioning. For Qwen3.8-27B, inferable personalization rises from 35% to 63% with the top-50 retrieved events and then plateaus with the full history, while ask-only personalization barely moves. Precision rises with more history, but recall does not improve and can decline (33.6 → 21.2 for Qwen3.8-27B).

• The evaluation machinery is validated. The user simulator's responses were grounded in the profile or task-specific hidden context in 96.6% of inspected question–answer pairs. LLM judges agreed with human labels on 100 sampled trajectories at 95.7% on rubric criteria (κ = 0.89), 97.0% on task-level success (κ = 0.94), and 95.2% / 93.9% on recall / precision (κ = 0.90 / 0.82).

Methodology in Plain English

The researchers built a simulated phone environment — 26 mock iOS apps with interconnected data about one fictional user, Jordan Avery — and turned that data into a chronological, natural-language activity log of 3,051 events. That log is the only prior context the agent gets about the user. Separately, they derived a structured profile (13 dimensions covering identity, relationships, finances, dining, travel, fitness, and so on) that is hidden from the agent; it grounds an LLM-based user simulator that answers the agent's ask_user messages in character without volunteering extra information.

Tasks were written to be deliberately incomplete. For each detail left out, the authors decided in advance whether the answer could be recovered from a dominant pattern or stable fact in the activity log, or whether only the user could supply it. Human experts reviewed every task twice: first to confirm each omitted detail's resolution source (and to revise seeded data where the evidence was too weak or mixed), then to execute the tasks in the simulator and verify feasibility and rubric correctness. Ask-only intended values were deliberately chosen to differ from app defaults where possible, so that blindly accepting a default would rarely succeed.

Evaluation runs on two tracks. For task completion, an LLM judge checks each rubric criterion against the trajectory, screenshots, actions, final answer, user profile, and hidden context; a task only counts as a success if all criteria pass. For interaction, the judge decomposes the agent's ask_user messages into individual requests, matches each to an ask-only requirement or to nothing, and computes recall (fraction of ask-only details elicited) and precision (fraction of request occurrences that obtain new ask-only information). Seven models were run under the same harness with a 50-step budget per episode, using Claude Sonnet 5 as both the user simulator and the judge.

Why This Matters

Impact on research. The paper reframes personalization as a decision problem about information provenance rather than a retrieval problem. By making each missing detail's source an explicit, controlled variable, iPrefBench makes it possible to distinguish an agent that guesses correctly by luck from one that reasons about what it can and cannot know — a distinction the authors note is missing from benchmarks that supply both histories and simulated users but do not label which details belong to which source.

Real-world applications:

• Mobile assistants and OS-level agents that book, order, and schedule on a user's behalf without interrogating them about things already visible in their app history.
• Travel and dining booking flows, where "my usual" must be resolved from past reservations while party size and timing must be asked.
• Consumer finance and payments, where an agent must recognize when a user-specific choice (an account, a split arrangement) is genuinely undetermined rather than assume a default.
• Health and fitness coaching apps, where stable habits can be inferred from logged activity but current context (injury, schedule) must come from the user.

Industry relevance. The measured gap — high action completion, much lower full task success — is a direct obstacle to shipping personalized agents, and the finding that simply adding a clarification phase trades under-asking for over-asking suggests that prompt-level fixes alone will not close it. The three perturbation categories also map onto production concerns: honoring explicit current instructions over historical patterns, tracking preference drift, and gracefully handling unavailable options.

Future Directions

• Teaching agents to recognize unresolved decisions. The error analysis shows two separable failures — never surfacing a decision at all (Silent Default, Missed Inference) and resolving a recognized decision incorrectly (Silent Guess, Wrong Inference). The paper frames identifying unresolved choices as a distinct capability that current agents lack.

• Necessity-aware clarification policies. A clarification phase raises recall dramatically (to 85.0% for Claude Sonnet 5) while collapsing precision (to 42.5%). The open question is how to get the coverage without the over-asking — the paper explicitly concludes the challenge is not to ask more but to know when asking is necessary.

• Closing the ask-only gap. Ask-only personalization barely responds to more user history, unlike inferable personalization, which plateaus quickly with a small retrieved subset. What additional signal or training would make agents recognize information that simply is not in the history remains unresolved.

• Beyond a single persona and default-on-demand interaction. The benchmark uses one coherent persona and an on-demand ask_user protocol; extending to multiple personas, other interaction regimes, and specialized GUI agents (which the authors report could not be evaluated meaningfully here due to context limits or frequent invalid actions) would test how far the findings generalize.

Target Audience

Researchers and engineers building LLM-based computer-use and mobile GUI agents will get the most from this paper, particularly those working on personalization, preference inference, or proactive clarification. It is also useful for benchmark designers interested in how to control information provenance in task construction, and for product teams evaluating whether an agent is ready to act on a real user's behalf. Readers need only moderate familiarity with agent evaluation to follow the main results; the appendix material on action space, per-task environment control, and judge validation will matter more to those replicating or extending the benchmark.

Authors’ abstract

An agent that truly knows and acts on behalf of its user has been a long-standing promise of AI assistants. Computer-use agents bring this promise within reach by capably executing fully specified tasks across apps. However, real-world user requests are often underspecified. To act correctly for the particular user, an agent must infer some missing information from the user's activity history and ask for other information that only the user can provide, without making unsupported assumptions or burdening the user with unnecessary questions. We present iPrefBench, a benchmark for evaluating the infer and ask capabilities of personal mobile-use agents in an executable iOS environment with an interactive simulated user. Grounded in a coherent persona with 3,051 past activity events, iPrefBench comprises 130 underspecified user tasks across 26 apps. Each omitted user-dependent requirement is, by construction, either history-inferable or ask-only. iPrefBench evaluates personalized mobile agents on both whether agents complete tasks correctly for the user (i.e., personalized task completion) and whether they seek user input when necessary (i.e., user-agent interaction). Evaluation of 7 frontier agents reveals a substantial gap between execution and personalization. Agents achieve high action completion scores (up to 91.6%) but low full task success (up to 47.7%) because personalization lags behind. They especially struggle with ask-only information, failing to request enough necessary information. Our findings highlight a challenge for personalized agents in knowing what to infer and when to ask. We hope that iPrefBench can serve as a valuable testbed for building agents that know their user and work with them.