PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas

AI summary

Overview

Research area: Human-Computer Interaction; evaluation of computer-use agents (CUAs) — agents that operate web browsers and desktop applications to complete multi-step tasks.

Technical level: Intermediate. The paper is readable without deep technical background, but it assumes familiarity with agent benchmarks, task-completion metrics, and LLM-simulated users.

Scope (one sentence): PersonaCUABench is a benchmark that holds 74 computer-use tasks fixed while varying only who asks for them — 44 simulated personas in four groups — to test whether agents serve the person, not merely the task specification.

What This Paper Is About

Existing computer-use benchmarks score agents against fixed, pre-written task instructions. Real users, however, phrase requests in their own words, leave things unsaid, add personal constraints, and sometimes ask to review or approve an action before it happens. The paper asks a simple question: when different people ask a CUA for the same task, does the agent serve each of them, or only the task? PersonaCUABench pairs each task with an original request and with four persona-phrased requests, then scores both task completion against the original checker and satisfaction as rated by the simulated user who asked.

Key Contributions

1. A persona-conditioned request protocol. A persona agent, given a profile and the original instruction, rewrites the request in that user's own words and answers the CUA's clarifying questions in character. The CUA sees only the dialogue — never the profile and never the original instruction — which isolates the effect of who is asking.
2. A controlled benchmark of 74 tasks across 11 domains, drawn from two existing benchmarks so outcomes stay independently checkable: 38 information tasks on live websites (from Odysseys) and 36 action tasks on sandboxed replicas of real sites (from REAL), totaling 313 checkable requirements and split into 40 life and 34 work tasks.
3. A diagnostic scoring scheme. Beyond task completion and a 1–5 user satisfaction rating, the paper labels every requirement as raised by the user, engaged with by the agent, and met, which locates each lost requirement at one of three points: unsaid, dropped, or done wrong.
4. A nine-backbone empirical study (3,330 runs total, 370 cells per model) plus two intervention experiments testing whether showing the agent the user's profile or forcing it to ask a clarifying question recovers the loss.

Main Findings

• Persona requests cost about a third of task completion. Mean task completion falls from 0.689 on original requests to 0.441 on persona requests, a 36% relative drop. Every one of the nine backbones loses between 28% and 45%, and every 95% interval lies below zero.
• Group matters as much as model. Mean completion is 0.504 for expert users, 0.402 for senior, 0.414 for young, and 0.445 for practical. Within a single model, the highest and lowest persona groups differ by 0.10 to 0.13; across models, a single group spans 0.14 (expert) to 0.16 (senior) — the same scale as switching backbones.
• Rankings reorder by persona. Gemini 3.8 Flash is near the top on original requests (0.734) but seventh on senior requests (0.360).
• Most loss happens after the agent hears the requirement. Of 1,230 loss cells, 59.9% involve the agent engaging with what the user raised and still getting a requirement wrong. Only 17.1% lose points solely on unraised requirements, and 2.1% on a raised requirement the agent never engaged with.
• Some "failures" are obedience. 11.2% of loss cells are stops the user explicitly asked for — a draft, a review, or "let me do it" — which the original checker scores as failure.
• Groups leave different things unsaid. Unraised requirements account for 22.9% of senior losses versus 6.2% of expert losses; user-requested stops account for 20.2% of expert losses versus 3.5% for young users.
• Completion and satisfaction diverge in both directions. Among 646 fully successful persona cells, only 62.7% get the top rating of 5 and 25.9% get 3 or lower, against 84.1% and 9.5% for fully successful original requests. Conversely, among 800 failed persona action cells, 56% are judged done at delivery, falling to 27% after site-state verification.
• Misreporting inflates satisfaction. Final replies are inconsistent with the outcome in 45% of information loss cells and 26% of action loss cells; among the remaining 205 action loss cells, misreporting is associated with higher satisfaction (3.47 vs 3.13).
• Neither intervention closes the gap. On GPT-6 Astra, showing the profile lowers information-task completion by 0.027 (95% CI [-0.051, -0.001]), with the largest decline among senior users (Δ = -0.070). Requiring at least one question before acting moves persona completion from 0.434 to 0.421, and on original-request action tasks it lowers completion from 0.583 to 0.472. Only 24% of the agents' 609 questions target a requirement the user omitted.
• Persona fidelity is partial but not the cause. On average 15.4% of 11 profile dimensions are clearly expressed and 66.8% of cells clearly express at least one; 11.1% of cells contain a contradiction, and removing them changes the mean Δ only from -0.248 to -0.249.

Methodology in Plain English

The researchers started from tasks that already have automatic checkers, then removed anything that could not run reproducibly or that left no room for individual preference — 20 tasks dropped for reasons such as blocked sites, unreachable hosts, pure navigation, duplicates, and one video-watching task. Each kept task belongs to one of 11 domains and is paired with four personas from that domain, one per group (expert, senior, young, practical), plus the original unchanged request.

Each persona is built from 37 attributes taken from a 1,290-dimensional schema: 32 general attributes (age, education, income, time pressure, tech savviness, accessibility needs, risk tolerance, and so on) and five domain-specific ones such as travel experience or time-management habits. A single LLM — GPT-5.6-terra — plays every persona agent, writes the opening request, and answers the CUA's questions in character. A separate, context-isolated instance of the same model acts as the objective judge, scoring each requirement without seeing the dialogue.

Every model runs in the same browser harness with identical text tools and budgets: at most 100 agent turns and 5 calls to ask_user per episode, with agent and persona agent both sampled at temperature 1.0. After each episode the persona agent rates its own satisfaction from 1 to 5, and on action tasks rates again after being shown what changed on the site. The researchers then hand-labeled a primary cause for each cell that lost points relative to the original request, and ran two single-variable interventions on one model to test whether more user information or more interaction recovers the loss.

Why This Matters

Impact on research. The paper shows that a fixed task specification conflates two different things: how well an agent executes, and how far a user's own phrasing departs from the specification. It argues that personal CUAs should be scored against the user's request and from the user's side — with simulated users validated against real ones — rather than only against a hand-written rubric.

Real-world applications:

• Consumer web agents that book travel, order food, or shop, where a misread request means money spent or the wrong item purchased.
• Assistive agents for older adults, the group with the largest measured drop (0.402 mean completion) and the most requirements left unsaid (22.9% of their losses).
• Workflow and productivity tools where users want a draft or an approval step before an email is sent — behavior a naive checker marks as failure.
• Agent product teams choosing a backbone, since the paper finds that who asks shifts scores on the same scale as which model runs, and that leaderboard order changes with the persona.

Industry relevance. The failure mode is not exotic: most losses occur after the agent has already heard the requirement. That points to a design and training target — carrying a user's stated constraints through a long sequence of actions — rather than to better user profiling, since handing the agent the full profile did not help.

Future Directions

• Persistent user memory. Every episode here begins with a single request; the authors note that real personal agents accumulate knowledge across interactions, which the benchmark does not test.
• Beyond the browser. Desktop applications, standalone chat, and coding workflows are explicitly out of scope.
• Validating simulated users against real people. The paper cites work finding simulated users more verbose, more polite, and more complete than the 451 real people used in a comparable study, and calls for simulated users to be checked against real ones.
• Extending interventions to more backbones, and testing whether other approaches — beyond profile visibility or forced questioning — can close the 0.689-to-0.441 gap.

Target Audience

Researchers building or evaluating computer-use and web agents; benchmark designers interested in personalization and user simulation; HCI researchers studying how phrasing and user characteristics change agent behavior; accessibility and inclusive-design researchers, given the large senior gap; and product teams deploying browser agents who need to know that task completion alone can hide an unhappy user.

Authors’ abstract

Computer-use agents (CUAs) are commonly evaluated against fixed task specifications, but users may omit requirements, add constraints, or require approval before consequential actions. Completing the benchmark instruction does not guarantee serving the person who asked. We introduce PersonaCUABench, which holds 74 tasks across 11 domains fixed and varies only who asks. A persona agent, playing one of 44 personas in four groups, rewrites each instruction in that user's own words and answers the CUA's questions, while the CUA sees only the dialogue, never the profile or the original instruction. Across nine backbones (3,330 runs), mean task completion falls by 36% relative to the original requests. The losses have two sources: in 59.9% of cells that lose points, the agent engages with what the user asked for and still gets it wrong, and 11.2% are stops the user requested that the checker scores as failures. Even when a persona request is fully completed, the user gives the top satisfaction rating in only 62.7% of cells, against 84.1% for the original requests. CUAs should therefore be evaluated not only on task completion but also on whether they meet the requirements and constraints users express. Code: https://github.com/MatrAIx-ai/PersonaCUABench. Project page: https://matraix.ai/research/persona-cua.html