From Fragmentation to Integration: Exploring the Design Space of AI Agents for Human-as-the-Unit Privacy Management

AI summary

Overview

• Research area: Human-Computer Interaction / usable privacy, at the intersection of privacy management and agentic AI.
• Technical level: Intermediate. The paper is a user study (interviews plus a survey), so it is readable without engineering background, but it assumes familiarity with privacy control concepts and with what LLM-based AI agents can do.
• Scope in one sentence: The paper maps users' cross-application, cross-time, and cross-relationship privacy struggles and evaluates nine AI agent design concepts intended to manage privacy with the person, rather than a single app, as the unit of control.

What This Paper Is About

Existing privacy controls are built for one app or one platform at a time, so people who live across many apps, whose preferences change over years, and whose relationships shift are left to improvise with manual, ad hoc strategies. The authors adopt a "human-as-the-unit" perspective and ask what users actually struggle with across boundaries, then generate and test AI agent concepts that could unify privacy management. The goal is to define a design space for agents that bridge the fragments of today's privacy tooling, rather than to build or ship a working system.

Key Contributions

1. The first solution-focused, human-as-the-unit privacy study. The authors state this is the first study that examines privacy management from a human-as-the-unit perspective rather than within bounded contexts such as a single app, device, or technology domain.
2. Nine empirically grounded privacy challenges. From 12 semi-structured interviews, the authors identify nine privacy-management challenges that are undersupported by current technology and left to manual user effort, spanning applications, temporal contexts, and relationships.
3. Two distilled design factors and nine AI agent concepts. The findings yield a timing factor (pre-sharing vs. post-sharing) and a user agency factor (fully-autonomous, half-autonomous, user-controlled), which are used to generate nine agent concepts: App Dictionary, Digital Identity Manager, Contextual Strategy Bot, Summary Bot, Dynamic Privacy Preference Agent, History Sweeper, Stimulated Privacy Community, Post Central Manager, and Auto Redactor.
4. A ranked evaluation with 116 US participants. A speed-dating survey of nine storyboards produces a global ranking of concepts, plus qualitative themes about when users accept AI in privacy management and what conditions they require to trust it.

Main Findings

• Top three concepts are all post-sharing tools: Digital Identity Manager ranked highest, followed by Dynamic Privacy Preference Agent and History Sweeper. The three concepts have an 80%, 75%, and 75% chance, respectively, to be preferred over the least-ranked idea.
• Autonomy was acceptable for remediation: The three highest-ranked concepts were all post-sharing management tools with half or full agent autonomy, which the authors describe as users' most-interested yet under-explored design opportunity.
• Users trusted AI over themselves: Participants expressed greater trust in AI accuracy than in their own efforts, and showed stronger acceptance of AI agents managing existing digital footprints than of pre-sharing privacy protection.
• General positivity toward both needs and solutions: Participants were generally positive about both the identified privacy needs and the perceived effectiveness of the proposed solutions.
• Challenges across applications: Participants reported insufficient information flow transparency (including data passing to unknown third parties and suspected sharing among apps under one parent company, such as Facebook, Instagram, and WhatsApp), inadequate privacy risk assessment (unreadable policy agreements, no way to compare apps, awareness only after harm), identification protection challenges (cross-platform linking of quasi-identifiers, e.g., a name search surfacing a Strava profile, and an inability to use pseudonyms on some platforms), and misaligned expectations across platforms with inconsistent privacy designs.
• Manual, overlapping strategies: People relied on ad hoc workarounds such as non-face profile pictures, nicknames, avoiding specific apps (Temu, Shein, Pinterest were named), and word-of-mouth or media coverage to learn about privacy problems.
• Conditions for adoption: Qualitative analysis separated factors that help users accept AI privacy solutions from factors that need improvement before they would accept them.
• A critical unresolved tension: The authors note that privacy management agents sit alongside documented AI privacy risks, including memorization of sensitive data, prompt injection, and extraction of private information from retrieval-augmented knowledge bases.

Methodology in Plain English

The study ran in two phases.

Phase 1 (formative, interviews). The researchers recruited US adults through Prolific, screening for people who reported concerns about how their input data is handled across applications. Out of 54 screening responses, 44 met eligibility criteria. They interviewed 12 participants in 45-minute semi-structured sessions, asking what information they had shared, what cross-boundary concerns they had, and what strategies they had tried. The protocol was piloted with four participants. Participants were not asked about AI-agent privacy tools during interviews, to avoid constraining the design space. They described types of information, not actual content. Participants received 15 USD; the sample included 2 with technical backgrounds, 4 non-technical, 5 mixed or interdisciplinary, and 1 in customer service. Analysis used bottom-up thematic analysis (affinity diagramming), with two researchers iteratively coding, clustering quotes, and refining higher-level themes.

Phase 2 (summative, speed dating). Two coders (a PhD student and a CS professor) brainstormed 20 ideas in FigJam, constrained by the nine interview insights, the two design factors, and an OpenAI-derived definition of an AI agent. They converged to 9 concepts by merging overlaps and discarding ideas that either did not require agentic capability or did not address a specific finding. Each concept became a five-frame storyboard (context, problem, technology, user action, resolution), illustrated with GPT-4o-mini, quality-checked, and comprehension-tested with six participants across 54 responses; one concern about technical feasibility was raised, which the authors judged not to affect interpretation.

116 US participants (from 144 collected responses, after attention and manual quality checks) each viewed three randomly selected storyboards and rated scenario relevance and perceived tool effectiveness, answered open-ended scenario questions, and ranked the three storyboards. Copy-paste was disabled to discourage AI-generated responses. Compensation was 4 USD. A global ranking was produced with the Plackett-Luce method, with 8 parameters estimated for 9 options and roughly 15 participants per parameter, above the minimum 10-per-parameter heuristic cited. Qualitative answers were thematically coded by two researchers into factors aiding acceptance and factors needing improvement; inter-rater reliability was not calculated because the goal was identifying emergent themes.

Why This Matters

The paper reframes privacy tooling from an app-centric to a person-centric unit of analysis and shows that users are willing to hand automation real control, but mainly over data they have already shared.

• Post-sharing cleanup tools: Automated detection and remediation of old posts, photos, and other shared traces that no longer match a person's current preferences.
• Cross-platform identity management: Tools that assess how a person can be re-identified by combining quasi-identifiers across services, and that unify ad hoc pseudonym and profile-picture strategies.
• Plain-language privacy comparison: Agent-assisted comparison of policies and data practices across apps, addressing the gap where people only learn of risks after harm occurs.
• Platform-level design guidance: Input for mobile operating system vendors designing cross-app privacy experiences, and for agent builders deciding where autonomy is acceptable.

Industry relevance: the findings give agent and privacy-product teams a ranked, user-validated ordering of concepts, and a specific signal that post-sharing remediation with half or full autonomy is the most promising near-term direction. The discussion of AI privacy risks also flags where deploying such agents could introduce new exposures.

Research impact: the work defines design factors (timing, user agency) and a concept set that later empirical work can build on, and it argues that context-specific usable privacy research leaves holistic cross-boundary needs unaddressed.

Future Directions

• Prototype and field-test the top-ranked post-sharing agents (Digital Identity Manager, Dynamic Privacy Preference Agent, History Sweeper) to move from storyboard perception to measured behavior.
• Resolve the trust and control conditions the study surfaced, since the analysis separated factors that help acceptance from factors that need improvement but does not specify how to satisfy the latter.
• Address AI privacy risks in the agent itself, given documented vulnerabilities such as memorization of sensitive data, prompt injection, and extraction from retrieval-augmented knowledge bases.
• Extend beyond mobile and beyond the US sample. The authors chose mobile as a concrete domain (91% of US adults owned a smartphone in 2025) and studied a US sample, leaving other platforms and populations unexplored. Technical feasibility of the illustrated concepts was also flagged by one comprehension-check participant and is not resolved in this paper.

Target Audience

Usable privacy and security researchers, HCI researchers studying agentic AI, privacy product designers and mobile platform teams, and LLM agent developers. It is also suitable for graduate students entering usable privacy or human-centered AI, since it demonstrates a full formative-to-summative design exploration pipeline with a shared codebook and survey instrument in the appendices.

Authors’ abstract

Managing one's digital footprint is overwhelming, as it spans multiple platforms and involves countless context-dependent decisions. Recent advances in agentic AI offer ways forward by enabling holistic, contextual privacy-enhancing solutions. Building on this potential, we adopted a ''human-as-the-unit'' perspective and investigated users' cross-context privacy challenges through 12 semi-structured interviews. Results reveal that people rely on ad hoc manual strategies while lacking comprehensive privacy controls, highlighting nine privacy-management challenges across applications, temporal contexts, and relationships. To explore solutions, we generated nine AI agent concepts and evaluated them via a speed-dating survey with 116 US participants. The three highest-ranked concepts were all post-sharing management tools with half or full agent autonomy, with users expressing greater trust in AI accuracy than in their own efforts. Our findings highlight a promising design space where users see AI agents bridging the fragments in privacy management, particularly through automated, comprehensive post-sharing remediation of users' digital footprints.