Personalized Privacy Control in LLMs via Attention Head Intervention

AI summary

Overview

Research area: Privacy-preserving NLP / LLM alignment and interpretability, specifically personalized privacy control in agentic LLM assistants.

Technical level: Intermediate (requires familiarity with transformer attention heads, probing, and activation steering, though the paper explains the mechanics).

Scope: The paper introduces the concept of personalized privacy, releases the P3Bench benchmark for evaluating it, demonstrates that prompting alone fails to enforce user-specific disclosure preferences, and proposes Repair, an inference-time attention-head intervention method that improves adherence to those preferences.

What This Paper Is About

Prior work on contextual privacy asks whether an LLM discloses user information appropriately for a given context, but assumes a single fixed disclosure norm per context. The authors argue that two users in the same context may want different things disclosed, so privacy control must be personalized to each user. The paper's goal is to formalize this personalized privacy setting, build a benchmark for it (P3Bench), and provide a method (Repair) that makes models actually obey user-specific disclosure policies rather than falling back on their own internal defaults.

Key Contributions

1. Introduction of personalized privacy as a formal extension of contextual privacy, in which the acceptable disclosure set is further restricted by user-specific preferences beyond contextual relevance, with disclosure states defined as Disclosure, Policy-Refusal, and Base-Refusal.

2. P3Bench (Personalized Privacy Preservation Benchmark), a benchmark that extends the contextual privacy policies of AirGapAgent-R (Green et al., 2025) with four user-specific policy settings: Privacy-Max, Contact-Open, Health-Open, and Preference-Open, built from 3,536 test instances covering 17 distinct user profiles.

3. Empirical evidence that prompt-based policies fail, showing that models often ignore personalized policies (average policy ignorance ratios of 51.25% for Qwen2.5-7B and 74.28% for Gemma3-4B) and exhibit conflicting default disclosure behaviors — Qwen2.5-3B and Qwen2.5-7B tend to over-refuse, while Gemma3-4B tends to over-share.

4. Repair, an inference-time attention-head intervention method that identifies policy-relevant heads via AUROC-scored linear probes, predicts the disclosure state by majority voting across those heads, and applies state-adaptive interventions (refusal patching vectors or a disclosure steering direction) without retraining.

Main Findings

• Direct prompting produces substantial policy violations. Across models and the four personal policies, direct prompting yields significant over-refusal (OR) and over-sharing (OS) rates, with different models showing different failure modes — some overly conservative, others overly permissive.

• Models largely ignore prompted personal policies. Policy Ignorance Ratio (PIR) is high across policies, and is especially pronounced for Gemma-3-4B, whose PIR remains above 70% across all four policies.

• Field-level priors are hard to override. Per-field PIR analysis for Gemma-3-4B shows several health and preference-related fields with very high PIR, indicating strong default answer/refuse tendencies that prompting struggles to change.

• Repair achieves the lowest Policy Error Distance (PED). Under Privacy-Max, Repair reduces PED from 71.11 to 6.78 (a 90.5% reduction) on Qwen2.5-3B and from 35.38 to 11.40 (a 67.8% reduction) on Gemma3-4B, compared with Direct Prompting.

• Repair reduces both error types rather than trading one for the other. Baselines such as CoT show an asymmetric error trade-off, and CAST and AdaSteer improve some settings but show inconsistent trade-offs across models and policies.

• AUROC-based head selection matters. On Qwen2.5-7B, random head selection only marginally reduces PED over Direct Prompting (for example, 74.16 to 56.54 under Privacy-Max, versus 32.97 with AUROC-based selection), while AUROC-based selection achieves lower PED across all policies.

• Repair generalizes to arbitrary field combinations. With random field-level policies sampled at n ∈ {4, 8, 16} accessible fields on Qwen2.5-3B, Repair consistently lowers OR, OS, and PED relative to Direct Prompting, maintaining low over-sharing while reducing over-refusal as policies become less restrictive from n = 4 to n = 16.

• Policy-agnostic calibration remains competitive. Repair-G, trained with a single global probe using the same total of N = 100 calibration examples (25 from each of four policies), achieves lower PED on three out of four policies (Privacy-Max, Contact-Open, Health-Open) compared with policy-specific Repair.

• Policy-relevant heads play complementary roles. Selected heads fall into balanced, disclose-specialist, refuse-specialist, and weak categories, with weak heads consistently rare. Selected heads concentrate in later layers, primarily beyond layer 20, suggesting policy-conditioned disclosure control relies on higher-level semantic representations.

Methodology in Plain English

The authors reframe privacy control as a per-query decision: given a task, a user's policy, and a requested field, the model should either answer or refuse. The ground-truth answer is decided by combining the task requirement with the user's policy, producing three states (Disclosure, Policy-Refusal, Base-Refusal) that map onto two observable actions (Answer, Refuse).

To measure how poorly prompting does, they put the personal policy directly in the system prompt with explicit rules and compare model outputs against ground truth, computing over-refusal, over-sharing, and the combined Policy Error Distance. Model outputs are mapped to Answer/Refuse by a rule-based classifier validated against an LLM judge (agreement ranging from 97.09% with Cohen's kappa 0.778 and MCC 0.790 on Qwen2.5-3B-Instruct to 99.57% with kappa 0.962 and MCC 0.962 on Qwen2.5-7B-Instruct).

Repair then works on the model's internals. First, individual attention head activations at the final input-token position are used to train logistic regression probes that predict the disclosure state, and each head is scored by the AUROC of its probe; the top k = 30 heads become the policy-relevant set. Second, intervention vectors are built from gold-conditioned forward passes: refusal states get mean activation patching vectors for Policy-Refusal and Base-Refusal, while the Disclosure state gets an L2-normalized steering direction pointing away from the average of the two refusal representations, so that input-specific content is preserved rather than overwritten. Third, at inference a single forward pass extracts activations from the selected heads, each probe votes on a state, and the majority-vote state selects which intervention is applied during generation. Calibration uses N = 100 examples per disclosure state across three user profiles, drawn from the AirGapAgent-R training set and kept disjoint from the test set.

Why This Matters

Impact on research: The paper shifts privacy evaluation from context-only norms to user-specific preferences, providing both a benchmark (P3Bench) and a mechanistic account of how disclosure decisions are represented in attention heads. It also shows that activation-level intervention can outperform prompting and existing steering baselines for policy compliance, and it offers evidence that policy-relevant heads are functionally heterogeneous rather than encoding a single refusal direction.

Real-world applications:

• Personal assistants and agentic LLM systems that book reservations, schedule appointments, or call third-party APIs on a user's behalf, where the same field may be shareable for one user and off-limits for another.
• Health and care coordination agents that must balance task usefulness against a user's comfort with disclosing conditions, medications, or disabilities.
• Customer-facing service bots that need to respect per-customer disclosure preferences for contact details, preferences, or health data.
• Privacy compliance tooling that enforces an audit trail of what a model would and would not disclose under a given user policy before deployment.

Industry relevance: Because Repair operates at inference time without retraining, it can be layered onto existing instruction-tuned models. The paper's finding that policy-agnostic calibration (Repair-G) remains competitive reduces the operational cost of per-user or per-policy customization, which matters for products serving heterogeneous user bases.

Future Directions

• Broaden field and scenario coverage. The authors note that P3Bench's coverage of PII fields and scenarios remains limited, with the benchmark focused on structured PII fields, and identify expansion to broader PII categories and more realistic interaction scenarios as future work.

• Handle unstructured disclosures. Privacy preferences are often expressed through open-ended disclosures, such as sensitive experiences or interpersonal situations, that cannot be represented as predefined fields; extending personalized privacy control to these settings is called out as future work.

• Understand why some fields resist control. The paper reports very high field-level PIR for certain health- and preference-related fields on Gemma-3-4B, leaving open how such strong default priors arise and how they can be reliably overridden.

• Generalize calibration further. While Repair-G shows that transferable heads exist, the paper reports that the policy-agnostic variant underperforms policy-specific Repair on Preference-Open, raising the question of which policies admit shared heads and which require dedicated calibration.

Target Audience

Researchers and practitioners working on LLM privacy, agentic AI safety, and model interpretability; engineers building assistants that handle personal data under user-specified constraints; and readers interested in inference-time activation steering and attention-head analysis as alternatives to retraining or prompt engineering.

Authors’ abstract

The rise of agentic AI enables LLMs to access diverse user data, raising critical privacy concerns. Prior work on contextual privacy studies whether LLMs regulate information disclosure according to context-dependent norms. However, acceptable disclosure boundaries may vary across users even within the same context. To address this limitation, we introduce \textit{personalized privacy}, which incorporates user-specific disclosure preferences into privacy control. We further present P3Bench~(\textbf{P}ersonalized \textbf{P}rivacy \textbf{P}reservation \textbf{Bench}mark), a novel benchmark extending contextual privacy policies with personalized disclosure policies. Experiments show that prompt-based policies fail to reliably enforce personalized privacy policies, with Qwen2.5-7B and Gemma3-4B showing average policy ignorance ratios of 51.25\% and 74.28\%, respectively. Finally, to address this problem, we propose \textsc{Repair}, a robust inference-time attention head intervention method that adjusts disclosure behavior toward policy-consistent responses. Our method significantly improves adherence to user-specific privacy preferences by reducing cases where the model fails to follow the given policy.