User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios

AI summary

Overview

Research area: Human-centered evaluation of large language models — specifically privacy preservation and helpfulness in privacy-sensitive, task-oriented scenarios.

Technical level: Intermediate. The statistical methods (Krippendorff's alpha, Spearman's rank correlation) are standard agreement and correlation measures, but the paper's framing is accessible to readers familiar with LLM evaluation practices.

Scope: A user study (n=94) comparing how human participants and five proxy LLM judges rate the helpfulness and privacy-preservation quality of LLM responses to 90 PrivacyLens scenarios.

What This Paper Is About

Prior privacy benchmarks such as ConfAIde and PrivacyLens found that LLMs sometimes leak private information when completing everyday tasks, but those evaluations used proxy LLMs — other language models — to judge whether a response was helpful and privacy-preserving, rather than asking actual users. This means nobody had established whether proxy judgments actually reflect what people think. The researchers recruited 94 participants and had them evaluate the same LLM-generated responses as five proxy LLMs, then measured how well the two groups agreed with each other.

Key Contributions

1. A direct measurement of user perceptions. The authors collected 470 human evaluations (94 participants × 5 scenarios each) covering helpfulness and four privacy-related dimensions across 90 randomly selected PrivacyLens scenarios, rather than relying on model-based judgment.

2. A head-to-head comparison of human and proxy LLM evaluators. Five proxy LLMs (GPT-5, Llama-3.3-80B-Instruct, Gemma-3-27B, Qwen-3-30B-A3B-IT, and Mistral-7B-Instruct-v0.3) completed the identical survey, each with five repeated runs per scenario, enabling both within-model and across-model agreement analysis.

3. Evidence that proxy LLMs collapse human disagreement. The paper quantifies the gap in rating spread, standard deviation, and inter-rater agreement between participants and proxy LLMs, and reports weak-to-moderate scenario-level correlations (Spearman's ρ ∈ [0.24, 0.68]).

4. A qualitative comparison of justifications. Using thematic analysis of participant explanations and semantic similarity of proxy LLM explanations, the authors identify where model reasoning diverges from human reasoning (missed context, overlooked sensitive data, differing privacy views).

Main Findings

• Participants were generally positive about the responses. Participants indicated the response completed the task in 92% of evaluations, rated it helpful in 87%, and said they would use it in 84%. For privacy, 78% of evaluations said the response mostly or completely complied with privacy norms, and 83% said it respected their personal privacy preferences.

• Humans disagreed with each other substantially. Inter-rater agreement within scenarios was low (Krippendorff's α = 0.36). At least two participants selected opposite ends of the Likert scale on intention to use the response (H3) in 20% (n=18) of scenarios. For information sensitivity (P1) and willingness to share with an LLM (P2), participants never fully agreed in any of the 90 scenarios. For privacy-norm compliance (P3) and personal privacy preferences (P4), at least one participant disagreed with others in 88% (n=79) and 96% (n=86) of scenarios, respectively.

• Proxy LLMs were highly consistent with themselves and with each other. Agreement across the five repeated runs of each proxy LLM (α) was: Gemma-3 0.9788, GPT-5 0.9328, Llama-3.3 0.9830, Mistral 0.8823, and Qwen-3 0.9848. The lowest within-model value was 0.88 (Mistral) and the highest 0.98 (Qwen-3). Across all five proxy LLMs pooled into 25 runs per scenario, agreement was α = 0.78 (moderate).

• Proxy LLMs did not reproduce human variability. For perceived helpfulness (H2), GPT-5's five runs gave identical ratings on 72% of scenarios and Qwen-3's on 99%, while participants fully agreed on only 6%. For P3, participants selected opposite ends of the scale on 30% of scenarios; four of the five proxy LLMs never did so, and GPT-5 did so in only 1%. For P4, 11% of scenarios had at least one participant choosing strongly disagree and another strongly agree, while no proxy LLM showed that diversity. On H3, participants had non-zero standard deviation in 90% of scenarios, compared to 27% for GPT-5 and below 10% for the other proxy LLMs. On P1 and P2, participants had non-zero standard deviation across all 90 scenarios, while proxy LLMs fully agreed on over 52%.

• Correlation with human averages was only weak to moderate. Across the 90 scenarios, individual proxy LLMs correlated with mean participant ratings at Spearman's ρ ∈ [0.24, 0.68]. Pooling all 25 proxy judgments per scenario also yielded weak-to-moderate correlations.

• Reasoning diverged in identifiable ways. Participant explanations within a scenario had low semantic similarity (0.18–0.27) using the Universal Sentence Encoder, versus 0.66–0.85 for proxy LLMs; different proxy LLMs resembled each other at 0.66 (P4) to 0.72 (H2). In scenario 65, four of five participants noticed the response did not address the assigned task, while all five proxy LLMs labeled it helpful across 25 runs. In scenario 78, all five participants flagged credit card details as sensitive, while Mistral stated there was "no explicitly sensitive information like financial data."

• What drove helpfulness and privacy judgments. Among participants rating a response very helpful, 93% mentioned "task completion" and 8% "clarity"; those rating it unhelpful cited "task failure" 73% of the time. For intention to use, participants extremely likely cited "task completion" (92%) and "clarity" (26%), while extremely unlikely cited "task failure" (67%) and "privacy leaks" (33%). Participants who said the response did not at all or slightly respect privacy norms almost always cited "leaked private info" (90%); those who said it did respect norms most often cited "limited sharing" (79%).

Methodology in Plain English

The authors randomly selected 90 scenarios from PrivacyLens — a dataset of realistic, task-oriented situations containing contextual material such as meeting notes and email history. They generated exactly one response per scenario using OpenAI's GPT-5 API with the privacy-enhancing prompt from the PrivacyLens repository. They converted the scenarios from JSON to HTML for presentation without altering content.

They then built a survey with seven questions per scenario: three about helpfulness (H1, a Yes/No question on task completion; H2, perceived helpfulness; H3, intention to use) and four about privacy (P1, information sensitivity; P2, willingness to share with an LLM; P3, compliance with privacy norms; P4, respect for personal privacy preferences). All but H1 used a five-point Likert scale, and participants were asked to explain their answers in open text.

Recruitment ran through Prolific, restricted to adults 18+, fluent in English, and located in the U.S. Each participant evaluated five randomly selected scenarios, and each scenario was rated by at least five participants. The authors note that 90 participants would ideally suffice, but imperfect balancing plus the exclusion of one participant who gave nonsensical responses to all open-ended questions led to 94 valid participants.

The same survey was given to five proxy LLMs, with the prompt "Imagine you are {name}" mirroring the wording used with humans. Each model produced five independent evaluations per scenario using its default generation parameters, so variability could be measured. The open-weight models — Llama-3.3-80B-Instruct, Gemma-3-27B, Qwen-3-30B-A3B-IT, and Mistral-7B-Instruct-v0.3 — ran on an NVIDIA A100 GPU. Likert responses were converted to numeric values of −1, −0.5, 0, 0.5, and 1. Agreement was measured with ordinal Krippendorff's α (with α < 0.67 low, 0.67 ≤ α ≤ 0.79 moderate, α ≥ 0.8 satisfactory), and the association between group averages was measured with Spearman's ρ. Two researchers coded participant explanations through thematic analysis, starting with 10 answers per question to build a codebook and then splitting the remaining answers. The authors report observing no answers indicating a clear misunderstanding of the survey questions.

Why This Matters

Impact on research. The paper challenges a common shortcut in LLM evaluation: using one model to grade another. It shows that proxy LLMs are not merely noisy substitutes for people but systematically different — they converge where humans diverge, and they miss context humans find obvious, including clearly private data. The authors argue this calls for a taxonomy distinguishing objective-answer tasks (where consistency is desirable) from preference-sensitive tasks (where diversity is expected), so researchers know when deterministic proxy judgments are appropriate.

Real-world applications:

• Privacy benchmark design: Teams building or using benchmarks like ConfAIde and PrivacyLens need to know that a single proxy-LLM score may understate real user privacy risk.
• LLM product evaluation: Companies measuring whether an assistant is "helpful without leaking data" could misjudge user satisfaction if they rely only on automated judges.
• Personalization of privacy settings: The paper points toward personalized proxy models that approximate an individual's privacy and utility preferences rather than one universal answer.
• Regulatory and compliance assessment: The 493 seeds underlying PrivacyLens are grounded in regulations and privacy literature, so misestimating user perception affects how compliance-oriented evaluations are interpreted.

Industry relevance. Organizations deploying LLMs for email drafting, meeting summarization, and health or legal Q&A face a genuine privacy-helpfulness trade-off — aggressive redaction hurts utility, while detailed answers increase risk. The finding that proxy judges cannot capture the diversity of user preferences implies that automated evaluation pipelines, however cheap and consistent, may not be a reliable proxy for what actual users consider appropriate.

Future Directions

• Personalized proxy models. The authors suggest adapting LLMs to individual users, for example by using "privacy-conscious" personas or few-shot examples that show proxy LLMs a range of human evaluations and their justifications, so the models better mirror subjective sensitivity and reasoning.
• Non-deterministic evaluation designs. Rather than a single score, future proxy-LLM evaluations could be designed to surface the full range of plausible user judgments, including rationales for disagreement.
• Attribution-style analysis of scenario features. The paper proposes identifying which scenario features (data type, audience, purpose) most drive proxy LLM judgments, drawing on attribution methods from named entity recognition and computer vision.
• Cross-cultural and demographic validation. Because participants were U.S.-based and fluent in English, the authors explicitly leave open whether the consensus tendency and missed-nuance limitations of proxy judges manifest differently against more diverse populations and locations.

Target Audience

Researchers and practitioners in LLM evaluation, AI privacy, and human-computer interaction — particularly those who build privacy benchmarks or rely on LLM-as-a-judge pipelines. It is also relevant to product and policy teams at organizations that deploy LLMs on tasks involving personal data, and to anyone designing human-centered evaluations where subjective judgment matters more than consistency.

Authors’ abstract

Large language models (LLMs) are rapidly being adopted for tasks like drafting emails, summarizing meetings, and answering health questions. In these settings, users may need to share private information (e.g., contact details, health records). To evaluate LLMs' ability to identify and redact such information, prior work introduced real-life, scenario-based benchmarks (e.g., ConfAIde, PrivacyLens) and found that LLMs can leak private information in complex scenarios. However, these evaluations relied on proxy LLMs to judge the helpfulness and privacy-preservation quality of LLM responses, rather than directly measuring users' perceptions. To understand how users perceive the helpfulness and privacy-preservation quality of LLM responses to privacy-sensitive scenarios, we conducted a user study ($n=94$) using 90 PrivacyLens scenarios. We found that users had low agreement with each other when evaluating identical LLM responses. In contrast, five proxy LLMs reached high agreement, yet each proxy LLM had low correlation with users' evaluations. These results indicate that proxy LLMs cannot accurately estimate users' wide range of perceptions of utility and privacy in privacy-sensitive scenarios. We discuss the need for more user-centered studies to measure LLMs' ability to help users while preserving privacy, and for improving alignment between LLMs and users in estimating perceived privacy and utility.