PACIFIC: Can LLMs Discern the Psychometric Traits Influencing Your Preferences? Personality-Driven Preference Alignment in LLMs

AI summary

Overview

Research area: Natural Language Processing, specifically personalized preference alignment in Large Language Models, combined with psychometrics (Big-Five / OCEAN personality theory) and retrieval-augmented generation.

Technical level: Intermediate. The paper combines a psychometric annotation protocol, a synthetic dataset pipeline, prompt-engineering ablations, and contrastive retriever fine-tuning. Familiarity with RAG and DPR helps, but the psychological framing is explained from first principles.

One-sentence scope: The paper proposes PACIFIC, a framework that uses Big-Five personality traits as a latent organizing signal for user preference histories, and evaluates it on a newly constructed 1,200-example psychometrics-based dataset alongside the existing PrefEval benchmark.

What This Paper Is About

LLMs struggle to keep following a user's preferences as conversations grow — prior work cited in the paper finds preference-following accuracy can fall below 10% after only 10 turns (about 3k tokens). The authors argue that humans handle this differently: instead of recalling individual statements, people form an abstract impression of someone's personality and use it to generalize to new situations. PACIFIC tests whether Big-Five (OCEAN) traits — Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism — can serve as that same kind of stable latent representation, so that preferences from unrelated domains can be connected through shared psychometric alignment rather than surface-level topic similarity.

Key Contributions

1. Evidence that personality traits improve preference following. The authors show that trait-aligned preference history substantially improves personalized question answering over mixed-trait or no-preference context, and that given clean trait-aligned context, models approach ceiling accuracy (Gemini-2.5-Pro reaches 99.25%).

2. The PACIFIC framework. A pipeline for incorporating personality-aligned preferences during generation, including label-aware prompting strategies (explicit trait labels, labeling of preferences and/or answer choices, and an implicit persona reminder) and a persona-aware retriever.

3. A psychometrics-based dataset. 1,200 preference–query pairs spanning 20 diverse domains (e.g., travel, movies, education, finance, entertainment, consumer electronics), annotated with Big-Five trait directions and balanced across all ten high/low trait directions — 120 preference–query pairs per trait label. Released at huggingface.co/datasets/TylerZ0931/PACIFIC-big-five-trait-preferences.

4. PiRAG (Personality-informed RAG). A persona-aware contrastive fine-tuning step for a Dense Passage Retrieval retriever that raises label-free accuracy from 30.25% (pretrained retriever) to 43.00%, without any trait annotations at inference time.

Main Findings

• Personality context helps on the existing benchmark. On PrefEval, zero-shot accuracy is 46.25%; mixed-trait history yields only 67.50%; trait-aligned preferences reach 82.50%; contaminated trait-aligned context (trait-aligned preferences plus two mismatched preferences as noise) drops to 72.92%. The authors conclude that unrelated preferences distract the model and weaken preference following.

• The pattern replicates on the new dataset. On the psychometrics-based dataset with Gemma-3-4B-IT, zero-shot is 25.75%, mixed is 29.25%, aligned is 63.00%, and contaminated aligned is 61.75%. Introducing a moderate amount of mismatched preference information causes only a small degradation.

• The bottleneck is context construction, not reasoning. With clean trait-aligned context, strong models approach the ceiling — Gemini-2.5-Pro reaches 99.25% — which the authors say confirms that reasoning capability is not the limiting factor.

• Explicit preference labels help; labeling answer choices backfires. Adding ground-truth trait labels to preferences gives the highest accuracy of the prompting strategies (76.00% vs. 63.00% for few-shot). Labeling candidate answers instead degrades accuracy sharply to 37.75% (traits only) and 57.25% (labeled preferences + choices), driven disproportionately by low-trait conditions, which the authors attribute to positivity bias in the LLM.

• Implicit reminders activate latent persona reasoning. A lightweight instruction to consider the user's personality scores 67.00%, trailing explicit labels but clearly beating the preference-only baseline of 63.00%.

• Persona-aware retrieval requires contrastive fine-tuning. A pretrained DPR retriever yields 30.25% because it favors semantic similarity over persona consistency; PiRAG fine-tuning raises this to 43.00%. The authors describe 43% as a conservative floor from a deliberately simple DPR retriever.

• More history is not better. Expanding mixed-trait context from 5 to 25 preferences slightly reduces accuracy, from 61.5% to 59.5%.

• Some persona profiles benefit more. Across all 32 personality profiles (2^5 = 32), personas (O^L, C^L, E^L, A^L, N^H) and (O^L, C^L, E^L, A^L, N^L) achieve the highest accuracy when prompted with trait-aligned preferences.

• Models infer traits better from abstract choices than from stated preferences. Trait prediction from candidate choices reaches 85.69% overall while prediction from explicit user preferences reaches only 54.50% (Gemma-3-4B-IT, 200 samples). Trait-wise, the model identifies "High" traits from preferences near-perfectly but accuracy collapses to 0–40% on "Low" dimensions. The authors attribute this to Social Desirability Bias induced by RLHF, which may conflate psychometric "low" scores with normatively negative behavior or safety violations.

• Human validation supports the annotation protocol. With 15 annotators, choice validation against their own dominant trait reached an average accuracy of 78.22% with Fleiss' κ of 0.8599; the GPT-4o-mini evaluator reached a Cohen's κ of 0.9170 against human consensus; the resonance check reported a 91% approval rate.

• Distractors are genuinely hard. With the user preference withheld, an LLM scores 25% — random guess chance — confirming all four options are equally attractive absent the personality constraint.

Methodology in Plain English

Annotation protocol. Each preference statement and answer choice is scored on a 7-point Likert scale along all five OCEAN dimensions, with a confidence value attached. The protocol splits traits into two families. For Openness, Conscientiousness, Extraversion, and Agreeableness, the authors use a "Mirroring Strategy" grounded in Self-Congruity Theory: users seek experiences that reinforce their self-concept (a highly Conscientious user prefers highly structured tools). For Neuroticism, mirroring would be psychometrically wrong — an anxiety-prone user does not seek stressful experiences — so they use a "Compensatory Strategy" based on a Safety vs. Efficiency trade-off. Their illustration: for a traveler choosing between a 45-minute and a 3-hour layover, a Low-N user takes the tight 45 minutes to save time, while a High-N user takes the 3 hours for peace of mind. Continuous scores are discretized into high (score > 4), low (score < 4), or unclear (score = 4) labels.

Dataset construction. Starting from PrefEval's 20 topics, the authors generate preference–question–answer triples anchored explicitly to each of the ten high/low trait directions rather than deriving them from topics. A distractor constraint requires that at least one wrong choice looks plausible from the query alone but becomes clearly inappropriate once the preference is known. An LLM judge and an independent model (GPT-4o-mini) filter invalid or insufficiently challenging samples. Each validated example is also augmented with a brief scenario and a 2–4-turn dialogue.

Experimental design. For each target question, four context settings are compared: zero-shot (no preference), mixed traits (question-irrelevant preferences from different traits), aligned traits (question-irrelevant preferences from other domains sharing the target's trait), and contaminated aligned (trait-aligned preferences plus unrelated or mismatched ones). Evaluation uses four-choice multiple-choice accuracy. Context and test sets are disjoint: five preferences form the prompt profile, two non-overlapping statements form the test, repeated ten times without replacement to produce 50 contexts and 20 tests per trait, run with two random seeds (200 tests per seed) with means reported.

Models and retrieval. Four models are evaluated: Gemma-3-4B-IT and Llama-3-8B-Instruct (open-weight, run on local GPU) and Gemini-2.5-Pro and GPT-4o-mini (via API). For the label-free setting, the authors use a Dense Passage Retrieval architecture with BERT-based bi-encoders, then fine-tune it with contrastive pairs where queries and preferences sharing the same OCEAN trait are positives and mismatched traits are negatives.

Why This Matters

Impact on research. The paper reframes personalization as a representation problem rather than a recall problem. It argues that storing and retrieving preference statements is insufficient, and that the community should evaluate whether models can extract psychologically plausible, trait-driven structure from interaction history. The authors position their dataset and framework as the first to study the causal link between latent Big-Five traits and downstream preferences, shifting evaluation from rule adherence or fact recall toward psychometric reasoning. The social desirability bias finding also raises a concrete concern about RLHF-trained models systematically suppressing "low-trait" signals.

Real-world applications:

• Long-horizon conversational assistants that must serve users across many turns and domains without re-stating preferences.
• Recommender systems for travel, movies, entertainment, finance, and consumer electronics, where cross-domain preference transfer is valuable.
• Memory and personalization layers in deployed chat products, where a persona-aware retriever could replace or augment semantic similarity search over user history.
• Evaluation tooling for auditing whether a model's recommendations favor socially desirable options over genuinely preference-aligned ones.

Industry relevance. The label-free PiRAG result matters most for deployment: trait annotations are impractical in production, and the paper shows a fine-tuned retriever can improve over standard semantic retrieval without them. The finding that scaling context from 5 to 25 preferences slightly hurts accuracy is directly relevant to teams that currently assume more conversation history is better.

Future Directions

• Stronger retrieval encoders. The authors explicitly note that 43% is a conservative floor from a deliberately simple DPR retriever and that stronger encoders should raise it, implying headroom in the retriever itself.
• Mitigating social desirability bias. Since accuracy on "Low" trait dimensions from preferences collapses to 0–40%, an open question is how to prevent RLHF-induced biases from bottlenecking personalization.
• Closing the gap between label-aware and label-free performance. Ground-truth-label prompting remains higher than PiRAG, so the paper leaves open how to approach that level without trait annotations.
• Extending beyond the current dataset. The framework is evaluated on a synthetic dataset of 1,200 pairs plus PrefEval; whether the mechanisms hold on real, messy user history is not established in the reported content.

Target Audience

Researchers and practitioners working on LLM personalization, long-context user modeling, and retrieval-augmented generation will get the most from this paper. It is also relevant to human-computer interaction and recommender-systems researchers interested in psychometric user models, and to applied engineers building memory or personalization layers who need to know when preference history helps and when it hurts. Readers looking for a purely architectural or training-method contribution will find this paper is more about data construction, prompting, and retrieval design.

Authors’ abstract

User preferences are increasingly used to personalize Large Language Model (LLM) responses, yet reliably leveraging preference signals remains under-explored. In practice, preferences can be noisy, incomplete, or even misleading, which can degrade answer quality when applied naively. Motivated by the observation that stable personality traits shape everyday preferences, we introduce PACIFIC (Preference Alignment for Choices Inference via Five-factor Identity Characterization), a personality-driven preference alignment framework that uses Big-Five (OCEAN) traits as a principled "latent" signal for organizing and reasoning over user preference history. To systematically evaluate this framework, we construct a psychometrics-based dataset containing 1,200 preference-query pairs spanning diverse domains (e.g., travel, movies, and education), with comprehensive coverage of high and low Big-Five trait directions. Extensive experiments show that trait-aligned preferences substantially improve personalized QA: given clean, trait-aligned context, LLMs reach near-ceiling accuracy (up to 99%), confirming that reasoning capability is not the bottleneck. The challenge is that real histories are mixed-trait and unlabeled. We show the true bottleneck is retrieval: a persona-aware contrastive retriever (PiRAG) raises label-free accuracy from 30% to 43% over standard semantic retrieval, without any trait annotations at inference.