Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Bias

AI summary

Overview

Research area: Natural Language Processing / LLM evaluation — specifically demographic bias auditing and personalization (persona prompting) research.

Technical level: Intermediate. The conceptual argument is accessible, but the paper's evidence relies on correlation analysis across cue types, bootstrap confidence intervals on outcome ratios, and fixed-effects regressions.

Scope: The paper systematically tests whether four different ways of signaling the same demographic identity (first names, dialect, dialog history, and explicit descriptors) produce consistent conclusions about LLM personalization and bias, using 14.8 million first-person advice-seeking prompts across three models and three advice domains (healthcare, salary, legal) in a U.S. context for race (Black/White) and gender (male/female).

What This Paper Is About

LLM bias and personalization research commonly signals a user's demographic group using a single cue — most often a name — and treats that cue as a stand-in for the underlying demographic category. This paper asks whether that assumption holds: do different cues for the same group actually produce the same behavioral shifts and the same conclusions about group differences? The authors find that they do not, and then investigate why.

Key Contributions

1. A systematic multi-cue comparison. The authors present what they describe as the first systematic comparison of cue-conditioned model behavior across multiple demographic operationalizations for the same groups under controlled conditions, covering four cue types (name, dialect, dialog history, explicit descriptors), three tasks, and three models over 14.8 million prompts.
2. Demonstration of cue-dependent personalization. Within-group behavioral shifts induced by cues for the same group are only moderately correlated and vary substantially by cue type, with dialect diverging most.
3. Demonstration of cue-dependent bias. Inter-group Black/White outcome ratios vary by cue in both magnitude and direction, with genuine sign reversals occurring in 8 of 9 task–model cells for race and 8 of 9 for gender.
4. Mechanistic explanation. The paper attributes cross-cue inconsistency to two factors: how strongly a cue activates demographic inference in the model, and the linguistic features bundled with the cue (measured via Flesch–Kincaid grade level).

Main Findings

• Convergence within a cue type is high. Name-based cues drawn from three different name lists (Tzioumis 2018; Elder and Hayes 2023; Rosenman et al. 2023) correlate at roughly r ≈ 0.99 on average across tasks and races, despite minimal overlap in the underlying names. Dialog-history cues compared across the CAD and PRISM datasets correlate at roughly r ≈ 0.92.

• Convergence across cue types is weaker and uneven. Name-based cues and explicit demographic descriptors correlate at roughly r ≈ 0.84 on average. Dialog-history cues correlate with other cue types at roughly r ≈ 0.7. Dialect-based cues are the weakest: AAVE correlates with all other cue types at roughly r ≈ 0.49 on average for race.

• Group differentiation is weak. Within a given cue, Black and White response deviations correlate at roughly r ≈ 0.98, while the same group across different cue types correlates at roughly r ≈ 0.66. How identity is cued matters more to model behavior than which group is cued. Name-based cues and dialog history show nearly no separation between groups (r ≈ 0.99), while explicit descriptors show somewhat lower cross-group similarity (r ≈ 0.89).

• Bias conclusions reverse depending on cue. Name-based and dialog-history cues sit close to parity in Black/White outcome ratios, while dialect and explicit cues frequently depart from parity. Reversals — cue pairs falling on opposite sides of parity with non-overlapping 95% bootstrap confidence intervals — occur in 8 of 9 cells for race and 8 of 9 for gender, including all three salary cells for both attributes.

• Single-cue audits frequently misstate direction. Taking the majority direction across cues within a cell as reference, a single-cue evaluation reaches a discordant conclusion in 32.4% of cue–cell pairs for race and 22.6% for gender. Discordance concentrates in implicit cues: dialect departs from consensus in 6 of 9 race cells and dialog history in 5, whereas name-based cues rarely do and never for gender.

• Cue–group association strength varies sharply. Explicit statements identify Black users in 99.4% (LLaMA-3.1), 99.3% (GPT-5.2) and 65.1% (OLMo-2) of cases. Implicit cues are far weaker: dialect is the strongest implicit signal for LLaMA-3.1 (14.5%) and GPT-5.2 (9.7%); names yield 4.7–11.2% for LLaMA-3.1 and 8.6–35.6% for GPT-5.2; conversational context stays below 2% across models. OLMo-2 defaults to White for essentially all implicit cues.

• Signal strength and behavioral impact are separable. OLMo-2 never infers Black from AAVE, yet its dialect-driven disparities are the largest of any model, with Black/White ratios of 1.04, 1.08 and 0.92 across tasks. A cue can therefore shift behavior through linguistic form with no measurable demographic inference.

• Cues bundle linguistic features. Cue type explains 45% of the variance in Flesch–Kincaid grade level. Dialog-history cues raise grade level relative to no-cue prompts while AAVE lowers it, with all effects statistically significant.

• Model behavior tracks inferred race more than cued race. In regressions including cued race, inferred race, Flesch–Kincaid grade, and prompt fixed effects, readability is a significant predictor in 9 of 9 model–task cells and inferred race in 7 of 9. Where significant, inferred race shows consistently larger effects than cued race — for example, LLaMA-3.1 salary coefficients of $427 for inferred race versus $164 for cued race, and GPT-5.2 salary coefficients of $1239 versus $238.

Methodology in Plain English

The authors start from 501 first-person advice-seeking prompts drawn from Kearney et al. (2025), covering healthcare, salary, and legal questions where the correct answer should not depend on race or gender. Because 501 prompts is small, they augment the data: healthcare expands to 4,440 prompts (24 care-seeking question variants generated with Claude Sonnet 4.5, combined with 185 symptom templates), salary to 5,000 prompts (100 job profiles paired with 66 resampled cities), and legal to 5,000 prompts (197 scenario templates paired with 63 resampled cities).

They then attach demographic cues to these base prompts in four ways. Names come from three published name lists, retaining the 50 most strongly race-associated names per race–gender subgroup (200 names per source), introduced via a memory-style prefix such as [MEMORY: User name is NAME]. Dialect is applied by translating prompts from Standard American English into African American Vernacular English using GPT-5 nano, with outputs reviewed by a native AAVE speaker. Dialog history prepends prior user turns and preferred model responses drawn from the Community Alignment Dataset (CAD) and PRISM, restricted to U.S. annotators and the four race–gender groups; 50 clusters are subsampled. Explicit descriptors use third-person labels such as [MEMORY: User is a Black male], yielding 23 variants via Cartesian expansion.

Each cue is prepended to the base prompt, except dialect, which is applied by translation. The authors evaluate LLaMA-3.1 8B, OLMo2-7B, and GPT-5.2 across three random seeds (one seed for GPT-5.2, since the OpenAI API does not support seeds) with default decoding.

To test personalization, they compute prompt-level deviations from a no-cue baseline and correlate the deviation vectors within groups across cues, averaging correlations across models with a Fisher z transformation. To test bias, they compute Black–White outcome ratios per cue, task, and model, with 95% bootstrap confidence intervals. To explain the inconsistencies, they ask the models to infer user race from the same cued prompts (Black, White, or Unknown), use the frequency of correct Black inference as a proxy for cue–group association strength, measure Flesch–Kincaid grade level as a proxy for bundled linguistic complexity, and regress model responses on cued race, inferred race, readability, and prompt fixed effects.

Why This Matters

The paper argues that a widely used methodological shortcut in LLM bias and personalization research — treating any single demographic cue as a proxy for the demographic category — produces conclusions that reflect the choice of cue rather than stable model properties. The authors frame their results as evidence that apparent "race" or "gender" effects are better understood as responses to specific linguistic signals (dialectal forms, names, discourse patterns) rather than cue-invariant category-level parameters. This aligns with distributional accounts of language model behavior and with measurement-theoretic arguments that social constructs are instantiated through operationalization choices.

Real-world applications:

• Bias auditing and fairness evaluation: Auditors using a single name-based probe could report near-parity while dialect-based probing on identical prompts suggests substantial disparity, or vice versa.
• High-stakes deployment review: The three domains tested (healthcare, salary, legal) are ones where demographic-conditioned variation has direct consequences, so unstable estimates complicate regulatory and internal review.
• Prompt and persona design: Systems that personalize responses using user characteristics need to know that the signaled attribute, not just the attribute itself, shapes output.
• Research reporting standards: The paper recommends reporting results per cue with the range across cues rather than a single pooled value.

Industry relevance: For teams building or auditing conversational AI in healthcare triage, salary guidance, or legal information contexts, cue selection is a substantive measurement decision rather than a detail. The finding that model behavior aligns more closely with a model's own demographic inference than with the cued attribute suggests that behavioral safeguards and evaluations should target the signals models actually respond to.

Future Directions

• Identify further drivers of cue-dependent behavior. The authors note their low explanatory power is consistent with prior work and that which properties drive model behavior remains an open question. Concurrent work (Neplenbroek et al. 2026) adds conversation topic and emotion, finding topic is an important predictor of advice given a conversational history.
• Extend beyond advice-seeking. The study is limited to first-person advice-seeking interactions in a U.S. context; the authors state their findings do not generalize to other interaction types (creative writing, information retrieval, multi-turn deliberation) or geographic contexts.
• Disentangle gender from race. Gender analyses derive from prompts associated with Black and White individuals, so observed gender effects may partly reflect gender–race interactions. The authors also note they use a binary gender operationalization due to dataset limitations and plan to extend to non-binary identities.
• Move toward mechanism-aware evaluation. The authors recommend a cue audit prior to drawing demographic inferences, examining how strongly a cue activates demographic inference, what linguistic, structural, or affective properties it introduces, and what correlated social attributes it plausibly conveys.

Target Audience

Researchers and practitioners working on LLM fairness, bias auditing, and personalization evaluation; NLP methodologists interested in measurement validity and operationalization; and policy or product teams at organizations deploying conversational AI in high-stakes advice domains. Readers should be comfortable with correlation analysis, bootstrap confidence intervals, and regression coefficients, though the central argument — that cues are not interchangeable — is stated in plain terms throughout.

Authors’ abstract

Demographic cue-based evaluation is widely used to study how large language models (LLMs) adapt their responses to signaled demographic attributes within and across groups. This approach typically relies on a single cue (e.g., names) as a proxy for group membership, implicitly treating different cues as interchangeable operationalizations of a single underlying identity-conditioned behavior. We test this assumption in realistic advice-seeking interactions spanning 14.8 million prompts, focusing on race and gender in a U.S. context. We find that cues for the same group induce only partially overlapping changes in model responses, yielding inconsistent conclusions about personalization, while bias conclusions are unstable, with both magnitude and direction of group differences varying across cues. We further show that these inconsistencies reflect differences in cue-group association strength and linguistic features bundled within cues that shape model responses. Together, our findings suggest that demographic conditioning in LLMs is not a cue-invariant category-level parameter but depends fundamentally on how identity is cued, reflecting responses to linguistic signals rather than stable demographic categories. We therefore call for multi-cue, mechanism-aware evaluations as a foundation for robust and interpretable claims about demographic variation in LLM responses.