Exposing Hidden Biases in Text-to-Image Models via Automated Prompt Search

AI summary

Overview

Research area: Fairness and bias in generative AI — specifically text-to-image (TTI) diffusion models — combined with automated prompt optimization using large language models.

Technical level: Intermediate. Readers should be comfortable with diffusion models, classifier-free guidance, beam search, and the general idea of probing model internals with linear classifiers.

Scope: The paper introduces Bias-Guided Prompt Search (BGPS), a framework that automatically discovers natural-sounding prompts which maximize demographic bias in TTI diffusion models, including models that have already been debiased.

What This Paper Is About

Text-to-image models like Stable Diffusion reliably skew their outputs along gender, race, and age lines, and the usual way of measuring this relies on hand-written or LLM-written prompt sets. Those prompt sets are expensive to build and only cover a small slice of prompt space, so they systematically miss the subtle, unanticipated phrasings that trigger bias — especially in models that have been "debiased" and therefore look clean on standard benchmarks.

The goal of this paper is to flip the evaluation problem around: instead of testing a model with curated prompts, automatically search for the prompts that produce the most biased images, while keeping those prompts readable and natural so a human auditor can understand what went wrong.

Key Contributions

1. Bias-Guided Prompt Search (BGPS): A framework that jointly maximizes (a) a bias score derived from linear classifiers trained on the diffusion model's internal UNet activations and (b) the likelihood of the prompt under a language model. This turns a gradient-free prompt-inversion technique (Visually-Guided Decoding) into a bias-discovery tool.

2. Exposure of hidden biases in debiased models: BGPS surfaces prompts that skew gender distributions up to 76% male in a debiased Stable Diffusion 1.5, despite that model showing balanced results on the curated prompts its mitigation method was designed around.

3. Bias discovery beyond occupations: Unlike most prior benchmarks (TIBET, HEIM, HRS, GELDA, etc.), BGPS finds bias in prompts about objects, activities, contexts, and places — not just profession templates.

4. A quantitative comparison against gradient-based prompt optimization: The hard-prompt optimizer PEZ produces prompts with 17–26× worse perplexity and that reveal the target attribute 94–100% of the time, showing that prior optimization-based approaches are unusable as auditing tools.

Main Findings

• All mainstream models are biased, and BGPS can push them further: SD 1.5, SD 2.1, SDXL, Flux, and SD 3.5 all show gender, race, or age skews. BGPS finds prompts that push the male attribute up to 91% in SD 1.5 and up to 95% in SDXL — substantially beyond what unguided or LLM-only baselines achieve.

• Debiasing generalizes poorly: Two state-of-the-art debiased SD 1.5 variants (LoRA fine-tuning from Shen et al., and Difflens from Shi et al.) look balanced on curated prompts, yet BGPS still finds prompts skewing up to 67–76% male. The Difflens model also appears to overcorrect, and BGPS counteracts that overcorrection.

• Bias follows systematic linguistic associations: Terms related to technology, music, and thought ("screens", "computer", "saxophone", "thoughtful", "focusing") push toward male representation. Terms related to arts, crafts, literature, and emotion ("creating", "library", "garden", "cozy", "tending") push toward female representation.

• Small modifiers cause large shifts: Adding "with intense focus" to "scientist" shifts the gender distribution from roughly 65% to 95% male. Adding "ethereally beautiful" to "artist" produces 84% female images. The paper argues these modifiers, not the occupation noun, carry most of the bias signal.

• Biases extend past occupation stereotypes: BGPS increases target-gender proportions in all four tested non-occupational scenarios: a person with an object, doing an activity, in a context, and in a place.

• Qualitative patterns reveal stereotyped depictions, not just demographics: Male-coded prompts use serious, professional, focused language; female-coded ones use emotive, aesthetic, or clothing-related language. Black-coded prompts surface political framings and incongruous sports references, while white-coded prompts use professional, technical language.

• Gradient-based hard prompt optimization fails as an auditing tool: PEZ matches or exceeds BGPS on raw bias frequency in a few settings, but its prompts are unreadable (perplexity 1,387–2,817 versus BGPS's 46–148) and almost always disclose the target attribute (94–100% for gender), making them adversarial attacks rather than interpretable evaluation.

• Higher classifier weight helps without hurting naturalness: Increasing λ from 10 to 100 yields stronger bias with little cost in perplexity, suggesting the language prior is strong enough to keep prompts readable even when the bias objective dominates.

Methodology in Plain English

The approach combines three ingredients.

A language model as the prompt generator. An instruction-tuned Mistral-7B is told to write attribute-neutral prompts (e.g., avoiding the words "man", "woman", or gendered names) so that any bias it later discovers comes from the model, not the prompt.

Attribute classifiers as the bias detector. Lightweight linear classifiers are trained ahead of time on the diffusion model's intermediate UNet activations to predict demographic attributes (male/female, white/black) from a generated image. Because these classifiers read internal representations rather than pixel outputs, they give a fast, differentiable-in-effect signal about where the diffusion model is heading before the image is even fully formed.

Beam search with a combined score. The system generates prompts token by token. At each step it keeps a beam of candidate prompts and scores each one as:

(log probability of the prompt under the LLM) + λ × (log average classifier bias score over K generated images)

The first term keeps text natural and on-instruction; the second term pushes toward prompts that produce biased images. The search is made stochastic — the first token is sampled from the full LLM distribution, and beams are resampled with an expansion factor — so that repeated runs yield diverse prompts rather than one deterministic answer.

Evaluation. For each method, 100 prompts are generated, attribute-revealing prompts are filtered out (with LLM assistance), 10 images are generated per prompt, and each image is classified. Three metrics are reported: mean attribute frequency with 95% confidence intervals (bias), GPT-2 perplexity (naturalness), and the fraction of prompts that leaked the attribute (interpretability).

Why This Matters

Impact on research. The paper reframes bias evaluation from a fixed-benchmark problem to a search problem. That shift has a direct consequence for the debiasing literature: a mitigation method that scores well on a curated dataset no longer provides much evidence that the underlying bias is gone. BGPS also supplies a reusable diagnostic — the prompts it discovers can be fed back into the training or test sets of mitigation methods to close the gaps they left open.

Real-world applications

• Content moderation and model auditing: Platforms that host or deploy image generators can run BGPS as a pre-release red-team step and flag prompts that reliably produce skewed demographics.
• Stock photography and advertising: Agencies using generated imagery for commercial campaigns can check whether their models produce inclusive outputs for a given brief, rather than only for generic "a photo of a doctor" prompts.
• Regulatory compliance: Frameworks like the EU AI Act require bias documentation for high-risk AI systems; automated prompt search offers a systematic way to produce that documentation.
• Media and publishing: Newsrooms and publishers using TTI for illustration can test whether specific contextual descriptors in their style guides inadvertently skew who appears in images.

Industry relevance. The method needs only grey-box access to intermediate activations, not gradients through the full diffusion chain, so it is far cheaper to run than gradient-based prompt optimization. The outputs are readable by non-technical stakeholders, which matters when bias audits must be reviewed by legal, policy, or editorial teams rather than ML engineers.

Future Directions

• Extending to more attributes and intersectionality. The paper tests gender, race, and (in the appendix) age. Disability, religion, nationality, socioeconomic status, and combined attributes like race-plus-gender remain open, and the linear classifiers would need to be retrained for each.

• Using discovered biases to improve mitigation. The most direct next step is to feed BGPS-generated prompts into the training or calibration procedures of debiasing methods and measure whether that closes the residual gaps BGPS exposes.

• Generalizing beyond image diffusion. The objective is model-agnostic in principle and could be adapted to video generation, autoregressive image models, and any multimodal system with a text prompt and an internal representation of the output.

• Understanding why these associations exist. The paper documents linguistic clusters that trigger bias (technology → male, crafts → female) but does not trace them back to the training data. Causal analysis of which data sources produce which associations would inform better data curation.

• Scaling and standardizing the audit pipeline. Questions remain about how many prompts constitute a sufficient audit, how stable the discovered biases are across seeds and checkpoints, and whether BGPS scores can be turned into a comparable benchmark metric across models.

Target Audience

Primary: Fairness, accountability, and transparency researchers working on generative models; TTI model developers and red-teamers; and NLP/CV graduate students studying bias in vision-language systems.

Secondary: Practitioners deploying image generators in commercial products who need a lightweight audit tool; policy and compliance staff who must interpret bias findings without deep ML expertise; and researchers building debiasing methods who need a harder evaluation protocol than curated prompt sets provide.

The paper is written for readers with a working knowledge of diffusion models and LLM decoding — the math is presented clearly but assumes familiarity with classifier-free guidance, beam search, and linear probing.

Authors’ abstract

Text-to-image (TTI) diffusion models have achieved remarkable visual quality, yet they have been repeatedly shown to exhibit social biases across sensitive attributes such as gender, race and age. To mitigate these biases, existing approaches frequently depend on curated prompt datasets - either manually constructed or generated with large language models (LLMs) - as part of their training and/or evaluation procedures. Beside the curation cost, this also risks overlooking unanticipated, less obvious prompts that trigger biased generation, even in models that have undergone debiasing. In this work, we introduce Bias-Guided Prompt Search (BGPS), a framework that automatically generates prompts that aim to maximize the presence of biases in the resulting images. BGPS comprises two components: (1) an LLM instructed to produce attribute-neutral prompts and (2) attribute classifiers acting on the TTI's internal representations that steer the decoding process of the LLM toward regions of the prompt space that amplify the image attributes of interest. We conduct extensive experiments on Stable Diffusion 1.5 and a state-of-the-art debiased model and discover an array of subtle and previously undocumented biases that severely deteriorate fairness metrics. Crucially, the discovered prompts are interpretable, i.e they may be entered by a typical user, quantitatively improving the perplexity metric compared to a prominent hard prompt optimization counterpart. Our findings uncover TTI vulnerabilities, while BGPS expands the bias search space and can act as a new evaluation tool for bias mitigation.