RAIGen: Rare Attribute Identification in Text-to-Image Generative Models

AI summary

Overview

Research area: Computer vision and generative AI, specifically bias auditing and interpretability of text-to-image diffusion models.

Technical level: Intermediate. The paper assumes familiarity with diffusion models, CLIP embeddings, and autoencoder-based interpretability, though the core idea (find rare concepts hiding inside a model) is explainable without that background.

Scope: The paper introduces RAIGen, a framework that uses Matryoshka Sparse Autoencoders to discover semantically meaningful attributes that a text-to-image model has learned but rarely generates, without requiring predefined fairness categories.

What This Paper Is About

Text-to-image models like Stable Diffusion generate stereotyped outputs: ask for "a doctor" and you get mostly one gender, one ethnicity, one framing. Existing bias research attacks this in two limited ways — it either measures known categories (gender, race) or it flags whatever majority attribute dominates the outputs. Neither approach asks the reverse question: which concepts has the model actually learned and could generate, but almost never does? RAIGen answers that question by looking inside the model's internal representations rather than at externally defined labels, identifying rare attributes that are encoded but systematically suppressed.

Key Contributions

1. First label-free framework for rare attribute discovery in diffusion models. RAIGen shifts bias auditing from predefined fairness categories and majority-dominance detection to the systematic identification of underrepresented attributes that are already present in the model's learned feature space.

2. A minority score combining rarity and semantic distinctiveness. The score multiplies a neuron's activation frequency by how far its activation-weighted CLIP centroid sits from the global dataset centroid, so that only neurons that are both infrequent and semantically separated get ranked highly.

3. Demonstrated generality across architectures and attribute types. RAIGen surfaces attributes beyond fairness categories (stylistic, contextual, compositional), scales from Stable Diffusion v1.4 to SDXL, and transfers to the transformer-based FLUX.1-schnell.

4. Evidence that discovered attributes are both real and actionable. A 25-participant user study confirms the attributes are perceptibly rare in generated images, and lightweight prompt revision guided by RAIGen's annotations measurably increases their presence.

Main Findings

• Activation frequency is a valid rarity proxy in controlled settings. In a tree-structured toy dataset with known feature frequencies, the least-active Matryoshka SAE latents disproportionately mapped to genuinely rare ground-truth features, with a mean Spearman correlation of about 0.991 between latent firing rate and true feature frequency across 20 seeds.

• RAIGen attributes are far rarer than majority attributes surfaced by prior work. Attribute Presence scores (the fraction of images containing the attribute) were 0.205 on WinoBias and 0.220 on COCO for RAIGen versus 0.941 and 0.933 for OpenBias on Stable Diffusion v1.4. On SDXL the RAIGen figures were slightly lower still (0.194 and 0.199), suggesting more model capacity does not translate into more expression of rare modes.

• Discovery extends well beyond fairness categories. Alongside socially salient cases like "female doctor," RAIGen surfaced contextual and stylistic attributes such as "doctor in a framed portrait," "doctor with a medical chart in the background," "afro-curly textured hair," "a woman explicitly holding a camera," and "front-facing trains with large smoke plumes."

• Humans confirm the attributes are rare in practice. In a user study with 25 participants across five WinoBias professions, participants reported RAIGen attributes in fewer than 3 out of 10 images on average. The rarest case was CEO at 0.70 out of 10; even the most present, Sheriff, averaged only 2.64.

• Prompt revision amplifies the discovered attributes. Injecting RAIGen's annotations into prompts cut the deviation from a uniform attribute distribution roughly in half — from 0.50 to 0.22 on SD v1.4 and from 0.49 to 0.23 on SDXL — while prompt alignment dropped only marginally (about 0.5 and 0.4 points respectively).

• Transformer-based diffusion works but is messier. On FLUX.1-schnell, discovered attributes reached an Attribute Presence of 0.11, and top-ranked neurons showed coherent modes like "female doctors with curly/afro-textured hair" and "masked doctors." However, a higher share of high-scoring neurons had diffuse, weakly localized heatmaps, which the authors attribute to the choice of transformer hook point lacking the spatial grounding of U-Net bottlenecks.

Methodology in Plain English

The pipeline rests on one intuition: if a concept exists inside a model's internal representation, some component of that representation should light up when the concept is present, even if the concept rarely makes it into the final image.

Step 1 — Look inside the model while it draws. For a set of prompts, the researchers generate images and record the intermediate bottleneck representations produced at each denoising step. They use the final timestep, where semantics are most fully formed.

Step 2 — Decompose those representations into interpretable parts. A standard autoencoder compresses data; a sparse autoencoder instead expands it into many more features than inputs, with only a few active at a time, which tends to make individual features correspond to human-interpretable concepts. The paper uses a Matryoshka sparse autoencoder, which is trained at several nested sparsity levels at once, producing a hierarchy from coarse, broad concepts down to fine details. The authors deliberately work only at the coarsest level, because finer levels fragment one concept into many brittle sub-features and inflate the search space with spurious candidates.

Step 3 — Score every feature for "minority-ness." Each neuron gets two measurements. The first is activation frequency: the fraction of samples in which it fires at all. The second is semantic distinctiveness: the cosine distance between the neuron's CLIP centroid — the average CLIP embedding of the images it responds to, weighted by activation strength — and the global CLIP centroid of the whole image set. Frequency alone is noisy in real data, since some rarely active neurons are just meaningless. Multiplying distinctiveness by one-minus-frequency means a neuron needs both signals to score highly.

Step 4 — Rank, deduplicate, and label. Neurons are sorted by score, and any neuron whose semantic centroid is too close to an already-retained one is dropped, so the final set is compact and diverse. A multimodal language model then writes a human-readable annotation for each surviving neuron, which is also used to turn the attribute into a verification query.

Step 5 — Verify and act. Each attribute is checked by asking two vision-language models whether it appears in each generated image and aggregating their answers. The annotations are then injected back into the prompts to see whether the model can be nudged toward producing the rare modes.

An important nuance: the authors call their method label-free rather than fully unsupervised. No minority categories or attribute labels are needed anywhere in the discovery pipeline, but a pretrained CLIP-style encoder is still required to define semantic distinctiveness. It supplies semantic geometry, not attribute supervision.

Why This Matters

Impact on research. Bias work in generative models has largely been a measurement problem framed around categories researchers already care about. RAIGen reframes it as a discovery problem: the space of things a model underproduces is much larger than the space of things anyone thought to measure. It also supplies a concrete bridge between mechanistic interpretability tools like sparse autoencoders and the fairness auditing literature, showing that internal features can be turned into externally verifiable claims about model behavior.

Real-world applications:

• Pre-deployment auditing. A developer could run RAIGen on a model before release and get a ranked list of concepts the model silently suppresses, without knowing in advance which categories to check.
• Regulatory and compliance documentation. Emerging AI regulations demand evidence about representational harms; RAIGen produces quantitative, model-specific artifact sets that are auditable across model versions.
• Creative and design tooling. Artists and product teams could use the discovered attributes as a palette for diversifying outputs, or to deliberately explore low-density regions of a model's output space.
• Dataset and training diagnostics. Rare internal features point to concepts that exist but are crowded out, informing data curation and fine-tuning decisions.

Industry relevance. Companies shipping generative image tools face both reputational and regulatory pressure around representation. RAIGen offers an automated way to produce a minority-attribute inventory per model version and to test whether lightweight mitigation actually moves the needle — the prompt-revision experiment shows a cheap intervention can meaningfully rebalance outputs while preserving most of the original prompt's semantics.

Future Directions

• Better hook point selection in transformer diffusion. The FLUX experiments showed more weakly interpretable neurons than in U-Net models. Systematically searching across blocks and submodules (attention versus MLP streams) could make rare-attribute discovery as clean in transformers as it is in U-Nets.

• Comparing rare attributes across architectures. The paper establishes that RAIGen works on SD v1.4, SDXL, and FLUX, but does not systematically characterize how the sets of suppressed attributes differ between them. Understanding whether rare modes are architectural artifacts or data-driven would sharpen the auditing story.

• Moving beyond prompt revision as mitigation. Prompt revision is a deliberately simple proof of concept. The authors note RAIGen is agnostic to the downstream method, leaving open whether embedding-space interventions, guidance modifications, or fine-tuning can amplify rare attributes more effectively and at lower cost to prompt alignment.

• Keeping pace with model scale. SDXL showed slightly lower attribute presence for RAIGen attributes than SD v1.4, which raises the question of whether larger models suppress rare modes more aggressively — and whether the minority score needs recalibration as model capacity grows.

Target Audience

Generative AI fairness and bias researchers will find the direct contribution to their literature. Interpretability researchers working with sparse autoencoders will find a new downstream application for their tooling. Practitioners responsible for auditing or deploying text-to-image systems will find a concrete, automatable pipeline. Machine learning engineers and graduate students with intermediate background in diffusion models and embedding spaces will be able to follow the method; the text-heavy math in the preliminaries is standard and skippable for readers interested only in the empirical claims.

Authors’ abstract

Text-to-image diffusion models achieve impressive generation quality but inherit and amplify training-data biases, skewing coverage of semantic attributes. Prior work addresses this in two ways. Closed-set approaches mitigate biases in predefined fairness categories (e.g., gender, race), assuming socially salient minority attributes are known a priori. Open-set approaches frame the task as bias identification, highlighting majority attributes that dominate outputs. Both overlook a complementary task: uncovering rare or minority features underrepresented in the data distribution (social, cultural, or stylistic) yet still encoded in model representations. We introduce RAIGen, the first framework, to our knowledge, for label-free rare-attribute discovery in diffusion models, requiring no predefined minority categories. RAIGen leverages Matryoshka Sparse Autoencoders and a novel minority metric combining neuron activation frequency with semantic distinctiveness to identify interpretable neurons whose top-activating images reveal underrepresented attributes. Experiments show RAIGen discovers attributes beyond fixed fairness categories in Stable Diffusion, scales to larger models such as SDXL, supports systematic auditing across architectures, and enables targeted amplification of rare attributes during generation. The project page is available at https://vssilpa.github.io/RAIGen_webpage/ .