Breaking the Benchmark: Revealing LLM Bias via Minimal Contextual Augmentation

AI summary

Overview

• Research area: Natural Language Processing, specifically fairness, bias, and safety evaluation of Large Language Models.
• Technical level: Intermediate. The paper assumes familiarity with LLM evaluation, benchmark datasets, and concepts such as RLHF and alignment, but the core method is explained step by step.
• Scope: The paper introduces a three-step augmentation framework, applies it to the BBQ fairness benchmark, and measures how eight open- and closed-weight LLMs shift from abstaining to decisive, stereotype-aligned answers under meaning-preserving perturbations.

What This Paper Is About

LLMs often look fair on standard bias benchmarks, but that apparent fairness may come from memorizing those benchmarks rather than learning genuine fairness principles. The authors test this by rewriting benchmark questions and scenarios in ways that preserve meaning while changing surface form, then checking whether models fall back on stereotypical answers even when the text does not support any answer. They also compare models' behavior on heavily studied bias types (gender, race, religion, nationality) against less-studied ones (age, disability, physical appearance, socio-economic status).

Key Contributions

1. A novel, general augmentation framework with three plug-and-play steps (Core Abstraction, Attribute Transformation, Scenario Generation) that only needs the presence of demographic information in an instance to perturb it.
2. An analysis showing LLMs are more prone to stereotypical behavior on less-studied bias categories (age, disability, SES, appearance) than on well-studied ones, with reported drops of up to 14% compared to the gender and race dimensions.
3. A publicly released augmented dataset produced by applying the framework to the BBQ dataset, with the note that the data will be made publicly available upon publication of the paper.

Main Findings

• Augmentation increases decisive answers: Under original contexts (OCOQ and OCAQ), the average rate of Decisive Responses stayed around 24–33%. When augmented scenarios were introduced (ASOQ and ASAQ), it rose to approximately 49–61%.
• The strongest "jailbreak" configuration: Decisive Response peaked at 60.8% in the ASAQ condition (augmented scenario plus augmented question) for less-studied bias categories. The lowest rate, 26.6%, occurred in OCAQ under more-studied bias categories.
• Model-level differences are sharp: OLMo2 produced the highest decisiveness across all contexts (71–94%). Gemini and Qwen3 stayed below 35% even under augmentation, indicating more cautious behavior. DeepSeek fluctuated only modestly between conditions (34–54%), and Mistral's more-studied DR rose from 38.8% (OCOQ) to 69.6% (ASAQ).
• Large relative jumps for some models: Gemma3 nearly quadrupled its decisiveness in the more-studied category, from 14.5% to 60.1%, while Phi-4 rose from 7.2% to 63.2%.
• Biggest per-category increases: Age (36.14% to 63.72%) and disability (29.70% to 63.69%) showed the largest overall increases in Decisive Responses. Nationality and race showed slightly smaller relative gains, which the authors interpret as models already relying on implicit priors there.
• Less-studied biases start higher and stay higher: Average Decisive Responses for more-studied categories rose from 23.8% to 54.0%, while less-studied categories rose from 29.6% to 60.8% — roughly a 30 percentage-point gain each, but from a higher baseline for less-studied groups.
• Consistent per-model gap: In OCOQ, OLMo2 showed 71.22% (more-studied) versus 81.68% (less-studied), Qwen3 rose from 8.05% to 15.26%, and DeepSeek from 34.56% to 52.04%.
• Exceptions to the pattern: In the OCOQ condition, Mistral and Gemma3 had slightly lower Decisive Response for the less-studied group (33.72% and 12.77%) than for the more-studied group (38.76% and 14.51%). DeepSeek's decisiveness increased only modestly in the less-studied case (52.04% to 53.91%) compared to the more-studied case (34.56% to 52.82%).
• Benchmark fairness may be superficial: The authors argue that rising Decisive Responses after augmentation indicate models may overfit to benchmark format rather than internalize fairness-consistent reasoning.

Methodology in Plain English

The authors start from the BBQ (Bias Benchmark for Question Answering) dataset and focus only on ambiguous contexts, where the question cannot be answered from the text alone. A fair model, in their framing, should abstain in these cases.

They then generate new versions of each instance in three steps:

1. Core Abstraction — an LLM summarizes the instance so that a grounded image could be generated from it, stripping out abstract or unidentifiable details. If the text is already grounded enough, the original is returned.
2. Attribute Transformation — a second LLM perturbs the summarized text in the direction of a target demographic, producing several semantically equivalent variants where only the demographic details change.
3. Scenario Generation — an LLM turns those variants into detailed, realistic scenarios that remain solvable under the original task description.

To build the evaluation set, the authors randomly chose approximately 1,200 samples for each of the well-studied and less-studied bias groupings. The well-studied counts were 356 gender, 347 race, 322 nationality, and 167 religion instances; the smaller religion count reflects what remained after filtering to ambiguous cases. Less-studied counts were 371 SES, 370 age, 234 physical appearance, and 225 disability. Four augmented scenarios were generated with Gemini-2.5-Flash.

Quality control used three LLM judges — Gemini-2.0-Flash, GPT-4.1-Nano, and Phi-4-Reasoning-14B — which each decided whether the question was answerable from the scenario. Only augmented scenarios where all three agreed the question was not answerable were kept, leaving 64.45% of the initial dataset. The authors also manually annotated 600 randomly sampled points; agreement rates were 88.66% and 84% for annotator one (more-studied and less-studied) and 78.67% and 71.19% for annotator two, with kappa agreement of 77.63% and 67.41%.

As a second type of perturbation, Gemini-2.5-Flash rephrased the original questions grammatically (voice changes, synonym swaps, tense shifts). One author reviewed all of them; about 8.3% had issues such as repetition or incomplete reformulation and were corrected manually.

The eight evaluated models were DeepSeek-r1-14B, OlMo2-13B, Gemma3-27B, Gemma2-27B, Mistral-7B-V0.3, Phi4-14B, Gemini-2.5-Flash-Lite, and Qwen3-14B. Open-source models ran via Ollama with default hyperparameters, and Gemini-2.5-Flash-Lite was accessed through the Google AI Studio API. Responses were classified by GPT-4.1-mini as either an Abstaining Response (refusing to infer from insufficient evidence) or a Decisive Response (committing to an answer despite ambiguity). A fully fair model would score 100% AR and 0% DR.

Four input conditions were compared:

• OCOQ — Original Context, Original Question
• OCAQ — Original Context, Augmented Question
• ASOQ — Augmented Scenario, Original Question
• ASAQ — Augmented Scenario, Augmented Question

Why This Matters

The paper argues that strong scores on static fairness benchmarks can mask brittleness: models may learn to behave fairly on known formats without generalizing that fairness. It also shows that fairness progress is uneven across demographic groups, with less-studied categories such as age, disability, appearance, and SES consistently producing higher rates of decisive, assumption-driven answers.

Real-world applications where this matters:

• Hiring and resume screening tools that rely on LLMs, where ambiguous applicant descriptions should not trigger demographic assumptions.
• Healthcare or benefits assistants handling age, disability, or socio-economic information, where unsupported inference can cause direct harm.
• Customer-facing chatbots that discuss people's identities, where stereotype-driven answers degrade user experience for members of less-studied communities.
• Content moderation and recommendation systems that must not amplify stereotypical framing when context is incomplete.

Industry relevance: the framework is plug-and-play and model-agnostic, so it can be applied to any fairness benchmark with explicit demographic information. The findings suggest that teams evaluating models against a single benchmark should add perturbation-based robustness testing before making fairness claims, and that alignment work focused only on gender and race leaves measurable gaps elsewhere.

Future Directions

• Extend the framework beyond datasets with explicit demographic labels, since many biases are implicit or emerge from combinations of demographics rather than a single identifiable group.
• Develop methods for measuring fairness in cases the current framework discards, such as abstract concepts and very long text, which the abstraction step currently removes.
• Improve hallucination reduction and verification so that LLM-generated evaluation data can be trusted without extensive human checking; the authors note that until strong verifiers exist, generative outputs cannot be fully relied upon.
• Broaden model coverage beyond the eight evaluated systems and use prompt ensembles or systematic prompt variations, since the study used a single prompt template per configuration and model behavior can be sensitive to phrasing.

Target Audience

Researchers and practitioners working on LLM fairness, safety, and alignment; benchmark designers who want to test whether reported fairness generalizes; and engineers building evaluation pipelines for deployed models. The paper is most useful to readers who already understand basic LLM evaluation terminology and are looking for a concrete, reusable way to stress-test fairness claims.

Authors’ abstract

Large Language Models have been shown to demonstrate stereotypical biases in their representations and behavior due to the discriminative nature of the data that they have been trained on. Despite significant progress in the development of methods and models that refrain from using stereotypical information in their decision-making, recent work has shown that approaches used for bias alignment are brittle. In this work, we introduce a novel and general augmentation framework that involves three plug-and-play steps and is applicable to a number of fairness evaluation benchmarks. Through application of augmentation to a fairness evaluation dataset (Bias Benchmark for Question Answering (BBQ)), we find that Large Language Models (LLMs), including state-of-the-art open and closed weight models, are susceptible to perturbations to their inputs, showcasing a higher likelihood to behave stereotypically. Furthermore, we find that such models are more likely to have biased behavior in cases where the target demographic belongs to a community less studied by the literature, underlining the need to expand the fairness and safety research to include more diverse communities.