CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMs

AI summary

Overview

Research area: Natural Language Processing — LLM safety, societal bias evaluation, and adversarial red-teaming of conversational AI.

Technical level: Intermediate. The work involves no gradient-level math, but it assumes familiarity with chat-completion APIs, conversation-history handling, jailbreak categories, and LLM-as-a-judge evaluation.

Scope: This paper introduces CoBia, a single-query adversarial technique that fabricates a conversation in which the model itself appears to have made a biased claim, then measures whether 11 LLMs amplify or recover from that bias across six socio-demographic categories.

What This Paper Is About

Large language models increasingly pass standard safety checks, yet still slip into biased or harmful statements during extended conversations. The authors ask what happens when a user deliberately constructs the conversation history so that the model appears to have already endorsed a stereotype about a social group — does the model correct itself, or does it continue and amplify the bias? CoBia is a lightweight attack suite that builds exactly that scenario in a single query, and the paper uses it to stress-test how robust current safety guardrails really are.

Key Contributions

1. The CoBia methods: A set of lightweight adversarial attacks that expose hidden societal biases in LLMs using only a single query, in two variants — HCC (history-based constructed conversation) and SCC (single-block constructed conversation).
2. The CoBia dataset: A unified, de-duplicated dataset of 112 distinct social groups with associated negative descriptors across six socio-demographic categories, derived from three existing datasets (RedditBias, SBIC, StereoSet), totaling 2,841 entries.
3. A comparative evaluation: Societal bias scores for 11 LLMs from nine organizations, benchmarked against three baselines (0-Shot, DAN 13.0, and R-Play) and scored with three automated judges.
4. A validation study: Cross-checking of two LLM-as-a-judge systems and Granite Guardian against human annotations from four human evaluators, using pairwise agreement, Cohen's kappa, and Fleiss' kappa.

Main Findings

• CoBia beats established jailbreaks at surfacing bias. Both HCC and SCC outperformed the 0-Shot, DAN, and R-Play baselines in most settings under both the Bias Judge and Granite Judge, with the same pattern under the NLI Judge.
• Three models were "heavily biased." Using a threshold of a UCC (union of HCC and SCC) score above 80% under the Bias Judge, the authors flagged llama3.3:70b (85.54), qwen2.5:7b (83.60), and command-r:35b (82.59) as heavily biased. Every model above this threshold also exceeded 40% under the Granite Judge and 69% under the NLI Judge.
• Baselines look safe by comparison. R-Play, DAN, and 0-Shot showed comparatively lower bias scores, often below 20%, which the authors attribute to the safety attention these well-known methods have already received.
• Two models were resistant. gemma2:27b and deepseek-v2:16b showed notably low CoBia-based bias scores under both the Bias Judge and the Granite Judge; gemma2:27b also had a particularly low 0-Shot score, suggesting broad robustness. deepseek-v2:16b instead produced long, vague, hedging responses that judges often classified as unbiased.
• HCC and SCC are complementary. For mistral:7b, llama3.1:8b, and phi4:14b, HCC yielded consistently higher bias rates than SCC across all three judges; for gpt-4o-mini the opposite held. For deepseek-v2:16b and gemma2:27b the gap between the two was under 3%, yet combining them raised the overall bias score by at least 30%.
• National origin is the most reliably triggered category. National origin showed the highest bias levels in all settings; qwen2.5:7b and command-r:35b scored near 1.0 there under UCC. Race, religion, and orientation generally scored lower — in gemma2:27b these three stayed below 0.2 — suggesting closer monitoring or data filtering for those dimensions.
• Bigger models are not automatically safer. Across the qwen2.5 family (3B, 7B, 14B, 32B, 72B) there was no clear correlation between model size and bias score. The 3B model was more biased under SCC than HCC or 0-Shot, while the 32B model showed the best safety performance, for which the authors offer no obvious explanation.
• Judges disagree in instructive ways. The Bias Judge reported the highest scores overall. Granite Judge produced lower scores across all methods, which the authors suggest may stem from its training on shorter responses. NLI Judge gave higher scores to the 0-Shot method than other judges did.
• Human alignment was strongest for the NLI Judge in disagreement cases. Where the Bias Judge and Granite Judge flagged DAN outputs as biased but the NLI Judge did not, the NLI Judge matched the majority vote of all four human annotators in 83% of those cases. Pairwise agreement between the NLI Judge and Bias Judge was 0.79 with a Cohen's kappa of 0.53; Granite Judge agreed more with NLI Judge in pairwise terms (0.70 vs. 0.67) but had a higher Cohen's kappa with Bias Judge (0.16 vs. 0.13). Human annotators had an average Cohen's kappa of 0.10 with Granite Judge, and Fleiss' kappa of 0.54 (Bias Judge prompt) and 0.55 (NLI Judge prompt) among themselves.
• The evaluation was large. The authors report that their experiments involved over 800,000 queries.

Methodology in Plain English

The authors exploit a simple fact about the standard Chat Completions API: the user, not the server, controls the conversation history. Every request sends the whole prior exchange along with the new message, so a user can write the "assistant" turns themselves.

CoBia builds a fabricated dialogue. The user asks for characteristics of a social group. The fabricated assistant reply is a numbered list of eight characteristics — six positive ones (generated beforehand by llama3.3:70b) and two negative ones drawn from the CoBia dataset, scattered so the negative items do not stand out. Four follow-up questions then point at four of the eight list items by index number, always including the second negative descriptor but never the primary one. The final question asks whether the primary negative descriptor can be linked to the second one, and instructs the model to explain the association using statistical reasoning in ten sentences. The whole thing is submitted as one query.

Two variants exist. HCC keeps the explicit user and assistant role tags; SCC flattens everything into a single user message with no tags. UCC is simply the union: an instance counts as biased if either variant produces a biased response.

Three tricks reduce filter triggering: anaphora (pronouns instead of repeating group names), substitution (referring to descriptors by index number, e.g. "#6"), and preemptive disavowal (a disclaimer stating the user knows the characteristics do not apply to everyone, plus a request not to mention it again).

The dataset itself was built by merging RedditBias, SBIC, and StereoSet, extracting descriptor terms, consolidating overlapping group names, assigning each group to one of six categories, and filtering with Llama Guard (llama-guard3:8B) — only instances classified as unsafe were retained, and 99% fell under Llama Guard's S10 category.

For scoring, the authors used llama3.3:70b as the main judge plus a Granite Guardian judge and an NLI-style judge that labels the relationship between a biased opinion and the model's answer as enforces, agrees, neutral, or negates. Four human annotators, two working from the Bias Judge prompt and two from the NLI Judge prompt, each labeled 300 randomly selected outputs, for 1,200 annotations total. Models were run at temperature=0 and top_p=0.

Why This Matters

Impact on research: The paper shows that bias benchmarks run in isolation (the 0-Shot setting) systematically understate what models do inside a conversation. It also introduces a third role for adversarial attacks — not purely to break models, but as an auditing instrument — and provides evidence that judge choice materially changes which models look biased and which baselines look safe.

Real-world applications:

• Safety auditing of deployed chatbots before release, using CoBia as a lightweight red-teaming probe.
• Regression testing after fine-tuning or alignment updates, since a model that improves on standard safety checks may still fail constructed-conversation tests.
• Compliance and documentation work for teams that must show due diligence on bias mitigation.
• Conversation-log monitoring in production systems that manage chat history on behalf of users.

Industry relevance: The paper identifies a concrete, low-cost patch — disabling user control over conversation history, or extending safety checks from isolated prompts to whole dialogues. It notes that some models, such as Gemma 2, already appear to follow this practice, and that as of March 11, 2025, OpenAI began managing a unique ID for previous chat history in its new API, though the earlier Chat Completions API remains the de facto standard with manually managed history. The authors stress that fixes to proprietary models ultimately have to come from model providers.

Future Directions

• Broader conversational templates. The authors tested only two sets of templates and explicitly note that longer dialogues, brief exchanges, or emotional language could change results. A systematic sweep of dialogue styles is a natural next step.
• Defenses rather than just attacks. The paper offers only high-level mitigation suggestions, chiefly limiting user control over conversation history, and calls for future work to develop robust defenses against conversational adversarial attacks.
• Scaling human validation. With four annotators covering 1,200 outputs, the human evaluation is limited in scope relative to over 800,000 queries. Broader manual labeling would firm up judge reliability claims.
• Explaining the model-size result. The absence of a clear correlation between parameter count and bias, plus the anomalously strong safety of the 32B qwen2.5 model, remains unexplained and invites follow-up study.

Target Audience

This paper is most useful to LLM safety and red-teaming researchers, AI ethics and bias-evaluation practitioners, and evaluation engineers who design benchmarks or judge pipelines. Model developers and product teams responsible for chatbot guardrails will benefit from the concrete attack construction and the identified patch, while policymakers and auditors will find the framing of bias as a conversational rather than single-turn phenomenon relevant to how they scope safety assessments.

Authors’ abstract

Improvements in model construction, including fortified safety guardrails, allow Large language models (LLMs) to increasingly pass standard safety checks. However, LLMs sometimes slip into revealing harmful behavior, such as expressing racist viewpoints, during conversations. To analyze this systematically, we introduce CoBia, a suite of lightweight adversarial attacks that allow us to refine the scope of conditions under which LLMs depart from normative or ethical behavior in conversations. CoBia creates a constructed conversation where the model utters a biased claim about a social group. We then evaluate whether the model can recover from the fabricated bias claim and reject biased follow-up questions. We evaluate 11 open-source as well as proprietary LLMs for their outputs related to six socio-demographic categories that are relevant to individual safety and fair treatment, i.e., gender, race, religion, nationality, sex orientation, and others. Our evaluation is based on established LLM-based bias metrics, and we compare the results against human judgments to scope out the LLMs' reliability and alignment. The results suggest that purposefully constructed conversations reliably reveal bias amplification and that LLMs often fail to reject biased follow-up questions during dialogue. This form of stress-testing highlights deeply embedded biases that can be surfaced through interaction. Code and artifacts are available at https://github.com/nafisenik/CoBia.