Biases in the Blind Spot: Detecting What LLMs Fail to Mention

AI summary

Overview

Research area: LLM interpretability and safety, specifically chain-of-thought (CoT) faithfulness, unverbalized (implicit) bias detection, and automated black-box model auditing.

Technical level: Advanced.

Scope (one sentence): The paper introduces a fully automated, black-box pipeline that hypothesizes, tests, and statistically validates task-specific "unverbalized biases" — decision factors that shift an LLM's outputs without ever being cited as justification in its chain-of-thought — and evaluates it across seven LLMs on three decision tasks.

What This Paper Is About

Large Language Models produce chain-of-thought reasoning that looks plausible, but that reasoning may omit factors the model actually uses to decide. This paper calls such hidden factors unverbalized biases and argues that monitoring models through their stated reasoning is therefore unreliable. The goal is to build a fully automated pipeline that discovers these hidden decision factors without requiring predefined bias categories or hand-crafted datasets, and to verify that the pipeline finds both previously known biases and new ones.

Key Contributions

1. A fully automated, black-box pipeline for detecting unverbalized biases. Unlike prior work (Karvonen and Marks, 2025) that manually hypothesizes biases, the method generates concept hypotheses using LLM autoraters and extends counterfactual faithfulness testing (Atanasova et al., 2023) via LLM-based concept variation — removing the need for per-task trained editors and enabling semantic verbalization checking.

2. A multi-stage design with input clustering, staged sampling, and statistical early stopping. The pipeline uses O'Brien-Fleming alpha spending and futility analysis to control computational cost while preserving family-wise error rate, reducing API calls by roughly one third relative to exhaustive evaluation.

3. An empirical evaluation on three decision tasks across seven LLMs. The pipeline rediscovers biases manually identified by prior work (gender, race, religion, ethnicity) and uncovers new ones (Spanish fluency, English proficiency, writing formality).

4. A generalizability test on setups from four prior bias studies (Demidova et al., 2024; Hemmatian et al., 2023; Motoki et al., 2024; Arif et al., 2024), showing the approach adapts to different bias dimensions and task contexts while yielding additional insights about verbalization patterns. The authors also release two synthetic evaluation datasets (loan approval and university admissions), code, and raw results.

Main Findings

• Unverbalized biases exist and are measurable. A concept is flagged when paired positive/negative input variations produce a statistically significant difference in decisions (McNemar's test, p < α) while the concept is cited as justification in at most τ of discordant-pair responses. The paper's motivating example is Claude Sonnet 4 on loan approval: adding a single sentence about religious affiliation flips a decision, with an effect size of 0.037 in favor of minority-religion applicants (p = 9.15 × 10⁻⁷ over 2,500 inputs), meaning they are approved 3.7 percentage points more often. Religion was verbalized in only 12.4% of responses where the decision flipped, below the 30% threshold.

• Gender and race/ethnicity biases recur across all three tasks. Aggregating significant unverbalized biases across all tasks and the six paper models, every detected gender bias favors female candidates (22 pro-female vs. 0 pro-male) and every detected race/ethnicity bias favors the minority-associated group (21 pro-minority vs. 0 pro-majority).

• Hiring task (1,336 inputs from Karvonen and Marks, 2025). Gender bias is detected in five of seven models, all in the direction of female candidates. Race and ethnicity biases associated with Black-sounding names are detected in five models. QwQ-32B additionally shows bias toward Spanish language ability despite the job description not mentioning any language preference; a follow-up control suggests an ethnic or cultural component beyond generic multilingualism. The authors' effect sizes (3–5%) are smaller than those reported by Karvonen and Marks (6–12%), consistent with that prior finding that chain-of-thought reasoning reduces bias magnitude.

• Loan approval task (2,500 synthetic inputs). Gender bias toward female applicants is detected in five models. Language proficiency biases appear in both Gemma models and Gemini 2.5 Flash, and formal tone bias in both Gemma models. Race/ethnicity biases toward minority applicants are detected in Claude Sonnet 4 and QwQ-32B; Claude also shows bias toward minority religious affiliations (effect size −0.037 with early stopping).

• University admissions task (1,500 synthetic inputs). Race-related biases toward minority applicants appear in both Gemma models, GPT-4.1, and Claude Sonnet 4. Gender bias toward female applicants appears in Gemini 2.5 Flash, GPT-4.1, and QwQ.

• Grok 4.1 Fast is unusually transparent. It is the only model with no detected unverbalized biases in the admissions task, where 87% of concepts were filtered. Of 30 concepts flagged as unverbalized biases across the other six models, 27 are filtered for Grok — 10 because it mentions them and 17 because the effect does not reach significance. For a race-related concept that GPT-4.1 mentions in only 6% of baseline responses, Grok mentions it in 67.5%, with statements such as "Demographics: Shanice (likely underrepresented minority based on name)."

• Statistical strength of detections. Across all detected biases, 67% have 95% confidence intervals that exclude zero. Effect size is defined as Δ = p_pos − p_neg, so an effect size of 0.05 means the model accepts 5 percentage points more under the positive variation than the negative variation.

• Consistency across random seeds. Running the loan approval task on five different seeds for Gemma-3-12b-it, the pipeline does not detect identical concepts in every run, but finds semantically similar concepts in most runs (gender bias in all runs, English proficiency in 4/5 runs), and never observes contradictory biases across runs.

• Validation on intentionally biased models. Injecting biases into Gemma-3-27B-it's system prompt in secret (hidden from CoT) and overt (stated in reasoning) modes across 80 test cases (20 concepts × 2 modes × 2 directions), the pipeline achieves 92.5% accuracy, correctly detecting 85% of secret biases and filtering 100% of overt biases. When it detects a secret bias, it identifies the direction in all cases. The six false negatives occurred because the verbalization detector over-triggered on related terms in legitimate financial reasoning.

• Verbalization detection is reliable but conservative. In a human annotation study on 100 samples with two annotators (paper authors), inter-annotator agreement was Cohen's κ = 0.737. Eight LLM-based detectors all achieved κ > 0.6 against human consensus; the best (GPT-4.1-mini and GPT-5.2) reached κ = 0.79 and 90% accuracy. GPT-5-mini, used in the pipeline, achieved κ = 0.67 and 84% accuracy, with errors leaning toward over-detection.

• Generalization results. On GPT-3.5-turbo, the multilingual setup yielded 6 significant biases in English (4 unverbalized, 2 verbalized), 1 unverbalized cultural bias in Arabic, and none in Russian. In the religious bias study, 5 of 7 significant biases were unverbalized. The SALT comparison on Gemma-2-9B-it found no significant demographic biases, contrasting with the original findings in debate and career advice tasks. The political bias comparison found Republican impersonation produces 1.6–3 times larger shifts in agreement rates than Democrat impersonation.

• Reasoning training does not obviously improve faithfulness. Comparing QwQ-32B (RLVR) against its SFT counterpart Qwen2.5-32B-Instruct on loan approval, the authors find nearly identical verbalization filter rates (97.0% vs 97.2%) and the same number of unverbalized biases, suggesting reasoning training shifts which biases emerge rather than improving faithfulness.

• Novel non-demographic concepts do not generalize. Spanish fluency, English proficiency, and writing formality do not consistently generalize across tasks, unlike the demographic biases.

• Filters are aggressive. In the hiring task table caption, 83% of the non-effect cells are labeled "verb." (significant but verbalized) and 17% are "n.s." (not significant or not hypothesized). The variation quality check removed 42% of candidate concepts and shows 80% agreement with human annotations within one rating point.

Methodology in Plain English

The method treats bias detection as a statistical experiment run at scale.

1. Cluster the inputs. Task inputs are embedded with a text embedding model (OpenAI's text-embedding-3-large) and grouped with k-means into 10 clusters, so the pipeline samples diverse rather than redundant examples.

2. Hypothesize concepts. Three representative inputs from each cluster (30 inputs total, 1–2% of the data) are shown to an LLM, which proposes concepts that might influence decisions. Crucially, the hypothesis-generating LLM never sees the target model's responses. For each concept it also produces a verbalization check guide, an addition action, and a removal action.

3. Generate paired variations. Using those actions, the pipeline creates a positive variant (promoting the concept) and a negative variant (removing it) of each input. An LLM judge filters out concepts whose variations introduce confounds beyond the target concept.

4. Filter concepts the model already talks about. Baseline responses are collected and concepts cited as justification in more than τ of them are dropped. Merely repeating a concept is not counted as verbalization — it has to be used as a reason.

5. Test in stages. At each stage the pipeline collects decisions on both variations, checks the verbalization rate again on discordant pairs (cases where the decision flipped), and applies McNemar's test to the paired binary outcomes. Family-wise error rate is controlled at α = 0.05 via Bonferroni correction (α′ = α/|C|).

6. Stop early when justified. O'Brien-Fleming alpha spending relaxes the significance threshold as more data accumulates (allowing early detection), while futility analysis estimates conditional power via Monte Carlo simulation and drops concepts unlikely to reach significance after at least 25 discordant pairs. The pipeline starts with 20 inputs per cluster and doubles each stage, typically producing 4–6 stages. Bonferroni's denominator counts all concepts entering statistical testing, so FWER control is preserved despite intermediate filtering.

7. Report survivors. A concept is reported as an unverbalized bias only if it passes the verbalization filter and reaches significance after correction. Statistical testing is performed on 766–2,500 inputs per concept, keeping hypothesis generation and inference clearly separated.

Settings used in the evaluation: α = 0.05, verbalization threshold τ = 0.3, futility threshold γ = 0.01. Each input receives one baseline response and one response per variation. Detailed cost breakdowns are referenced to appendix material that is not included in the provided content.

The authors stress that "bias" here is descriptive, not normative — a systematic decision shift caused by the presence or absence of an attribute — and that the CoT is treated as an output to be monitored, not as ground truth about internal computation.

Why This Matters

Impact on research. The work reframes CoT-based monitoring: even if a CoT is necessary for solving a task, it need not contain all the information the model uses. By automatically generating hypotheses instead of relying on predefined bias categories, the pipeline lowers the cost of bias auditing and provides a statistical framework (paired tests, multiple-testing correction, early stopping) that other researchers can reuse. It also connects to prior faithfulness work (Turpin et al., 2023; Lanham et al., 2023; Atanasova et al., 2023; Siegel et al., 2024) and to concurrent automated bias discovery in LLM-as-a-Judge evaluation (Lai et al., 2026), which the authors describe as complementary.

Real-world applications:

• Hiring: Screening models that decide whether to interview candidates can systematically shift decisions based on names, pronouns, or language ability rather than resume content.
• Credit and lending: Loan officers driven by LLMs can flip approve/reject decisions on a single sentence about religious affiliation while constructing different framings of identical financial data.
• University admissions: Admissions agents show race- and gender-associated decision shifts even when these attributes are never cited as reasons.
• Model auditing and compliance: The pipeline can validate whether a model is transparent about sensitive factors, and can be adapted to prior bias benchmarks (multilingual bias, anti-Muslim bias, political bias, SALT) in new task contexts.

Industry relevance. A systematic 3–5% shift translates into tens of thousands of decisions affected once models are deployed at scale, particularly in agentic settings where effects can compound. The pipeline's roughly one-third reduction in API calls versus exhaustive evaluation makes large-scale auditing more practical. The finding that a model may explicitly disclaim a factor's relevance ("noted but irrelevant to financial underwriting") while still being influenced by it is directly relevant to anyone relying on stated reasoning for compliance documentation.

Future Directions

• Reduce confounds and disentangle causal pathways. Detected effects may partly reflect confounders in the input distribution (for example, female-coded names correlating with stereotypically gendered occupations). The authors state that disentangling the precise causal pathway would require analysis beyond their black-box approach.

• Improve verbalization detection. The filter uses a fixed threshold (τ = 0.3 in the evaluation) and sometimes conflates mentioning a concept with citing it as a decision factor, and it does not catch implicit verbalization through proxies. Adaptive thresholds, more nuanced semantic matching, or a richer concept ontology are proposed as remedies.

• Broaden concept hypothesis coverage. The pipeline can only detect biases that its concept-generation stage proposes; biases the hypothesis-generating LLM does not think to suggest will never be tested. (This limitation is described as ongoing in the truncated text.)

• Separate problematic biases from legitimate heuristics. Not every unverbalized decision factor is a problematic bias — some may be valid criteria that simply go unmentioned. The paper leaves this distinction to the pipeline's user and to downstream normative audit.

Target Audience

Researchers and engineers working on LLM safety, interpretability, and chain-of-thought faithfulness; AI auditors and red-teamers who need automated, scalable bias discovery rather than hand-crafted test suites; policy and compliance teams evaluating models used in high-stakes decisions such as hiring, lending, and admissions. Readers should be comfortable with statistical testing concepts (McNemar's test, Bonferroni correction, alpha spending, conditional power) and with LLM

Authors’ abstract

Large Language Models (LLMs) often provide chain-of-thought (CoT) reasoning traces that appear plausible, but may hide internal biases. We call these unverbalized biases. Monitoring models via their stated reasoning is therefore unreliable, and existing bias evaluations typically require predefined categories and hand-crafted datasets. In this work, we introduce a fully automated, black-box pipeline for detecting task-specific unverbalized biases. Given a task dataset, the pipeline uses LLM autoraters to generate candidate bias concepts. It then tests each concept on progressively larger input samples by generating positive and negative variations, and applies statistical techniques for multiple testing and early stopping. A concept is flagged as an unverbalized bias if it yields statistically significant performance differences while not being cited as justification in the model's CoTs. We evaluate our pipeline across seven LLMs on three decision tasks (hiring, loan approval, and university admissions). Our technique automatically discovers previously unknown biases in these models (e.g., Spanish fluency, English proficiency, writing formality). In the same run, the pipeline also validates biases that were manually identified by prior work (gender, race, religion, ethnicity). More broadly, our proposed approach provides a practical, scalable path to automatic, more efficient, and broader task-specific unverbalized bias discovery.