The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits

AI summary

Overview

Research area: Natural Language Processing / algorithmic fairness evaluation — specifically the methodology of bias audits of large language models used as decision-making instruments.

Technical level: Intermediate to Advanced. The experiments are conceptually simple (ask five models to rate, decide, and rank job/loan/triage applications), but the paper's argument rests on pre-registration, Benjamini-Hochberg correction across 36 contrasts, equivalence testing (TOST), Latin-square randomization, and mixed-effects and rank-ordered logit models.

Scope: A large pre-registered, multi-domain audit (40,726 requests, five models, three high-stakes domains) testing whether the published rating-versus-ranking "reversal" in LLM demographic bias generalizes beyond charitable aid — and finding that it does not, while the audit's own construction produces effects as large as any demographic one.

What This Paper Is About

A prior study on charitable-aid allocation reported that the same language models favored minority applicants when rating requests one at a time but penalized some of them when ranking candidates side by side. If that format-driven reversal were general, much of the disagreement in the LLM fairness literature would be an artifact of how audits are designed. This paper tests that claim in hiring, lending, and medical triage — domains regulators actually scrutinize — and concludes that the reversal does not generalize, with audit construction (candidate position, instruction wording, tie-permitting format) driving verdicts at least as much as demographics.

Key Contributions

1. A pre-registered cross-domain replication attempt. The researchers fixed their primary hypothesis and 36 planned contrasts before collecting data, then tested whether the hiring Black–White contrast reverses between rating and ranking formats across hiring, lending, and triage — the first such test outside charitable aid.

2. Equivalence bounds rather than bare nulls. Instead of only reporting "no significant effect," the paper declares equivalence margins from the published study (0.090 rating points, 0.067 rank positions) and shows the data positively exclude a hiring ranking penalty of the published size, while explicitly noting that lending and triage floors sit above that margin and are therefore inconclusive.

3. Evidence that audit instrument effects rival demographic effects. Position on the list (first-listed candidates gain 0.11 rank positions, 0.094 SD) is comparable to or larger than the largest demographic effect measured (0.049 SD), and an apparent instruction-wording reversal turned out to be a composition confound inside the authors' own robustness arm.

4. A demonstration that transparency and ties are entangled. Models tie 100% of bundles with identical content whether the varying detail is race or a hobby, and they call both "fairness tests" at nearly identical rates — showing that recognition rates and tie rates measure the probe design, not model preferences.

Main Findings

• The reversal does not appear. The pre-registered format-by-group interaction for hiring is +0.01 [-0.15, +0.17], and none of the 36 planned contrasts survives Benjamini-Hochberg correction in any domain, group, or format. None of twelve format-by-group interactions is distinguishable from zero.

• The rating advantage persists but shrinks. On Rate, the hiring Black contrast keeps its sign at roughly half the published size: +0.047 points [+0.002, +0.092], or +0.04 SD. It is reported as below the correction threshold rather than as zero, and an equivalence test bounds it at 0.084 points against the 0.090 margin (p = .029).

• The hiring ranking penalty is excluded, but only for hiring. Adding 12,000 disguised hiring rank slots narrows the contrast to +0.008 positions [-0.044, +0.060], bounding any penalty at 0.052 positions against the 0.067 pre-declared margin (p = .013). Lending and triage floors are 0.132 and 0.232 positions — above the margin — so a penalty there is neither detected nor excluded.

• The nulls are bounded, not blind. The instrument resolves effects down to 0.22 SD at worst. Planted disparities track their injected sizes (0.48 recovered against 0.50 planted), though recoveries at 0.05 and 0.10 do not individually exclude zero. The original aid stimuli, rerun on the same five models, reproduce the direction of both published effects (-0.064 vs. -0.067 rank; +0.193 vs. +0.090 rate) but neither interval clears zero.

• Models recognize transparent audits almost always. 100% under forced choice, 89% [84, 93] when asked in their own words. Recognition of camouflaged audits is far lower (2% [1, 4] volunteered) but the forced-choice probe also labels 74% of placebo bundles — which contain no demographic contrast — as fairness tests, so that probe measures the offered options rather than recognition.

• Ties are not about demographics. Models tie 100% of demographic-matched bundles and 100% of bundles differing only in a hobby or neighborhood, while tying 0% of disguised and 0% of placebo bundles. Identical content and a recognizable audit always arrive together, making the transparent rank cell uninformative by construction.

• Position is the largest single effect. First-listed candidates gain 0.11 positions [0.03, 0.18], or 0.094 SD, versus 0.049 SD for the biggest demographic contrast. The difference between the two covers zero, so position is comparable to or larger — not reliably larger.

• A wording "reversal" was a composition artifact. Two paraphrases initially appeared to reverse the contrast, but that arm assigned each application a fixed set of name pairs, tying race to application quality (Black-named applications rated 2.96 at baseline, White-named 3.39). Once race and application were decoupled, the effect collapsed to -0.01 and -0.02.

• Function-call formats change nothing. Native submit_decision and submit_ranking tool calls mirror their text counterparts almost exactly (+1.5pp and +0.01 positions).

Methodology in Plain English

The researchers took 12 publicly released, demographically neutral profiles per domain — résumés, credit files, and emergency presentations — and rendered each under 40 first names validated to signal race and gender, fully crossed so every profile appears under every name. Each profile is then judged three ways: a 1–5 priority rating, a yes/no decision, and a four-candidate ranking.

The ranking task is where the design gets clever. Bundles come in four varieties: a demographic matched cell (one profile under four names, one per race — the classic transparent audit), an irrelevant attribute matched cell (one profile under four same-race names differing only in a hobby), a disguised demographic cell (four profiles of varying quality under balanced Latin squares, so each race appears in each quality tier and each position equally often, and no single prompt reads as an audit), and a disguised placebo cell that estimates the design's false-positive rate. Ties are permitted everywhere, which is a deliberate departure from prior side-by-side audits.

Five models — two proprietary, three open-weight — produce 40,726 requests. Contrasts come from mixed-effects regressions with random intercepts for profile, name pair, and bundle, with ranking contrasts re-estimated using a rank-ordered logit and bundle-clustered errors. All 36 pre-registered contrasts are corrected together. Before interpreting any null, the authors verify that planted disparities are recovered, report the minimum effect each outcome could detect, and run leave-one-name-out and leave-one-profile-out checks. Five extensions — including a 12,000-slot precision top-up and a reasoning-enabled rerun — were run after the primary analysis, are reported separately, and are explicitly tagged so they never enter the pre-registered family.

Why This Matters

Impact on research. The paper argues that a bias audit certifies the specific interface and wording it tests, not the model. A single-format verdict says as much about the configuration as about the system, and a node cleared under one format carries no certificate for the formats a deployed workflow will call it under. The authors also report a confound — race tied to application quality — inside their own robustness arm, exactly the error correspondence audits exist to prevent, which is a candid methodological warning for the field.

Real-world applications:

• Hiring and recruiting agents that screen résumés: the finding that list position moves outcomes as much as demographics suggests workflow design (ordering, batching) needs auditing alongside model behavior.
• Credit and lending decision nodes: the paper is explicit that its hiring exclusion does not transfer to lending, where the detection floor sits above the published effect size — a direct caution against treating a null audit as a fairness certificate.
• Medical triage and patient routing: same caveat, and the triage floor is the loosest of the three (0.232 positions).
• Compliance and model-evaluation tooling: the results argue for audits that include multiple formats, a planted control, and composition checks on every subsample.

Industry relevance. Deployed agents call decision nodes repeatedly and pass outputs downstream without human review. The paper shows that serializing a decision as a function call changes nothing, that disabling reasoning (the deployed default for most systems) points the same way as enabling it, and that models overwhelmingly recognize standard audit framings — meaning benchmark-style evaluation may systematically overstate how fairly a system behaves in less legible deployment contexts.

Future Directions

• Isolate what carries the original aid phenomenon. The directional replication localizes the effect to the FairFund materials but does not separate whether it comes from the domain's semantics or the way its stimuli are built. The authors offer this as a conjecture, not a finding.

• Test demographic signals beyond names. Names are the only cue used here and are a thin one; the authors note conclusions may not transfer to race and gender conveyed by dialect, explicit statements, or photographs, and that names carry socioeconomic connotations the design cannot separate.

• Untangle recognition from tie behavior. With transparent recognition and tie rates both at their maximum, the link between them cannot be tested in this design. A construction that separates the two would clarify what the models are actually doing.

• Resolve the position-versus-demographics ordering. The confidence interval for the difference between the largest demographic effect and the first-position effect covers zero, so which is larger remains open. A design that estimates both with equal precision — every bundle currently contributes all four positions while a demographic contrast depends on where names landed — would settle it.

• Extend the equivalence framework. The precision extension was commissioned after the frozen estimate was seen, making it a disclosed post-hoc test rather than a fixed-sample one; a fully pre-registered replication with the same margins is the natural next step.

Target Audience

Researchers and practitioners in AI fairness evaluation, algorithmic auditing, and LLM agent reliability — particularly those who design bias benchmarks or interpret their results. It is also directly useful for ML engineers building hiring, lending, or triage agents who need to understand why a passing audit does not certify a deployed system, and for policy and compliance staff who need to read audit findings with appropriate skepticism. Readers should be comfortable with confidence intervals, multiple-comparison correction, and the logic of equivalence testing; the paper's core argument is accessible without following every regression specification, but its persuasiveness depends on the statistical machinery.

Authors’ abstract

Whether a language model looks demographically biased can depend on how the audit asks its question. A charitable-aid benchmark reports that the same models favor minority applicants when rating requests one at a time and penalize some when ranking side by side. We test whether that reversal generalizes to hiring, lending, and medical triage: 40,726 requests to five models, applications differing only in the applicant's name, and a primary test fixed before collection. It does not. None of 36 planned contrasts survives correction. The rating advantage keeps its sign at roughly half the published size, and a precision extension bounds any hiring ranking penalty below the published effect, though the lending and triage ranking floors sit above that margin, so the exclusion is conclusive for hiring ranking and for rating in all three domains only. Planted disparities tracking their injected sizes and a directional replication on the original aid materials bound these nulls. The audit is livelier than the demographics: models recognize transparent audits nearly always, tie every identical-content comparison whether the varying detail is race or a hobby, and reward first-listed candidates as much as any demographic effect we measure. Audit verdicts reflect audit construction more than demographic bias.