When to Invoke: Refining LLM Fairness with Toxicity Assessment

AI summary

Overview

Research area: Natural Language Processing / fairness in LLM-based content moderation, with a focus on implicit hate speech.

Technical level: Intermediate. The paper assumes familiarity with transformer classifiers, prompt engineering, and fairness metrics, but its core ideas are explained conceptually.

Scope: The paper proposes FairToT, an inference-time prompting framework that decides when to correct an LLM's toxicity judgements so that demographic substitutions in otherwise identical sentences receive consistent scores.

What This Paper Is About

LLMs used for online moderation can assign noticeably different toxicity scores to sentences that differ only in the demographic group they mention, even when the underlying meaning is unchanged — a problem that is especially acute for implicit hate speech. Fine-tuning, balanced data collection, and post-hoc calibration are often impractical for deployed models, so the authors ask a question existing work overlooks: when during inference should a corrective mechanism actually be invoked? FairToT answers this by detecting likely demographic-driven inconsistency and selectively triggering a prompt-based refinement, without touching model parameters.

Key Contributions

1. Framing the "when to invoke" question. The authors state this is the first work to examine when fairness correction should be invoked during LLM inference, particularly for implicit hate speech where demographic disparities are easily amplified.
2. The FairToT framework. An inference-time framework that refines LLM fairness through prompt-guided toxicity assessment, designed to be practical for moderation systems built on large pre-trained models where fine-tuning is unavailable.
3. Two interpretable fairness indicators. Sentence Fairness Variance (SFV) and Entity Fairness Dispersion (EFD), which quantify systematic disparities and reveal entity-conditioned bias in toxicity assessment, alongside internal detection signals θ̂^S_n (sentence-level) and θ^E (entity-level).
4. Experiments on three benchmarks. Results across Latent Hatred, Offensive Slang, and ToxiGen show FairToT reduces demographic disparities and produces more consistent toxicity assessments.

Main Findings

• Broad fairness gains across models. FairToT reduced both SFV and EFD across all baselines evaluated, on both GPT-3.5-Turbo and Llama-3.1-8B-Instruct backbones, independent of architecture or augmentation strategy. Example from the GPT-3.5-Turbo BERT (AAV) row: on Latent Hatred, SFV fell from 0.113096 ± 0.079823 to 0.000114 ± 0.000027 and EFD from 0.142037 ± 0.081033 to 0.025200 ± 0.000073.
• Distribution contraction. Before mitigation, SFV and EFD distributions were broad and high-variance; after FairToT they contracted sharply with markedly lower mean and variance, indicating reduced entity-conditioned bias at both sentence and entity levels.
• A concrete worked example. For entities Black, White, and Immigrant, the original model output [0.88, 0.69, 0.82]; after the three-stage mitigation the outputs became [0.72, 0.71, 0.73], reducing variance by more than 90% with the input text unchanged.
• Every component matters. The ablation shows the full FairToT configuration achieves the lowest SFV and EFD. Removing either θ̂^S_n or θ^E substantially degrades performance, and omitting In-Context Learning (ICL) or Three-Step Prompting (3SP) weakens mitigation quality.
• Efficiency trade-off is favourable. Removing ICL or 3SP slightly lowers token usage but noticeably weakens fairness; enabling both yields the lowest fairness scores for only a modest increase in token consumption.
• Low temperature is best. At temperature 0, SFV and EFD reach their lowest values across most datasets. Raising temperature to 0.5 and 1.0 increases both metrics in most conditions, with GPT-3.5-Turbo declining gradually and Llama showing stronger dataset-dependent fluctuations on Offensive Slang and ToxiGen.
• Thresholds were empirically chosen. The clip value C_θ = 0.25 and trigger threshold R_n ≥ 0.35 were selected from threshold sweeps over C_θ ∈ {0.15, 0.20, 0.25, 0.30, 0.35} and R_n ∈ {0.25, 0.30, 0.35, 0.40, 0.45}.
• No accuracy metrics are reported. The study focuses exclusively on fairness within implicit hate speech, explicitly stating that accuracy and AUROC are not used.

Methodology in Plain English

The framework has three parts.

Detecting bias. For each sentence, the authors build a template with an entity placeholder (<ENT>), then substitute a set of demographic entities and ask the model for a toxicity probability each time. They compare each entity's probability against the neutral template's baseline probability to get an "entity sensitivity." Sentence-level bias (θ^S_n) is the variance of these sensitivities within one sentence; entity-level bias (θ^E_k) is the variance of a single entity's sensitivities across many sentences. A clipped and normalised sentence score is combined with the aggregated entity score into a single fairness risk: R_n = λ·θ̂^S_n + (1−λ)·θ^E with λ = 0.5. Correction is triggered only when R_n ≥ 0.35.

Correcting bias. When triggered, a three-stage prompt runs: (1) a semantic equivalence check confirming the variants differ only in the demographic token, (2) entity-neutral harm inference judging the message's offensiveness while ignoring which entity is named, and (3) probability assignment aimed at keeping the spread across entities at ≤ 0.02, using tiny deterministic offsets of [−0.01, 0.00, +0.01] by input order to avoid degenerate identical values after rounding.

Measuring fairness. SFV averages the per-sentence variances and EFD averages the per-entity variances, each reported as mean ± standard deviation over the evaluation set.

Experimental setup. Three datasets were used: Latent Hatred (22,056 tweets, including 6,346 implicit hate instances), Offensive Slang (user-generated tweets annotated as hate speech, offensive language, or neither), and ToxiGen (approximately 274,000 toxic and benign statements spanning 13 minority identity groups). Baselines included BERT, HateBERT variants H1-BERT (fine-tuned on ISHate) and H2-BERT (fine-tuned on ToxiGen), DeBERTa (fine-tuned for four epochs on ISHate), and ReBERTa (fine-tuned on ToxiGen). Data augmentation used Add-Adverbs-to-Verbs and Back Translation (English to Russian and back). H2-BERT and RoBERTa used no augmentation. Runs used Kaggle Notebooks with an NVIDIA Tesla P100 GPU (16 GB VRAM), Hugging Face Transformers with a PyTorch backend, Python 3.10 on Ubuntu 20.04, and accessed GPT-3.5-Turbo and Llama-3.1-8B-Instruct through the Microsoft Serverless API.

Why This Matters

The work targets a practical gap in moderation pipelines: fairness problems appear dynamically at inference, yet most fixes require retraining or post-hoc calibration that cannot respond to individual inputs.

Real-world applications:

• Social media and platform moderation, where inconsistent decisions across demographic groups erode user trust and can reinforce stereotypes.
• Deployed third-party LLM APIs, where teams cannot fine-tune the model but can change how they prompt it.
• Community forums and comment systems handling coded or indirect hostility that surface-level keyword filters miss.
• Fairness auditing and compliance review, using SFV and EFD as model-agnostic, interpretable measures of entity-conditioned disparity.

Industry relevance: the framework requires no parameter updates, works on both open-source and proprietary backbones, and offers a tunable trigger so that extra computation is spent only on inputs where bias is likely — relevant for organisations weighing moderation quality against inference cost.

Future Directions

• Broadening demographic coverage. The experiments focus on identity groups in implicit hate speech datasets; extending to other demographic axes, languages, and cultures is an open question.
• Calibrating the trigger. The thresholds C_θ = 0.25 and R_n ≥ 0.35 were fitted empirically and the authors note they may need adjustment for different datasets or application-specific fairness requirements.
• Reducing token overhead. ICL and 3SP both add inference cost; finding cheaper ways to obtain the same fairness gains is a natural next step.
• Understanding the fairness–utility relationship. Because the study reports only fairness metrics and not accuracy or AUROC, how the refinement affects detection performance remains unexamined.

Target Audience

NLP fairness and bias-mitigation researchers; content moderation engineers and platform trust-and-safety teams; practitioners deploying LLM APIs who cannot fine-tune models; and applied researchers interested in inference-time, prompt-based alternatives to retraining for fairness.

Authors’ abstract

Large Language Models (LLMs) are increasingly used for toxicity assessment in online moderation systems, where fairness across demographic groups is essential for equitable treatment. However, LLMs often produce inconsistent toxicity judgements for subtle expressions, particularly those involving implicit hate speech, revealing underlying biases that are difficult to correct through standard training. This raises a key question that existing approaches often overlook: when should corrective mechanisms be invoked to ensure fair and reliable assessments? To address this, we propose FairToT, an inference-time framework that enhances LLM fairness through prompt-guided toxicity assessment. FairToT identifies cases where demographic-related variation is likely to occur and determines when additional assessment should be applied. In addition, we introduce two interpretable fairness indicators that detect such cases and improve inference consistency without modifying model parameters. Experiments on benchmark datasets show that FairToT reduces group-level disparities while maintaining stable and reliable toxicity predictions, demonstrating that inference-time refinement offers an effective and practical approach for fairness improvement in LLM-based toxicity assessment systems. The source code can be found at https://aisuko.github.io/fair-tot/.