GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Models

AI summary

Overview

Research area: AI Safety & Ethics — inference-time bias mitigation for generative vision–language models (VLMs).

Technical level: Intermediate overall, with Advanced mathematical machinery in the method section. The core idea (steer visual tokens along a sphere instead of subtracting a bias vector) is intuitive; the formalization relies on Riemannian/spherical geometry, Fréchet means, tangent-space log/exp maps, and spherical linear interpolation (Slerp).

Scope: The paper proposes GGSS, a training-free, norm-preserving, token-adaptive geodesic steering rule for debiasing frozen generative VLMs at inference time, and benchmarks it against ten adapted debiasing baselines plus a prompt-based mitigation family across four open generative VLMs, three bias protocols, and a general-capability benchmark (MMStar).

What This Paper Is About

Generative VLMs can produce demographically biased outputs even when the only thing that changes between two images is a controlled attribute such as perceived race or gender. Most existing inference-time debiasers (INLP, R-LACE, LEACE, BendVLM-style retrieval equalization) were built for CLIP-like models, where an image is one global embedding — not for generative VLMs, where an image becomes many visual tokens fused into a decoder-only language model. The paper's goal is a debiasing hook that reduces demographic bias while leaving the frozen model's general visual–language ability essentially intact, without retraining or extra GPU-heavy data.

Key Contributions

1. Diagnosis and a geometric alternative. The authors identify three reasons hard, token-uniform, single-direction debiasing underperforms on generative VLMs — geometry distortion from Euclidean subtraction, token-level bias heterogeneity, and single-direction rigidity for multi-class attributes like race — and propose GGSS as a norm-preserving geodesic replacement.
2. An adaptive, training-free token gate. A per-token confidence gate is calibrated from the distribution of bias-projection magnitudes observed during offline discovery (median and standard deviation), assigning stronger steering to tokens with unusually large protected-attribute components and little or none to low-bias tokens, with no additional training.
3. A unified comparison suite. Representative CLIP- and LM-era debiasers (INLP, MeanDiff with SVM and logistic-regression probes, BendVLM, LEACE, plus prompt-based mitigation) are ported to the same late vision-to-language projection layer of generative VLMs so differences reflect the steering rule rather than the hook location.
4. A controlled four-model evaluation. Four generative VLMs are evaluated under one best-average-α operating-point protocol with bootstrap confidence intervals and paired permutation tests, covering race and gender bias tasks plus MMStar capability preservation.

Main Findings

• Lowest average bias on every backbone. GGSS attains the lowest Avg Δ% among all external baselines on all 4 models: −55% vs. −37% on Pixtral-12B (BendVLM (Eucl.) strongest external), −90% vs. −86% on LLaVA-1.6-Vicuna-7B, −80% vs. −78% on LLaVA-1.6-Mistral-7B, and −60% vs. −49% on Qwen3-VL-4B (the latter three against INLP (sph.), the strongest competitor overall).
• Large per-task reductions. Up to −96% on Nurse/Doctor for LLaVA-1.6-Vicuna-7B (0.600 → 0.025), −84% on MCQ for LLaVA-1.6-Vicuna-7B (8.70 → 1.36), and −61% on 2AFC for Pixtral-12B (0.243 → 0.096).
• Capability is preserved. MMStar stays within ±0.6 p.p. of the unsteered baseline across the paper's operating points, with per-model MMStar deltas under race/gender steering ranging from −0.40 to +0.60 p.p.; McNemar per-question p-values range from 0.09 to 1.00.
• Statistically significant on three of four backbones. Paired sign-flip permutation tests of GGSS vs. the unsteered model give p < 10⁻⁴ (Pixtral-12B), p = 0.007 (LLaVA-Mistral-7B), p = 0.002 (Qwen3-VL-4B), and p = 0.068 on LLaVA-Vicuna-7B (where the Nurse/Doctor task alone has p < 10⁻⁴).
• Several baselines damage the model. BendVLM (sph.) collapses on Pixtral and LLaVA-Vicuna, and some runs produced no parseable answers; the paper attributes this to difficulties porting a CLIP-space method to generative activations rather than a flaw of the original method.
• Ablations attribute the gains. On Qwen3-VL-4B MCQ salary (race) at α = 1.0, JSD×1k drops from 14.646 (unsteered baseline) to 4.261 for Pooled SVD (Eucl., hard projection, no gate, 70.9% bias reduction), 5.612 for Pooled SVD (sph., 61.7%), 5.612 for GGSS without the gate (gᵢ ≡ 1, 61.7%), 4.533 for GGSS without Slerp (hard projection + gate, 69.1%), and 2.414 for full GGSS (83.5% reduction). The authors conclude that on-sphere steering drives robustness at scale while the gate drives selectivity.
• Gate hyperparameters matter. A sensitivity sweep over κ ∈ {1, 2, 5, 10} and g_floor ∈ {0.0, 0.3, 0.5} at α = 1.0 on the same Qwen3-VL-4B MCQ salary (race) setting shows JSD×1k values ranging from 2.414 (κ = 5, g_floor = 0.3, the configuration used throughout) up to 10.971 (κ = 5, g_floor = 0.0).
• Added baselines comparison. GGSS shows a four-model mean Avg Δ% of −71.4 with worst case −55.3, versus LEACE + our gate (−63.5 mean, −30.8 worst), INLP (sph.) (−61.1 mean, −31.2 worst), and LEACE (sph.) (−48.5 mean, −26.4 worst).
• Prompt choice is immaterial. Re-running discovery with four alternative prompts yields subspaces identical to numerical precision (principal-angle cosines ≥ 0.9997) and identical end-to-end results, because the vision-to-language projection computes visual-token activations before interacting with the text prompt.
• Reporting caveat. Avg Δ% averages only over non-zero-baseline tasks — three on Pixtral-12B and Qwen3-VL-4B, two on the LLaVA models — so it is not strictly comparable across models. The two LLaVA models show zero baseline 2AFC bias.

Methodology in Plain English

GGSS works in two stages and never updates the model weights.

Offline discovery. The team takes counterfactual image sets where the same person is shown with only a demographic attribute changed, and passes them through the frozen VLM to collect pooled visual activations. Because the intervention layer sits at the vision-to-language projection, these activations are unaffected by the text prompt. Each image is normalized to a unit-length point, and per-context Fréchet means on the unit hypersphere serve as local centers. The shifts from each center to each attribute variant are mapped into tangent space, stacked into a matrix, and decomposed with SVD. The top k right singular vectors (k = |A| − 1 by default) form the bias subspace — a multi-dimensional, norm-aware description of how demographic attributes move activations. A global Fréchet mean over all discovery points provides the reference point, and the distribution of bias-projection magnitudes across discovery points (median and standard deviation) calibrates the gate.

Online inference. Each visual token is split into its radius and its direction. The direction is mapped to the tangent space at the reference point, split into a protected-attribute component and a clean remainder via orthogonal projection, and the clean remainder is mapped back to the sphere as a target direction. A per-token gate compares the token's protected-coordinate magnitude to the discovery statistics, so face- or clothing-like tokens with strong demographic signal get steered more while pose, background, or textual tokens barely move. The token direction is then rotated toward the target using Slerp, with strength βᵢ = α · gᵢ, and the original radius is restored. A proposition proves the projection is the Euclidean least-squares projection onto the zero-protected-coordinate set and that the token norm is exactly preserved.

Evaluation protocol. Four generative VLMs (Pixtral-12B, LLaVA-1.6-Vicuna-7B, LLaVA-1.6-Mistral-7B, Qwen3-VL-4B-Instruct) are tested at the same late projection layer. Counterfactual images are the 480 real-photograph, face-only images from REFLECT/FOCUS, covering 6 occupations × 8 source identities × 5 perceived races × 2 genders. Bias is measured on MCQ (race, scored as mean_race_jsd ×10³), 2AFC (race, scored as race_bias_std), and Nurse/Doctor (gender classification gap). MMStar (1,500 questions, six dimensions, scored with VLMEvalKit) measures general capability. For each method–model pair, α is swept over {0.25, 0.5, 0.75, 1.0, 1.5} and a single best-average-α is chosen for all bias metrics and MMStar. GGSS fixes κ = 5 and g_floor = 0.3, discovers on held-out occupations (race tasks: cook, doctor, lawyer, nurse, teacher; gender task: cook, lawyer, teacher, holding out nurse and doctor), and is scored on probes that do not share identities with the discovery pool.

Why This Matters

For research, the paper reframes inference-time debiasing of generative VLMs as a geometry problem rather than a subtraction problem, and argues that an adaptive, selective, norm-preserving operator is a better primitive than hard uniform projection. It also supplies a porting suite — ten adapted baselines at a shared intervention layer — that makes future comparisons more meaningful than reusing CLIP-era numbers, and it separates two questions the authors say are often conflated: whether a steering rule reduces demographic sensitivity, and whether the same operating point preserves general multimodal reasoning.

Real-world applications the setting enables (the paper motivates human-centered and high-stakes use; specific deployments are not evaluated):

• Recruitment and hiring pipelines that use VLM-based screening or candidate summarization, where perceived race or gender could otherwise shift outputs.
• Decision-support tools in which a model's categorical answers about people influence downstream human judgments.
• Captioning, visual question answering, and retrieval systems that describe people in images, where biased descriptions propagate to users.
• Frozen production checkpoints, where a lightweight inference hook is viable but retraining or curating a large bias corpus is not.

Industry relevance: the intervention requires no retraining, works on frozen checkpoints, can be reconfigured as the target notion of bias changes, and is packaged as code at https://github.com/dukesun99/GGSS. That combination — no weight changes, token-level selectivity, and preserved benchmark capability within ±0.6 p.p. — is what makes it plausible as a deployment-stage mitigation rather than a research-only artifact.

Future Directions

• Multi-attribute and intersecting bias. The subspace dimensionality is set to |A| − 1 by default; how well the approach extends to several interacting protected attributes simultaneously (rather than race or gender treated separately) is not established here.
• Generalizing beyond the tested notions of bias. The evaluation covers race-related MCQ/2AFC and a binary nurse/doctor gender gap; whether the same discovery procedure transfers to age, nationality, religion, or disability, and to attributes without clean counterfactual image pairs, remains open.
• Hook-site and architecture generality. Results are reported at one late vision-to-language projection layer; additional hook sites appear in the appendix, but broader questions — transfer to closed models, to encoder-decoder VLMs, or to generative image models — are not answered.
• From benchmark bias metrics to real outcomes. The paper measures JSD, race_bias_std, and occupation-classification gaps under a single best-average-α protocol; whether those reductions translate into fairer behavior in open-ended generation, long-form captions, or downstream human decisions is not reported, and the authors note the Avg Δ% column is not strictly comparable across models.

Target Audience

Researchers and practitioners working on AI fairness, VLM safety, and activation steering will get the most from this paper, particularly those who need an inference-time method that does not touch model weights. It is also relevant to ML engineers deploying frozen generative VLMs in human-centered settings, and to benchmark designers, since the paper contributes a unified porting suite of ten adapted debiasing baselines at a shared intervention layer. Readers without a geometry background can follow the motivation, protocol, and results tables, but the framework section assumes comfort with tangent-space operations, Fréchet means, and Slerp.

Authors’ abstract

Generative vision-language models (VLMs) are increasingly used in human-centered settings, yet they can produce demographically biased outputs even when images differ only in controlled attributes such as perceived race or gender. However, existing inference-time debiasers were largely designed for static embeddings or CLIP-like models rather than generative VLMs. We propose GGSS---Geodesic-Gated Spherical Steering---a norm-preserving intervention that discovers a counterfactual bias subspace on the unit hypersphere, steers visual tokens along geodesic arcs, and uses an adaptive gate to focus correction on tokens that carry stronger demographic signal. We evaluate four generative VLMs against ten adapted inference-time debiasing baselines and prompt-based mitigation under a single operating-point protocol across categorical, pairwise, and occupation-gender bias tests, while also measuring general visual-language capability. GGSS achieves the lowest average bias on all four models, significant on three of four backbones under paired permutation tests, while preserving MMStar accuracy within +/- 0.6 p.p. of the unsteered baseline. Code is available at https://github.com/dukesun99/GGSS.