Fairness Without Labels: Pseudo-Balancing for Bias Mitigation in Face Gender Classification

AI summary

Overview

• Research area: Computer vision fairness, specifically bias mitigation for automated face gender classification, using semi-supervised learning (pseudo-labeling/self-training).
• Technical level: Intermediate. The method itself is simple (a change to the sample-selection step of existing pipelines), but the experimental design assumes familiarity with FixMatch, FlexMatch, selection rate, and demographic fairness metrics.
• Scope in one sentence: The paper proposes "pseudo-balancing," a label-free modification to self-training that enforces gender parity during pseudo-label selection, and evaluates it on the All-Age-Faces (AAF) benchmark under both realistic and deliberately imbalanced training conditions.

What This Paper Is About

Face gender classification models inherit and amplify demographic biases from their training data, performing unevenly across gender and racial subgroups. The authors ask whether a biased model can be made fairer using only unlabeled images drawn from a demographically balanced source (FairFace), without ever accessing its ground-truth labels. They introduce pseudo-balancing, which equalizes the number of male and female pseudo-labeled samples fed back into training, and test how far this simple constraint can go in reducing accuracy gaps on the predominantly East Asian All-Age-Faces benchmark.

Key Contributions

1. Pseudo-balancing. A simple bias-mitigation method that enforces equal pseudo-gender representation during self-training, requiring no labeled source data and no adversarial training or distributional assumptions.
2. An evaluation framework for compounding biases. Controlled experiments combining FairFace and All-Age-Faces that isolate how dataset skew and model bias interact, spanning a realistic scenario and a synthetic severe-bias scenario.
3. Empirical evidence of fairness gains. Results showing that pseudo-balancing reduces gender disparities while maintaining or improving accuracy on AAF, with 79.81% overall accuracy (a 6.53% gain over the baseline) and a 44.17% reduction in the gender accuracy gap.

Main Findings

• Headline result: Pseudo-balancing reaches 79.81% overall accuracy, a 6.53% improvement over the baseline, and reduces the gender accuracy gap by 44.17%. In the East Asian subgroup, where baseline disparities exceeded 49%, the gap narrows to 5.01%.
• Baseline is heavily skewed: The pre-trained Kaggle ResNet18-based model scores 73.28% accuracy with a Selection Rate (SR) of 49.63% (97.94% male / 48.61% female). The paper defines SR as min accuracy divided by max accuracy, with SR of 80% or above indicating mitigated bias.
• Pseudo-balancing helps when the unlabeled data is balanced or moderately biased: With FixMatch at confidence threshold 0.9, FairFace with pseudo-balancing yields 79.81% accuracy and 93.80% SR, versus 75.73% accuracy and 55.59% SR without it. At threshold 0.6, FairFace with pseudo-balancing yields 80.05% accuracy and 76.80% SR, versus 76.92% and 59.93% without.
• Highest accuracy comes from FlexMatch on FairFace with pseudo-balancing: 83.36% accuracy with 86.03% SR (89.62% male / 77.10% female), compared with 79.18% accuracy and 74.77% SR without pseudo-balancing.
• Pseudo-balancing can backfire under severe bias: On the Severe East-Asian Female FairFace subset with FixMatch at 0.9, pseudo-balancing gives 71.63% accuracy and 48.82% SR, while the non-pseudo-balanced run gives 81.30% accuracy and 84.78% SR. The authors attribute this to the baseline model's poor initial performance on East Asian faces irrecoverably skewing pseudo-label quality.
• Racial subset results differ: On the Severe Black FairFace subset with FixMatch at 0.9, pseudo-balancing reaches 74.41% accuracy and 53.87% SR, versus 71.35% accuracy and 44.08% SR without. The authors suggest the Kaggle model's Western-centric pre-training provides better initial representations for African facial features than for Asian ones.
• Higher confidence thresholds help: FixMatch at threshold 0.9 with pseudo-balancing drives the gender accuracy gap to 5.01% by iteration 7, surpassing FlexMatch by 7.51%.
• Including the target race matters more than balance alone: Fine-tuning with FixMatch on severely biased subsets such as East-Asian FairFace (79.33% accuracy, 76.36% SR at threshold 0.6) and East-Asian Female FairFace (81.30% accuracy, 84.78% SR at 0.9) performs on par with models fine-tuned on all of FairFace or the gender-balanced GB-FF, when the training data matches the target domain's demographics.
• Supervised fine-tuning context: With ground-truth labels, fine-tuning on FairFace reaches 87.54% accuracy and 83.59% SR, mixed data reaches 95.17% and 97.46% SR, and sequential FairFace plus mixed fine-tuning reaches 96.22% and 99.95% SR — establishing the ceiling that the label-free method is measured against. Fine-tuning on FairFace alone leaves a gender accuracy gap of 26%, reduced from the original 49%.
• FlexMatch is unstable under extreme bias: It achieves its best result of 83.36% accuracy with pseudo-balancing on FairFace, but underperforms FixMatch and even the baseline on severely biased distributions. The paper reports that pseudo-balancing improves FlexMatch on GB-FF (67.49% to 83.36% as stated in the discussion text), while the GB-FF row in Table 4 lists 67.49% accuracy and 42.19% SR with pseudo-balancing.
• A figure caption reports a slightly different peak: The caption of Figure 3 states the best model achieves 82.4% accuracy at 93.8% selection rate, while the main text reports 79.81% accuracy at 93.80% SR for FixMatch at threshold 0.9 with pseudo-balancing.

Methodology in Plain English

The authors start from a publicly available ResNet18-based classifier pre-trained on the Kaggle Gender Classification Dataset, which performs well on Western faces but poorly on others. They then fine-tune it using the FixMatch and FlexMatch self-training frameworks on unlabeled images from FairFace and its variants, using confidence thresholds of 0.1, 0.3, 0.6, and 0.9 for FixMatch and a base threshold of 0.95 with a non-linear x/(2-x) adjustment for FlexMatch. Standard self-training accepts every confident prediction the model makes, which tends to reinforce whatever the model already believes. Pseudo-balancing changes only the sample-selection step: after each round, the confident predictions are split into male and female groups and an equal number are drawn from each before retraining. Training ran for 10 epochs per iteration using SGD with learning rate 0.001, momentum 0.9, batch size 16, and early stopping after 2 epochs without improvement. Data splits included 86,744 FairFace training samples (40,758 female, 45,986 male), a gender-balanced test set of 2,504 AAF samples, and variants such as a 43,372-sample gender-balanced GB-FF, race-specific subsets (12,287 East-Asian, 12,233 Black, and 6,141 East-Asian samples), and gender-skewed sets of 41,803 samples at 80% female and 44,941 at 80% male. Evaluation used overall accuracy and Selection Rate on the AAF benchmark.

Why This Matters

Impact on research. The work shows that the demographic composition of unlabeled data can be a usable fairness lever, not just the composition of labeled data. It frames pseudo-balancing as a drop-in complement to existing self-training methods, and it documents a clear boundary condition: balancing helps on balanced or moderately skewed pools but can amplify bias when severe dataset bias overlaps with the base model's own bias — the "Matthew effect" in pseudo-labeling.

Real-world applications.

• Video surveillance and law enforcement face analysis, where biased systems have been linked to disproportionate targeting.
• Demographic analysis and audience measurement pipelines built on automated face attribute prediction.
• Targeted advertising, where gender classification governs who sees which content.
• Human-computer interaction and automatic gender recognition, where misgendering carries social harm amplified by the perceived objectivity of computer systems.

Industry relevance. Recollecting balanced labeled datasets such as FairFace is expensive. Pseudo-balancing offers a cheaper route: reuse an existing biased model, point it at a balanced unlabeled pool, and rebalance the pseudo-labels. The results also set expectations — the method is not a fix for extreme imbalance, and threshold choice (0.6 versus 0.9) materially changes the accuracy-fairness trade-off.

Future Directions

1. Hybrid methods for intersectional bias. The authors propose combining pseudo-balancing with demographic-aware augmentation to handle complex intersecting biases, such as the East Asian female case where the method currently fails.
2. Handling severe imbalance better. FlexMatch's adaptive thresholding did not stabilize training under extreme racial or gender imbalance, indicating a need for thresholding that accounts for demographic skew.
3. Dependence on the base model. Pseudo-balancing relies on the initial model producing reasonable pseudo-labels; this breaks down for demographic groups entirely absent from pre-training data, so alternative initialization strategies are needed.
4. Beyond gender and race. The paper notes that age, expression, and cultural context variations remain unaddressed, and that extending to other underrepresented groups requires validation through larger-scale multicultural studies beyond the East Asian populations where results were most promising.

Target Audience

Researchers and practitioners working on fairness in computer vision, semi-supervised learning, and face analysis who need debiasing methods that do not depend on demographic labels. It is also useful for applied engineers deploying face attribute models in surveillance, advertising, or analytics, and for readers who want a concrete, carefully controlled case study of when balancing tricks help and when they backfire.

Authors’ abstract

Face gender classification models often reflect and amplify demographic biases present in their training data, leading to uneven performance across gender and racial subgroups. We introduce pseudo-balancing, a simple and effective strategy for mitigating such biases in semi-supervised learning. Our method enforces demographic balance during pseudo-label selection, using only unlabeled images from a race-balanced dataset without requiring access to ground-truth annotations. We evaluate pseudo-balancing under two conditions: (1) fine-tuning a biased gender classifier using unlabeled images from the FairFace dataset, and (2) stress-testing the method with intentionally imbalanced training data to simulate controlled bias scenarios. In both cases, models are evaluated on the All-Age-Faces (AAF) benchmark, which contains a predominantly East Asian population. Our results show that pseudo-balancing consistently improves fairness while preserving or enhancing accuracy. The method achieves 79.81% overall accuracy - a 6.53% improvement over the baseline - and reduces the gender accuracy gap by 44.17%. In the East Asian subgroup, where baseline disparities exceeded 49%, the gap is narrowed to just 5.01%. These findings suggest that even in the absence of label supervision, access to a demographically balanced or moderately skewed unlabeled dataset can serve as a powerful resource for debiasing existing computer vision models.