SynthFaces: Balanced Large Scale Human Dataset

AI summary

Overview

Research area: Computer vision and generative modeling — specifically synthetic human-face dataset construction, identity-preserving image generation, and model fairness.

Technical level: Intermediate. The paper combines Riemannian flow matching on the unit sphere with a multi-stage diffusion-based data pipeline, but the core argument and results are readable with a working knowledge of diffusion models and face-recognition metrics.

Scope: The paper proposes and evaluates a three-stage pipeline (an identity-latent sampler called ID-Flow, Arc2Face-based face synthesis with filtering, and Flux/InfiniteYou contextualization) that produces SynthFaces, a label-balanced, ID-grouped, captioned synthetic human dataset of roughly 150,000 images, and tests it on personalized generation, face-recognition fairness, and demographic VQA.

What This Paper Is About

Human-centric datasets built from scraped web images are demographically skewed and carry privacy and consent risks, so models trained on them — especially identity-preserving generators — perform unevenly across subgroups. The authors build a controllable generator that samples face-identity embeddings directly in their natural geometry (a unit hypersphere), conditioned on race, age, and gender labels, and then turn those identities into images. The goal is a released dataset whose label distribution is close to uniform, whose identities are synthetic and empirically separated from the real embeddings used for training, and whose images are high-resolution, captioned, and grouped by identity.

Key Contributions

1. ID-Flow, a geodesic flow-matching model on the identity manifold. Rather than treating identity embeddings as unconstrained Euclidean vectors, the authors learn a velocity field on the unit hypersphere using geodesic conditional flow matching, with a classifer-free-guidance interface over three label axes (race, age, gender). The model is trained on roughly 50k deduplicated ArcFace embeddings from CosmicMan-HQ, FFHQ, and EasyPortrait, and is sampled with a guidance scale of 2.0.
2. A three-stage data construction pipeline with quantitative filtering. Stage 1 samples 28 thousand identity embeddings; Stage 2 renders faces with Arc2Face and filters them via image statistics (entropy, saturation), vision-language-model artifact/label-alignment checks, and per-subgroup similarity thresholds, removing approximately 3k images on the statistics step alone and leaving 15 thousand filtered identities; Stage 3 contextualizes them with InfiniteYou and Flux 2 into roughly 150,000 captioned high-resolution images at approximately 10 images per identity.
3. An empirical non-leakage measure (ID-NonLeak). The paper reports average nearest-neighbor cosine separation from real training identities, showing that ID-Flow samples are nearly orthogonal to training data (0.741) whereas embedding interpolation, which scores better on purely generative metrics, stays close in angular proximity (0.43).
4. Downstream fairness evaluations across three tasks. Fine-tuning SD-1.5 + IPAdapter-FaceID, training a face-recognition model on 500K synthetic images, and LoRA fine-tuning LLaVA-1.5-7B for demographic VQA, each compared against baselines on subgroup-disparity metrics.

Main Findings

• Label balance far exceeds conventional datasets. Averaged KL divergence from a uniform label distribution is 0.048 for SynthFaces, against 0.449 (FFHQ), 0.533 (SHHQ), 0.508 (EasyPortrait), 0.576 (CosmicHQ-1.0), 0.49 (HyperFace-10k), and 0.75 (PCA sampling).

• Identity sampling beats both learned samplers and naive baselines on diversity, at competitive image quality. ID-Flow achieves FID-e 0.073, recall 0.75, F1 0.48, and ID-NL 0.741, versus Arc2Face (FID-e 0.118, F1 0.29), HyperFace (0.13, 0.26), StyleGAN (0.111, 0.28), and SDXL-PhotoMaker (0.207, 0.02). Image FID is 39.9, close to Arc2Face's 37.3. Parameter-free interpolation scores better on FID (30.59, FID-e 0.035) but leaks identity information (ID-NL 0.43).

• Filtering trades label uniformity for image quality. After Stage 2 filtering, FID-e improves from 0.163 to 0.127 and FID-i from 65.59 to 57.66, while F1 drops from 0.496 to 0.444 and KL-age worsens sharply from 0.000108 to 0.121146 — the authors attribute this to the problematic tails not being uniformly distributed across demographic labels.

• Balanced prompting of a text-to-image model is not a substitute for identity-manifold sampling. Comparing 1,000 Flux.2-dev-Turbo seed faces against 1,000 ID-Flow samples, balanced prompting reaches label KL 0.101 and identity diversity 0.201, versus 0.037 and 0.751 for ID-Flow — described as 2.7× farther from uniform and 3.7× lower identity diversity. Unconditional Flux.2 scores 0.891 and 0.257.

• SynthFaces images score better than Cosmic-HQ on human-perception and text-alignment metrics while keeping faces moderate in size. ImageReward 1.38 vs. −0.69, CLIP 33.04 vs. 31.20, face-size ratio 0.29 vs. 0.22. FluxSynID scores higher on both metrics (1.54, 34.07) but with a face-size ratio of 0.61, which the authors say leaves little background context.

• Fine-tuning on the dataset improves identity preservation without a large average cost on contextual prompts. SD-1.5 + FaceID goes from ID 68.63 / CLIP-T 32.91 to 71.31 / 33.27 on FairBench, and from 28.81 / 53.56 to 28.49 / 51.56 on the Context benchmark; the authors report the per-label gains are broadly distributed rather than isolated.

• Face-recognition fairness improves on balanced evaluation, with an accuracy caveat. On BFW (about 900K images, 4 races × 2 genders), SynthFaces-500K reaches accuracy disparity 0.043 (Vec2Face 0.087, HyperFace 0.068), SER-max 1.074 at 1% FMR (vs. 1.107 and 1.143), and 1.011 at 0.1% FMR (vs. 1.057 and 1.101). Across the five standard benchmarks at FPR = 0.1%, mean maximum racial SER is 1.097 vs. 1.531 for Vec2Face and 1.280 for HyperFace — reported as a 40% reduction relative to Vec2Face. The paper notes that SynthFaces-500K's lower global accuracy on established benchmarks reflects the balance/accuracy trade-off.

• Demographic VQA accuracy and cross-attribute parity both improve. LLaVA-1.5-7B fine-tuned on SynthFaces-HQ reaches 0.700 average accuracy and Δ̄ 0.151 (baseline 0.551 / 0.183); SynthFaces-nonHQ reaches 0.797 / 0.138, including +23.3 pp race accuracy over baseline. Against Vec2Face on the domain-matched setting, it gains +0.125 lower Δ̄ and +23.8 pp average accuracy, while Vec2Face fine-tuning regresses on gender and age.

• Standard FR benchmarks are themselves skewed. The paper reports AgeDB-30 as 88% White and LFW as 66% White and 75% male, which motivates fixing FMR at a calibrated threshold and reporting per-group FNMR rather than raw accuracy disparity.

Methodology in Plain English

The authors attack the problem in three stages, each aimed at a specific defect of existing datasets.

Stage 1 — generating diverse identities without copying real ones. Face-recognition models represent a person as an embedding vector of unit length, so all real identities sit on the surface of a high-dimensional sphere. Instead of training a generator in ordinary flat space, the authors train a flow-matching model that moves points along shortest paths (geodesics) on that sphere, from a uniform starting distribution toward the distribution of real identity embeddings. Because the projections onto and off the sphere have closed-form equations, the model can be trained without simulating the process. The model also takes a three-part label condition — race, age, gender — so sampling with uniformly chosen labels produces a balanced set of identities. This yields 28 thousand identity embeddings.

Stage 2 — turning embeddings into clean faces. Each embedding is fed to Arc2Face, a diffusion model that renders a face from an identity vector. Because generated images include failures, the authors filter in three passes: dropping images at the tails of the entropy and saturation distributions (over-stylized or black-and-white outputs, roughly 3k images removed); asking several Qwen-VL models whether each face matches its intended labels and contains artifacts, keeping images approved by majority vote and only rejecting when the model's stated reasoning names a concrete defect; and thresholding pairwise embedding similarity within subgroups, because rare label combinations in the ID-Flow training data produced clusters of near-duplicate faces. This leaves 15 thousand identities.

Stage 3 — placing faces in context. Captions are drawn from real datasets (Cosmic-HQ, FFHQ-in-the-Wild) and human-caption collections, filtered to strip out any facial attributes that were controlled during identity sampling, and bucketed by suitable age so that a caption for a child is not paired with an adult face. Each identity is then rendered into multiple captioned scenes with InfiniteYou (with added realism and deblurring LoRAs) and Flux 2 Turbo, producing the final set of roughly 150,000 images at about 10 per identity.

Evaluation. Balance is scored as KL divergence from a uniform label distribution using a FairFace classifier. Leakage is scored with ID-NonLeak, the average cosine distance to the nearest real training embedding. Downstream, the authors fine-tune an IPAdapter-FaceID pipeline using "shortcut finetuning" (a frozen ControlNet conditioning on pose and face position, so the adapter learns only face-relevant features rather than the dataset's face-size distribution), train a face-recognition model on 500K synthetic images, and LoRA-tune LLaVA-1.5-7B on demographic VQA, using a custom FairBench probe set for the identity-generation analysis and a 37,316-question held-out VQA benchmark.

Why This Matters

Impact on research. The paper makes the case that fairness in human-centric models can be intervened on at data-construction time, and it provides a concrete, released artifact plus a leakage metric that distinguishes "diverse" from "copied." Its demonstration that embedding interpolation wins on FID while quietly leaking training identities is a useful methodological caution: generative metrics alone do not certify a privacy-safe synthetic dataset. The geodesic flow-matching formulation is also a small but clean contribution to manifold-aware generative modeling.

Real-world applications.

• Training identity-preserving portrait tools (avatars, virtual try-on, video dubbing) that do not degrade for underrepresented demographic groups.
• Building face-verification systems for deployment contexts where a fixed false-alarm rate is required and differential miss rates across groups carry legal or safety consequences.
• Constructing fairness audits and benchmark suites (the paper's FairBench) for face technologies without exposing real people's images.
• Data augmentation and demographic-attribute recognition for assistive or accessibility systems, where consent-safe and balanced imagery is a practical bottleneck.

Industry relevance. Organizations that cannot scrape faces for legal, consent, or regulatory reasons — including those affected by withdrawn datasets — gain a synthetic alternative with captions and multiple images per identity, which matches the data format identity-preserving generators actually need. The paper's explicit attention to captioning, ID-grouping, and high resolution maps directly onto training-data requirements for adapter-based personalization products.

Future Directions

• Reducing dependency on biased foundation models. The authors state that the expressive range of generated identities is bounded by Arc2Face's identity-preservation capacity and Flux's generation quality, and that within-group visual diversity and style remain constrained by upstream training corpora — improving or replacing these models is an open lever.
• Closing the synthetic-to-real gap. VLM filtering removes overt defects, but residual distributional artifacts could limit transfer when SynthFaces-trained models meet real photographs; measuring and mitigating that gap is left open.
• Developing the usage license and release terms. Table 11 lists SynthFaces' license as "in process," with the stated aim of preventing application in unethical or harmful domains.
• Hybrid real–synthetic balancing. The conclusion explicitly calls for investigating strategies that combine real and synthetic data for demographic balancing, rather than relying on synthetic data alone.
• Stronger privacy guarantees. The authors decline to claim a formal privacy guarantee because ID-Flow is trained on ArcFace embeddings from real datasets, relying instead on the empirical ID-NonLeak metric.

Target Audience

Researchers and practitioners working on generative face models, identity-preserving personalization, and fairness in computer vision will get the most from this paper. It is also relevant to dataset curators and ML engineers who need consent-safe, balanced human imagery for training or evaluation, and to fairness researchers interested in error-rate-ratio metrics for biometric systems. Readers without a background in diffusion models or Riemannian geometry can still follow the pipeline, the filtering rationale, and the downstream results, since the mathematical content is concentrated in Stage 1 and the Appendix.

Authors’ abstract

The pursuit of fair and equitable AI in human-centric tasks is critically hindered by the inherent biases and privacy risks of real-world datasets, which offer imbalanced representations of identity and often contain sensitive personal information. Consequently, state-of-the-art generative models trained on this skewed data exhibit unfair performance, particularly in identity-preserving generation. In this work we propose a pipeline for controllable generation of dataset with diverse faces for generative models training and finetuning. We achieve this by expanding foundational Arc2Face diffusion model with a flow matching generative model in the manifold of identity embeddings. We demonstrate that such approach achieves both goals of generative fidelity and label diversity not present in real human-centric datasets. As a result we collect and further contextualize a highly balanced identity-centric dataset of humans named SynthFaces. By design, our released images and identities are synthetic, reducing direct reliance on real personal images; we empirically check identity separation with a nearest-neighbor non-leakage metric. We purpose it for training and tuning of fair models for various downstream tasks, and showcase its usability in finetuning an existing adapter-based method for personalized generation.