How Private is Private? A Comparative Study for Face De-Identification

AI summary

Overview

• Research area: Face de-identification (FDeID) — privacy-preserving computer vision, benchmark and evaluation design.
• Technical level: Intermediate. The paper assumes familiarity with face recognition embeddings, adversarial perturbations, GANs, diffusion models, and remote photoplethysmography (rPPG), but its central argument is conceptual rather than mathematically dense.
• Scope (one sentence): The paper revisits how face de-identification is evaluated, introducing a curated benchmark (UtilFace), a unified hierarchical metric (HiFD), and a comparative study of 12 de-identification methods spanning adversarial, GAN-based, and diffusion-based paradigms.

Affiliations: ELLIS Institute Finland and the Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland. Posted as arXiv:2610.10334v1 [cs.CV], 07 Oct 2026, under CC BY 4.0. Project page: https://cv-ac.github.io/hifd/.

What This Paper Is About

Face de-identification aims to alter a face image so that identity recognition fails while other useful facial information survives — but the paper argues that the field cannot currently answer the question in its title, "how private is private?", because every method is measured differently. The authors reviewed over 100 FDeID-related papers and found more than 60 datasets and 20 evaluation metrics in use, so a method optimized for expression preservation cannot be compared against one designed for detection preservation. The paper's goal is to replace this fragmented landscape with one shared benchmark and one shared metric, then use them to run the first method-to-method comparison across the three dominant FDeID paradigms.

Key Contributions

1. UtilFace, a curated, demographically balanced FDeID benchmark. A four-stage pipeline (source aggregation, identity-aware cleaning plus resolution enhancement, perceptual quality filtering, and hierarchical stratified sampling) reduces roughly 69.55M images across roughly 3.12M identities from WebFace42M, Glint360K, MS1MV2, and MegaFace down to 99,928 images across 2,069 identities (mean 48.3, median 30 images per identity). Gender balance is near-uniform (51.5% / 48.5%) and the largest-to-smallest ethnicity category ratio drops from 53x to 2.3x. UtilFace carries no attribute annotations by design.

2. A three-level facial utility hierarchy. The first structured formalization of facial utility for FDeID, organizing signals by perceptual accessibility: macro-cues (L1: age, gender, ethnicity, macro-expression, landmarks), micro-cues (L2: gaze, micro-expression), and imperceptible cues (L3: rPPG-derived blood-volume-pulse and heart rate).

3. HiFD, a unified consistency-based metric. HiFD combines identity suppression (P̄), image quality (Q), and the three hierarchical utility scores (U₁, U₂, U₃) into a single scalar via weighted harmonic mean, with configurable application-specific weighting profiles and graceful adaptation across image and video via L ∈ {1, 2, 3}. Every component is computed from pretrained estimators' outputs on the original face and its de-identified counterpart — no per-image ground-truth labels are required.

4. A comprehensive comparative study of 12 FDeID methods across adversarial perturbation (MI-FGSM, PGD, TI-DIM, TIP-IM, Chameleon), GAN-based generation (CIAGAN, AMT-GAN, Adv-Makeup, DeID-rPPG, G²Face), and diffusion-based generation (DiffAM, WeakenDiff), surfacing trade-offs and failure modes invisible under existing protocols.

Main Findings

• The privacy-utility frontier is not monotonic across paradigms. Adversarial methods retain the highest macro-cue scores (U₁ > 0.92 for PGD and MI-FGSM), as expected from minimally visible pixel perturbations, but only TI-DIM and TIP-IM — the two with explicit transferability objectives — reach P̄ ≈ 0.50. MI-FGSM and PGD stall at P̄ < 0.14, de-identifying only 8% of faces at any false-accept rate (Appendix C.2). GAN methods span the widest privacy range: CIAGAN reaches P̄ = 0.45 by replacing the face entirely, but at severe cost to U₁ (0.47) and landmarks (0.08), while Adv-Makeup preserves macro-cues almost perfectly yet achieves P̄ = 0.01. Diffusion methods sit between: WeakenDiff reaches P̄ = 0.22 and U₁ = 0.83, while DiffAM scores only P̄ = 0.16 on the ArcFace/CosFace/AdaFace ensemble because its makeup perturbations were trained against IRSE50/IR152/MobileFace/FaceNet and transfer imperfectly — yet DiffAM's better U₂ (0.81), U₃ (0.56), and Q (0.47) give it a higher HiFD of 0.394 versus WeakenDiff's 0.332.

• Hierarchical signals degrade at different rates. Preservation drops from L1 to L3 for almost every method, with the steepest collapse in generative families: AMT-GAN, CIAGAN, G²Face, and WeakenDiff all score U₃ < 0.33, indicating that face synthesis decouples macro appearance from fine-grained skin-color dynamics. Adversarial methods preserve U₃ best (TIP-IM 0.62, Chameleon 0.80) because bounded ℓ∞ or ℓ₂ noise leaves low-frequency skin signals largely intact. Only Adv-Makeup (U₃ = 0.93) and DeID-rPPG (U₃ = 0.91) combine high U₃ with high U₁, but both trade this for a complete loss of identity suppression.

• Hierarchical preservation is asymmetrically coupled. Across the 12 methods at threshold τ = 0.80, every method that preserves U₃ also preserves U₁ and U₂ (Pr(U₁ ≥ τ | U₃ ≥ τ) = Pr(U₂ ≥ τ | U₃ ≥ τ) = 1.00), yet only 22% (two of nine) of methods preserving U₁ also preserve U₃. The asymmetry holds for every τ ∈ [0.70, 0.90]. This refutes the shortcut of certifying fine-grained preservation by inspecting macro cues alone — exactly what the harmonic mean's vetoing property enforces. TIP-IM achieves the highest HiFD of 0.607 because it is the only method reaching at least 0.44 on every dimension simultaneously.

• Application-specific profiles reshape the ranking. Under Privacy-First (w_P = 0.5), CIAGAN rises within the GAN family, but TIP-IM still leads. Under Clinical (w₃ = 0.4), adversarial methods (TIP-IM, Chameleon) dominate because they retain physiological rPPG signals, while most GAN-based methods collapse. Adv-Makeup and DeID-rPPG remain at the bottom across all profiles: despite excellent utility preservation, their near-zero identity suppression (P̄ < 0.02) is mathematically penalized by the harmonic mean. No profile rescues a method that fails on identity suppression.

• FDeID methods are largely demographic-agnostic. Across all 12 methods, no per-gender gap exceeds 0.035 and no seven-way ethnicity range exceeds 0.064 — an order of magnitude below the inter-method spread. The four widest ethnicity ranges are PGD (0.064), Chameleon (0.050), MI-FGSM (0.049), and DiffAM (0.039), all gradient-based or makeup-style perturbations depending on local skin-tone gradients. Top-ranked TIP-IM has a gender gap of 0.006 and ethnicity range of 0.014; the bottom-ranked Adv-Makeup and DeID-rPPG show disparities ≤ 0.021. Neither leadership nor failure concentrates on any subgroup.

• Consistency scores track true utility. On macro-expression, the RAF-DB test set (3,068 images) was de-identified with all twelve methods; the consistency score and the fraction of ground-truth accuracy retained differ by at most 0.06 for every method, and both place CIAGAN far below the rest and agree on the three best-preserving methods. Per image, consistency predicts whether the true label survives with AUC 0.98 over 28,614 pairs, with only 1.1% of pairs above a consistency score of 0.8 hiding a flipped label. For rPPG, MMPD's contact sensor supplies the true heart rate; the paper notes the two scores live on different scales because heart-rate error saturates.

Methodology in Plain English

The authors made two changes — one to the data, one to the measurement.

Fixing the data. No existing face dataset jointly offered enough identity diversity, enough image quality, and enough demographic balance for multi-dimensional evaluation (see the paper's annotation-coverage table, which lists datasets from LFW's 13,233 images across 5,749 identities to GazeCapture's 2,445,504 images). The authors merged four large recognition-training datasets, discarded images with zero or multiple detected faces, non-RGB inputs, heavy blur, or cross-dataset duplicate identities identified via pretrained ArcFace similarity, upsampled surviving low-resolution images to 448×448 or 512×512 using blind face restoration (with the upsampling ratio capped at 4x), and filtered on NIQE ≥ 5.0. Because the remaining pool was skewed across gender, ethnicity, expression, pose, and age, they split identities into 14 strata by gender and ethnicity, ranked identities within each stratum by a composite score over image count plus expression and pose diversity, and filled an equal per-stratum budget.

Fixing the measurement. Existing protocols measure utility as classification or regression accuracy against human-annotated labels, which is why no single benchmark can support multi-attribute evaluation and why the field fragmented across task-specific datasets in the first place. The authors instead define utility as the consistency between an estimator's output on the original face and on the de-identified face. This requires no labels — only global normalization constants defined once per task (Δ_age = 100, τ_NME = 0.10, θ_max = 90°, τ_HR = 20 bpm, τ_N = 5, α = 0.5). Privacy uses an ensemble of ArcFace, CosFace, and AdaFace recognizers rescaled from cosine similarity to [0, 1], which the authors note mitigates the known vulnerability of single-recognizer evaluation against adversarial methods. Quality combines LPIPS and NIQE with equal weight. All five components are bounded in [0, 1] and aggregated with a weighted harmonic mean, chosen because it penalizes imbalanced methods more strongly than arithmetic or geometric averaging.

Estimators: MiVOLO (age, gender), FairFace (ethnicity), POSTER (macro-expression), MediaPipe Face Landmarker (68-point landmarks), L2CS-Net (gaze), SAMER (micro-expression), and an rPPG ensemble of PhysNet, EfficientPhys, and FactorizePhys. Evaluation uses UtilFace (99,928 images, 2,069 identities) for privacy, L1, L2 gaze, and quality; DFME test-set A (474 video clips) for micro-expression; and PURE (10 subjects, 60 video clips) for rPPG. All inference ran on 8 AMD MI250X GPUs.

Why This Matters

Impact on research. The paper diagnoses a structural problem rather than a performance gap: with more than 60 datasets and 20 metrics in use across over 100 reviewed papers, published FDeID numbers are largely incomparable. Its consistency-based reformulation removes the per-attribute annotation requirement that caused this fragmentation, and its three-level hierarchy exposes a failure mode — near-total collapse of imperceptible physiological signals — that a single scalar utility score cannot show.

Real-world applications:

• Surveillance and public-space video: choosing a de-identification method under the Privacy-First profile, where identity suppression carries the largest weight.
• Telehealth and remote patient monitoring: the Clinical profile, which weights rPPG preservation at w₃ = 0.4 — relevant since most GAN-based methods here score U₃ < 0.33.
• Affective computing and human-computer interaction: applications depending on gaze and micro-expression preservation, the L2 level.
• Dataset release and privacy compliance: the annotation-free metric lets data curators certify what survives de-identification without collecting new labels.

Industry relevance. Teams deploying anonymization for autonomous-vehicle footage, smart-device capture, or biometric pipelines need a decision procedure that reflects their own utility priorities, not a single leaderboard. The three weighting profiles and the vetoing property give that, and the paper's negative result — that two of the twelve methods offer essentially no identity suppression (P̄ < 0.02) — is a direct warning against adopting cosmetically convincing de-identification. The benchmark and evaluation toolkit are released.

Future Directions

• Closing the L3 gap in generative methods. AMT-GAN, CIAGAN, G²Face, and WeakenDiff all fall below U₃ = 0.35. Whether synthesis-based anonymization can be redesigned to preserve sub-pixel skin-color dynamics while still suppressing identity is left open.
• Breaking the U₁–U₃ asymmetry. The paper establishes that preserving macro cues implies nothing about imperceptible cues (only 22% of U₁-preserving methods preserve U₃), but does not propose a mechanism for making the two levels jointly achievable rather than trading off.
• Explaining and removing adversarial demographic disparities. The widest ethnicity ranges (PGD 0.064, Chameleon 0.050, MI-FGSM 0.049) are attributed plausibly to dependence on local skin-tone gradients; whether this can be mitigated is not tested.
• Extending the profile space and validation coverage. The paper validates HiFD on RAF-DB for macro-expression and MMPD for heart rate, but the supplied content does not report validation of every component; broader validation across the remaining estimators and finer-grained demographic strata remains an open problem.

Target Audience

Researchers and practitioners working on face de-identification, privacy-preserving computer vision, and face analysis benchmarking — particularly those who need to compare methods that target different utility dimensions. It is also useful for dataset curators and compliance-oriented engineers who need an annotation-free way to certify what survives anonymization, and for reviewers assessing whether a reported FDeID result is comparable to prior work. Readers should have some background in face recognition embeddings and generative models; the hierarchical metric itself is explained in accessible terms.

Authors’ abstract

Face de-identification (FDeID) has emerged as a critical privacy-preserving technology, yet its evaluation remains fundamentally fragmented. Existing protocols rely on inconsistent metrics, heterogeneous datasets, and partial annotation coverage, so methods targeting different utility dimensions, such as landmark versus expression preservation, are reported on different benchmarks under different metrics, rendering cross-method comparison infeasible. We revisit FDeID evaluation from both the data and metric perspectives. On the data side, we introduce UtilFace, a curated, demographically balanced benchmark with high identity diversity, assembled from four large-scale face datasets through identity-aware cleaning, resolution enhancement, and stratified filtering. On the metric side, we propose HiFD, a Hierarchical Face De-identification metric that unifies identity suppression, multi-level utility preservation, and image quality under a single consistency-based paradigm: every component is computed from pretrained estimators' outputs on the original face and its de-identified counterpart, directly quantifying how much identity is suppressed and how much downstream-perceivable utility survives. HiFD organizes facial signals into a three-level utility hierarchy spanning macro cues (L1), micro cues (L2), and imperceptible cues (L3), and aggregates the five resulting components into a single interpretable score via weighted harmonic mean, with configurable application-specific profiles. Using this unified protocol, we conduct a comprehensive comparative study spanning adversarial, GAN-based, and diffusion-based methods, surfacing trade-offs and failure modes that remain invisible under existing protocols. We release the benchmark and evaluation toolkit to foster systematic and reproducible research in privacy-preserving human face analysis.