Privacy Beyond Pixels: Latent Anonymization for Privacy-Preserving Video Understanding

AI summary

Overview

Research area: Computer vision — visual privacy preservation and fairness for video foundation models (action recognition, temporal action detection, anomaly detection).

Technical level: Advanced. The paper assumes familiarity with video foundation model architectures (ViT-style encoders, [CLS] tokens, CNN average pooling), contrastive self-supervised losses, adversarial/GAN-style training, and downstream video task metrics.

Scope: One-sentence scope: The paper proposes SPLAVU, a plug-and-play Anonymizing Adapter Module (AAM) that strips private information (e.g., skin color, gender, clothing) from the frozen latent features of video foundation models while preserving utility across multiple video understanding tasks.

Note: the supplied text is truncated mid-sentence in the Limitations paragraph, so the full limitations discussion is not available here.

What This Paper Is About

Video foundation models produce rich spatio-temporal features that are commonly extracted, stored, and reused across many downstream tasks — but those same features leak sensitive personal attributes such as skin color, gender, or clothing, which an attacker can recover with a simple classifier. Existing privacy methods anonymize at the input pixel level, which forces retraining the entire utility model and only works for one task at a time (e.g., action recognition only, or anomaly detection only). This paper instead anonymizes in the latent feature space, keeping the video encoder frozen and learning a small adapter that removes private information while retaining task utility.

Key Contributions

1. A new formulation of privacy preservation for general video understanding — anonymizing the latent embedding space of a frozen video encoder rather than the input pixels, which avoids retraining the utility model and works across multiple tasks.

2. Three training objectives for latent anonymization: a clip-level self-supervised privacy budget objective (mutual information destruction between static clips), a co-training utility objective to retain performance on seen tasks, and a latent consistency loss for generalization to unseen tasks.

3. First demonstration of privacy preservation across multiple downstream tasks simultaneously, reporting a decrease in privacy leakage of over 35% while preserving performance within 1–2% across each utility task, plus ablations on data efficiency and applicability across video backbones.

4. New protocols for assessing gender bias in action recognition models, showing that self-supervised privacy preservation also mitigates gender bias without any explicit bias objective.

Main Findings

• Privacy–utility trade-off with I3D backbone: on VISPR, cMAP drops from 63.64 (raw videos) to 41.07 for the proposed method, a relative reduction of 35.5%, while Kinetics400 top-1 goes from 62.67 to 62.11 (down 0.9%), UCF101 from 90.30 to 90.14 (down 0.2%), THUMOS14 mAP from 25.29 to 24.92 (down 1.5%), and UCF-Crime AUC from 77.68 to 75.69 (down 2.6%).

• Scales to large video foundation models: with VideoMAE-B, VISPR cMAP falls from 70.47 to 49.92 (down 28.9%) with Kinetics400 74.86 to 74.23, UCF101 96.80 to 96.11, THUMOS14 60.82 to 60.50, and UCF-Crime 85.79 to 85.08. With VJEPA-H, VISPR cMAP falls from 72.44 to 51.42 (down 29.0%) with Kinetics400 77.03 to 76.62, UCF101 97.67 to 97.54, THUMOS14 66.66 to 66.30, and UCF-Crime 85.79 to 84.81.

• Matches supervised privacy training without privacy labels: on VP-UCF101, the method reaches 96.8 top-1 accuracy and 69.6 cMAP, comparable to a supervised baseline at 96.5 top-1 and 69.5 cMAP (raw videos: 96.8 top-1, 75.9 cMAP). On VP-HMDB51 it reaches 72.1 top-1 and 70.5 cMAP versus supervised 72.4 and 70.4 (raw: 72.6 and 76.4).

• Gender bias is reduced without a bias objective: on the NTU-Bias-F protocol, the baseline subclass accuracy gap of 9.42% falls to 5.44%, a relative reduction of 42.3%. On NTU-Bias-M the gap falls from 5.00% to 4.03% with overall accuracy rising from 52.78% to 53.06%. On Toyota Smarthome, the naturally occurring gap falls from 5.75% to 3.48% (a relative reduction of 39.5%) while overall accuracy improves from 67.02% to 67.64%.

• Data efficiency: an anonymizer trained only on HMDB51 still generalizes across tasks (VISPR cMAP 54.35, K400 74.55, UCF101 96.56, HMDB51 73.92, ToyotaSH 65.82, UCF-Crime 84.52, THUMOS14 56.50), versus raw data at 70.47, 74.86, 96.80, 72.94, 65.05, 85.79, 60.82. Training on Toyota Smarthome shows the weakest transfer to THUMOS14 (41.30 mAP).

• Generalization to unseen tasks: training with action detection only (row c of the task ablation) yields VISPR 50.17, K400 73.86, THUMOS14 58.80, UCF-Crime 83.67 — staying within 1.3% of the non-anonymized score on the tasks it never saw. Training on all three tasks gives VISPR 49.92, K400 74.23, THUMOS14 60.50, UCF-Crime 85.08.

• Each loss component is necessary: removing the utility loss drops HMDB51 accuracy to 4.71 and THUMOS14 mAP to 1.52; removing the privacy budget objective gives essentially no privacy gain over the baseline (70.44 cMAP vs 70.47); removing latent consistency preserves seen-task performance (HMDB51 72.88) but harms unseen tasks (THUMOS14 mAP 3.81 vs 56.50 with all losses).

• Latent consistency controls temporal private attributes: on Casia-B gait recognition, the baseline scores 69.73; with latent consistency the score is 53.45, and without it 26.67 — meaning removing latent consistency more strongly suppresses a motion-based sensitive attribute, but risks overfitting.

• Robustness to black-box inversion attacks: in the black-box setting, action performance is partly recovered on VP-HMDB51 (67.3 top-1 versus 72.1 for the unattacked anonymizer) while attribute prediction stays at random-chance cMAP (58.8, with random chance reported as 58.3). The authors state the model is not robust to white-box attacks (VP-HMDB51 cMAP 73.7, VP-UCF101 cMAP 72.6), which they describe as typically unrealistic in deployment.

Methodology in Plain English

The starting point is an off-the-shelf video encoder whose weights are never updated. A lightweight Anonymizing Adapter Module — a multi-head self-attention transformer encoder — sits on top of the encoder's clip-level embedding (the [CLS] token for transformer encoders, or the average-pooled feature for CNN encoders) and outputs a modified embedding of the same shape. The adapter is initialized as an identity function, and the task classifier heads are first trained on non-anonymized features for stability.

Training then runs an adversarial game with three signals. First, two random frames from each clip are tiled into static clips and pushed through the same frozen encoder; the similarity between those two clips is maximized rather than minimized (the reverse of the usual SimCLR NT-Xent objective), which destroys the spatial information that the two frames share — this is the privacy budget loss and it needs no private attribute labels. Second, gradients from the utility heads (a linear layer for action recognition, TriDet for temporal action detection, MGFN for anomaly detection) are back-propagated through the adapter so it learns to keep whatever temporal information the tasks need. Third, a latent consistency loss penalizes the squared L2 distance between the original and anonymized embeddings, stopping the adapter from drifting into a space overfit to the training tasks.

The intuition for the design is that an anonymizer operating on 3D clip features can reason across time, unlike prior 2D frame-level U-Nets, so it can destroy spatial detail while leaving the temporal signature the utility tasks depend on. Because only the small adapter is trained, features do not need to be re-extracted and the large encoder does not need retraining. All privacy, utility, and bias evaluations then reuse the anonymized embeddings with frozen downstream models.

Why This Matters

Impact on research: The paper reframes visual privacy from an input-transformation problem to a representation-learning problem. That shift matters because modern video foundation models are trained on millions of videos with specific recipes and cannot practically be retrained for each privacy regime. The paper also argues that privacy preservation can act as a form of debiasing, echoing earlier work on debiasing–privacy connections, and extends bias study into the video domain where human-attribute bias has been under-examined.

Real-world applications:

• Healthcare and patient monitoring, where video features are extracted and stored for multiple analysis tasks but must not expose patient appearance or identity.
• Surveillance and public safety, where long-video anomaly detection pipelines keep feature archives that would otherwise allow attribute inference about people in the footage.
• Sports analytics, where broadcast footage is processed for action and event detection while protecting athletes' private attributes.
• Robotics and smart-home assistance, including elder-care settings like Toyota Smarthome, where daily-activity models risk encoding demographic information about residents.
• Video search and archival, where latent features are the stored artifact and are shared or reused across teams.

Industry relevance: The plug-and-play design means organizations can apply the adapter to existing frozen backbones including large foundation models, avoiding the compute cost of full model retraining and feature re-extraction. The results also show the method works on Kinetics-pretrained I3D, VideoMAE-B, and VJEPA-H, which matters for teams deploying heterogeneous model families. Robustness to black-box attacks, and explicit acknowledgment of white-box vulnerability, gives a realistic deployment picture.

Future Directions

1. Defense against white-box inversion attacks. The authors explicitly state the model is not robust to white-box attacks, where the attacker has full access to the anonymizer and performs gradient-based reconstruction. They name this a promising avenue for future work.

2. Extending temporal sensitive attribute handling. Latent consistency does not suppress motion-based attributes like gait (Casia-B score of 53.45 with latent consistency). The authors suggest that if task overfitting is not a concern, dropping latent consistency defends against temporal attribute recognition, or a temporal sensitive attribute task can be additionally included in the budget privacy loss.

3. Broadening task coverage and evaluation. The paper reports the framework evaluates across five distinct tasks and mentions additional generalization results on video retrieval in the appendix, suggesting further expansion to more downstream video understanding tasks and larger-scale anonymizer pretraining.

4. Addressing the limitations of the latent formulation. Because the supplied text is truncated mid-sentence at the Limitations paragraph, the complete set of acknowledged constraints is not reported here; the visible portion indicates the framework cannot mitigate a threat described at that point in the text.

Target Audience

Researchers and practitioners working on video privacy, trustworthy machine learning, and video foundation models will get the most from this paper, particularly those who need to store or share extracted video features rather than raw pixels. It is also directly relevant to fairness researchers studying human-attribute bias in action recognition, to engineers deploying multi-task video pipelines where retraining an encoder is infeasible, and to readers already comfortable with self-supervised contrastive objectives, adapter modules, and adversarial training. Readers seeking an introduction to video privacy or to the underlying video architectures will find the paper assumes substantial prior background.

Authors’ abstract

We introduce a novel formulation of visual privacy preservation for video foundation models that operates entirely in the latent space. While spatio-temporal features learned by foundation models have deepened general understanding of video content, sharing or storing these extracted visual features for downstream tasks inadvertently reveals sensitive personal information like skin color, gender, or clothing. Current privacy preservation methods focus on input-pixel-level anonymization, which requires retraining the entire utility video model and results in task-specific anonymization, making them unsuitable for recent video foundational models. To address these challenges, we introduce a lightweight Anonymizing Adapter Module (AAM) that removes private information from video features while retaining general task utility. AAM can be applied in a plug-and-play fashion to frozen video encoders, minimizing the computational burden of finetuning and re-extracting features. Our framework employs three newly designed training objectives: (1) a clip-level self-supervised privacy objective to reduce mutual information between static clips, (2) a co-training objective to retain utility across seen tasks, and (3) a latent consistency loss for generalization on unseen tasks. Our extensive evaluations demonstrate a significant 35% reduction in privacy leakage while maintaining near-baseline utility performance across various downstream tasks: Action Recognition (Kinetics400, UCF101, HMDB51), Temporal Action Detection (THUMOS14), and Anomaly Detection (UCF-Crime). We also provide an analysis on anonymization for sensitive temporal attribute recognition. Additionally, we propose new protocols for assessing gender bias in action recognition models, showing that our method effectively mitigates such biases and promotes more equitable video understanding. https://joefioresi718.github.io/SPLAVU_webpage/