Enhancing Noise Resilience in Face Clustering via Sparse Differential Transformer

AI summary

Overview

• Research area: Computer vision / graph-based face clustering, specifically similarity measurement and attention-based neighbor relationship prediction.
• Technical level: Advanced (the paper assumes familiarity with Graph Convolutional Networks, Jaccard similarity, Transformer attention, and mixture-of-experts designs).
• Scope: The paper proposes a face clustering pipeline that replaces cosine-distance kNN graphs with a prediction-driven Top-K Jaccard similarity coefficient and a Sparse Differential Transformer with Top-K sparse attention, evaluated on MS-Celeb-1M, MSMT17, and DeepFashion.

What This Paper Is About

Face clustering groups face images of the same identity together. Prior methods such as Ada-NETS and FC-ESER replaced cosine distance with the Jaccard similarity coefficient to compare faces, but they pull in many irrelevant neighboring nodes, which makes the coefficients of different faces look very similar and hard to distinguish. This paper's goal is to make that similarity measure purer and more discriminative — first by predicting how many neighbors (Top-K) actually matter, and then by using a noise-suppressing attention network to examine relationships near that Top-K boundary.

Key Contributions

1. A prediction-driven Top-K Jaccard similarity coefficient that improves the purity of neighboring nodes and thus the reliability of similarity measurements.
2. A neural network-based approach for predicting neighbor relationships near the Top-K, described as a more reliable strategy for adaptive neighbor discovery.
3. A Sparse Differential Transformer (SDT) network that addresses the noise-attention problem in the vanilla Transformer, plus a Mixture of Experts variant (MoE-SDT) that uses attention masks before and after Top-K.
4. Comprehensive experiments on multiple datasets showing state-of-the-art (SOTA) performance in both clustering accuracy and robustness.

Main Findings

• SOTA on MS1M across scales: The method ("Diff-Cluster (Our)") reports the best results from 584K to 5.21M unlabeled test images. At 584K it reaches F_P = 95.46 and F_B = 94.14, versus FC-ESER at F_P = 95.28 and F_B = 93.85. At the largest scale (5.21M) it reports F_P = 90.08 and F_B = 89.14, versus FC-ESER at 89.40 and 88.80.
• Stability across scale: The method drops only from F_P = 95.46 at 584K to 90.08 at 5.21M, which the authors attribute to their adaptive neighbor discovery strategy.
• Distance metric matters: The sigmoid-based metric (Eq. 6) outperforms the FC-ES baseline (Eq. 4) by 0.18% in F_P and 0.16% in F_B. The optimal setting is δ = 7.5 (F_P = 94.67, F_B = 93.18, NMI = 0.9820); δ = 5.0 under-amplifies (F_P = 94.09, F_B = 92.29) and δ = 10.0 over-amplifies (F_P = 94.45, F_B = 92.85). Adding the Top-K step on top of Eq. 6 yields F_P = 95.02, F_B = 93.48, NMI = 0.9820.
• Differential attention beats vanilla attention: Vanilla Transformer scores F_P = 94.25, F_B = 92.73, NMI = 0.9805; Vanilla + Top-K reaches 94.78 / 93.25 / 0.9818; Diff Transformer reaches 95.05 / 93.59 / 0.9830; Diff + Top-K (SDT) reaches 95.34 / 93.93 / 0.9839; Diff + MoE Top-K (MoE-SDT) reaches 95.46 / 94.14 / 0.9840.
• Robust to hyperparameters: MoE-SDT varies only from F_P = 95.34 to 95.46 as the score threshold η moves from 0.5 to 0.9 at K = 80, and K = 40 and K = 160 both give F_P = 95.32 and 95.34 respectively.
• Noise robustness: At noise ratios of 10%, 20% and 40% on MS1M 584K, the Diff Transformer's F_P, F_B and NMI show a slight upward trend, while the Vanilla Transformer's performance decreases significantly. The authors suggest the added noise may make some same-identity faces that were under-recognized (pose, expression, illumination, age) easier to cluster.
• Generalizes beyond faces: On MSMT17, the method achieves F_P = 70.83 and F_B = 75.73, compared with FC-ESER at 70.71 / 75.66.
• Improves an existing method: Plugging SDT into LCEPCE raises MS1M 584K from F_P = 93.43 and F_B = 92.44 (vanilla) to 94.68 and 93.31, and DeepFashion from 38.43 / 60.55 to 41.76 / 63.93.

Methodology in Plain English

The pipeline has four stages. First, a face graph is built: each image is encoded as a feature vector, cosine similarity is computed between all pairs, and the K nearest neighbors form a similarity matrix and adjacency matrix. Second, instead of using all K neighbors when computing Jaccard similarity, the method predicts the optimal neighbor count (Top-K) with a Transformer-based adaptive filter (replacing Ada-NETS's LSTM), and only nodes before Top-K are used, which cuts down irrelevant nodes. The predicted number is also rounded with a rule (multiples of 10, capped at K) to make it steadier. Third, because the predicted Top-K can still be slightly off, a Sparse Differential Transformer examines the central node's relationships with neighbors near that boundary. It adopts the Differential Transformer's trick of subtracting one softmax attention map from another to cancel attention noise, and adds a Top-K attention mask so only relevant nodes are attended to. A Mixture of Experts version (MoE-SDT) applies masks at Top-(K−u), Top-K and Top-(K+u) with u = 5 and learnable weights α, β and γ, which handles the case where the true optimum sits just outside the predicted cut. The distance transform was also changed from an exponential decay to a sigmoid-like function with δ = 7.5 and ε = −5, amplifying small distance differences. Finally, the Map Equation method performs the actual clustering. Training details: 3 layers, 8 attention heads, hidden dimension 1024, learning rate 1e-2, SGD with momentum 0.9 and weight decay 1e-4, λ_init = 0.8, K = 80 for MS1M and K = 40 for MSMT17, and score threshold η = 0.90 for MS1M and 0.88 for MSMT17. MS-Celeb-1M contains 5.8M images of 86K identities after cleaning, with training on part 0 and evaluation on parts 1, 3, 5, 7 and 9.

Why This Matters

• Research impact: The work shows that attention-level denoising, not just graph construction, is a lever for face clustering quality, and that the differential attention idea from language modeling transfers to vision clustering. It also provides a drop-in improvement for existing pipelines, as demonstrated by the LCEPCE experiment.
• Real-world applications:
  • Photo management and personal photo library organization by person.
  • Social media organization and tagging of user-uploaded images.
  • Data collection and cleaning for training face recognition models.
  • Person re-identification and clothing retrieval, as tested on MSMT17 and DeepFashion.
• Industry relevance: The authors are at Samsung R&D Institute China-Beijing, and the method is designed to scale: it keeps high accuracy even at 5.21M unlabeled test images and tolerates a range of K and η settings, which reduces tuning cost in production deployments.

Future Directions

• Predicting the optimal Top-K remains difficult; the paper notes the prediction frequently deviates from the true optimum, motivating MoE-SDT, but does not solve the prediction itself.
• The observation that adding random noise slightly improved Diff Transformer results on MS1M is offered as a hypothesis about under-recognized same-identity faces, and the paper does not investigate it further.
• The MoE-SDT introduces extra learnable weights (α, β, γ) and three mask scales with u = 5; how these should generalize to other datasets is not established.
• The method is tested on MS1M, MSMT17 and DeepFashion, all image-grouping domains; extension to other clustering problems is left open.

Target Audience

Researchers and engineers working on face clustering, graph-based representation learning, or attention mechanism design. It is most useful to readers who are already comfortable with GCN pipelines, kNN graph construction, and Transformer internals, and to practitioners who need a scalable clustering method that degrades gracefully with dataset size and noise.

Authors’ abstract

The method used to measure relationships between face embeddings plays a crucial role in determining the performance of face clustering. Existing methods employ the Jaccard similarity coefficient instead of the cosine distance to enhance the measurement accuracy. However, these methods introduce too many irrelevant nodes, producing Jaccard coefficients with limited discriminative power and adversely affecting clustering performance. To address this issue, we propose a prediction-driven Top-K Jaccard similarity coefficient that enhances the purity of neighboring nodes, thereby improving the reliability of similarity measurements. Nevertheless, accurately predicting the optimal number of neighbors (Top-K) remains challenging, leading to suboptimal clustering results. To overcome this limitation, we develop a Transformer-based prediction model that examines the relationships between the central node and its neighboring nodes near the Top-K to further enhance the reliability of similarity estimation. However, vanilla Transformer, when applied to predict relationships between nodes, often introduces noise due to their overemphasis on irrelevant feature relationships. To address these challenges, we propose a Sparse Differential Transformer (SDT), instead of the vanilla Transformer, to eliminate noise and enhance the model's anti-noise capabilities. Extensive experiments on multiple datasets, such as MS-Celeb-1M, demonstrate that our approach achieves state-of-the-art (SOTA) performance, outperforming existing methods and providing a more robust solution for face clustering.