Cooperative Pseudo Labeling for Unsupervised Federated Classification

AI summary

Overview

Research area: Unsupervised federated learning (UFL) combined with vision-language models (CLIP) for image classification.

Technical level: Advanced. The paper assumes familiarity with federated learning, contrastive language-image pre-training, prompt tuning, and pseudo labeling.

Scope: The paper proposes FedCoPL, a federated method that lets clients with no labels and no shared data train a CLIP-based classifier using server-coordinated pseudo labels and partial (visual-only) prompt aggregation.

What This Paper Is About

Unsupervised federated learning normally stops at representation learning or clustering, because without labels clients cannot directly train a classifier. This paper extends UFL to classification for the first time by initializing every client with the same pre-trained CLIP model and using CLIP's zero-shot predictions to create training labels from unlabeled local data. The goal is to keep that pseudo labeling balanced and accurate even though CLIP has an inherent prediction bias and the unlabeled data on each client may be skewed toward only some classes.

Key Contributions

1. The paper extends the unsupervised federated learning paradigm to classification tasks using CLIP, a setting the authors state remains largely unexplored in previous work.
2. It proposes FedCoPL, built around a cooperative pseudo labeling strategy in which clients upload estimated pseudo label distributions and the server adjusts and redistributes them to avoid global class imbalance and to mitigate CLIP's inherent bias.
3. It introduces a partial prompt aggregation protocol: visual prompts are uploaded and aggregated at the server, while textual prompts stay local for personalization, based on the observation that textual prompts diverge far more across clients than visual prompts.
4. It reports experiments on six datasets (DTD, RESISC45, CUB, UCF101, CIFAR10, CIFAR100) under both Dirichlet-based and quantity-based label skews, plus ablations of each component.

Main Findings

• Large gains under Dirichlet skew with ViT-B/32 (Table 1): FedCoPL reaches averages of 72.49 (β = 0.1) and 74.06 (β = 0.05), compared to 60.19 for zero-shot CLIP and 63.06 / 62.84 for the strongest baseline (FedOPT with FPL). Per-dataset results at β = 0.1 / β = 0.05 are DTD 60.89 / 66.37, RESISC45 75.76 / 80.26, CUB 56.09 / 54.80, UCF101 73.20 / 74.97, CIFAR10 95.38 / 95.11, CIFAR100 73.59 / 72.84.

• Large gains under quantity-based skew (Table 2, s = C × 0.2): FedCoPL averages 72.31 overall, with DTD 56.18, RESISC45 81.06, CUB 56.31, UCF101 72.03, CIFAR10 94.86 and CIFAR100 73.39. The best baseline average in this table is 63.55 (pFedPrompt with CPL).

• Baselines can fall below zero-shot: With quantity-based skew, FedOPT combined with CPL performs 7.42% and 7.28% worse than zero-shot accuracy on RESISC45 and CUB respectively. Under Dirichlet skew, all baseline methods perform worse than zero-shot on CUB.

• The pseudo labeling strategy transfers to other federated trainers (Table 3): Swapping in cooperative pseudo labeling under Dirichlet skew (β = 0.1) raises PromptFL by +1.55 (DTD), +1.92 (RESISC45), +1.79 (CUB), +0.58 (UCF101) and +2.69 (CIFAR10); pFedPrompt by +4.69, +2.02, +2.29, +3.56 and +3.68; and FedOPT by +5.96, +17.09, +7.73, +7.38 and +2.70. The largest single gain is 17.09% on RESISC45.

• Results hold with a stronger backbone (Table 4, ViT-B/16, β = 0.1): FedCoPL scores DTD 56.83, RESISC45 77.50, CUB 59.78 and UCF101 78.29, versus zero-shot CLIP at 42.87, 56.61, 55.16 and 65.13 and versus the best listed baseline per dataset among PromptFL, PromptProx, pFedPrompt and FedOPT.

• Ablation of the labeling components (Table 5, Dirichlet skew): Adding global allocation alone gives DTD 46.35, RESISC45 69.60, CUB 51.98 and UCF101 65.31; entropy filtering plus global allocation gives 58.83, 72.28, 54.91 and 70.23; confidence filtering plus global allocation gives 55.59, 73.66, 50.38 and 69.13; using both filters with global allocation gives the best scores of 60.89, 75.76, 56.09 and 73.20.

• Ablation of prompt aggregation (Table 6, β = 0.1): Aggregating both visual and textual prompts performs worst at DTD 47.34, RESISC45 61.68, CUB 49.08 and UCF101 64.97, below aggregating nothing (50.45, 61.72, 49.08, 66.02). Textual-only aggregation gives 55.31, 67.12, 51.76 and 69.53. Visual-only aggregation — the proposed protocol — is best at 60.89, 75.76, 56.09 and 73.20.

• Robustness to client count: With DTD, RESISC45 and CIFAR partitioned among 5, 10, 30 and 50 clients, baseline methods decline significantly as the client count grows while FedCoPL maintains high accuracy.

• Robustness to partial participation: Across client joining rates of 0.5, 0.8 and 1.0, FedCoPL outperforms competing methods, though all methods degrade as the participation rate falls.

• Prompt divergence justification (Figure 2): Measured with drift diversity and cosine distance on CIFAR10, textual prompt differences across clients are significantly greater than visual prompt differences, supporting the decision to aggregate only visual prompts.

Methodology in Plain English

Each client starts from the same frozen CLIP model and holds only unlabeled images plus the list of class names. Training optimizes two small sets of learnable prompt tokens rather than the full model: visual prompts and textual prompts.

Pseudo labels are created in a cooperative loop that repeats every few communication rounds. A client first filters its own unlabeled samples using two criteria — prediction confidence and prediction entropy — keeping only samples whose maximum class probability exceeds the median confidence and whose entropy falls below the median entropy. The client then counts how many kept samples fall into each class and sends only that count vector to the server.

The server assumes a uniform joint label distribution across all clients and sets a global budget of pseudo labels per class, computed as the total number of counted pseudo labels divided by the number of classes. Each client's budget for a class is proportional to its share of the counts for that class, taken over all clients, rather than to the raw counts themselves. Clients then keep, for each class, the assigned number of highest-confidence samples from their original unlabeled data to form the training set. Because the allocation uses relative shares, noisy client estimates still produce representative and accurate pseudo labels.

For collaboration, clients upload only their visual prompts. The server averages them with weights proportional to each client's assigned sample count and sends the result back as the next round's initialization. Textual prompts are never uploaded, so each client keeps a personalized classifier head that matches its own label distribution. The paper states that transmitting only one modality reduces communication overhead compared to aggregating both. The final objective is the sum, over clients, of the cross-entropy loss between the model output and the pseudo labels on the reconstructed training set.

The paper also notes a privacy property: uploaded estimated distributions come only from CLIP's predictions, so the exact label distributions of clients cannot be inversely inferred, unlike methods that transmit exact category distributions.

Why This Matters

Impact on research. This is presented as the first extension of unsupervised federated learning to classification via CLIP, opening a direction that earlier UFL work on representation learning and clustering could not support. The paper also shows the labeling contribution is portable: it improves PromptFL, pFedPrompt and FedOPT when dropped into them, which points to pseudo label quality, not a specific aggregation scheme, as a key bottleneck.

Real-world applications.

• Federated medical or clinical image classification, where institutions hold unlabeled scans and cannot share patient data or afford annotation.
• Remote sensing and satellite imagery analysis on distributed data sources, relevant to the RESISC45 dataset used here.
• On-device photo and video categorization, relevant to the CIFAR10, CIFAR100 and UCF101 benchmarks, where labels are unavailable and bandwidth is limited.
• Camera-trap and biodiversity monitoring, relevant to the CUB fine-grained bird dataset, where species labels are scarce across field sites.

Industry relevance. Because the method only tunes prompts rather than full model parameters and only transmits one modality of prompts, it suits cross-device settings with constrained communication and limited client compute. The evaluation with ViT-B/32 and ViT-B/16 backbones, and the stated inclusion of RN50 results in the supplementary material, speaks directly to deployments on modest hardware; experiments ran on NVIDIA 3090 GPUs. The method needs only class names on each client and no labels, which lowers the data annotation cost that the paper identifies as a barrier in practice.

Experimental setup for reference: default backbone ViT-B/32; K = 5 clients for CUB and UCF101 and 10 for other datasets; full client participation by default; 20 communication rounds; pseudo labels updated every 5 rounds using the latest local model; 10 local epochs per round; SGD with learning rate 0.1, momentum 0.9 and cosine annealing decay; three trials per setting with mean accuracy reported; PyTorch implementation.

Future Directions

1. A theoretical analysis of FedCoPL, which the authors explicitly name as future work, covering convergence, privacy and fairness.
2. Testing whether cooperative pseudo labeling helps under stronger forms of heterogeneity than the Dirichlet-based and quantity-based skews studied here, since performance of all methods degrades as the client participation rate drops.
3. Examining scalability beyond the tested client counts, since the paper evaluates 5, 10, 30 and 50 clients but not larger federations.
4. Extending the partial prompt aggregation idea to other vision-language backbones and tasks, given that the paper only reports main-text results for ViT-B/32 and ViT-B/16, with RN50 left to the supplementary material.

Target Audience

Researchers and practitioners working on federated learning, unsupervised or label-free learning, and parameter-efficient adaptation of vision-language models. It is most useful to readers already comfortable with CLIP prompt tuning and non-IID federated settings, and to engineers designing privacy-preserving, communication-constrained distributed classification systems where client data is unlabeled.

Authors’ abstract

Unsupervised Federated Learning (UFL) aims to collaboratively train a global model across distributed clients without sharing data or accessing label information. Previous UFL works have predominantly focused on representation learning and clustering tasks. Recently, vision language models (e.g., CLIP) have gained significant attention for their powerful zero-shot prediction capabilities. Leveraging this advancement, classification problems that were previously infeasible under the UFL paradigm now present promising new opportunities, yet remain largely unexplored. In this paper, we extend UFL to the classification problem with CLIP for the first time and propose a novel method, \underline{\textbf{Fed}}erated \underline{\textbf{Co}}operative \underline{\textbf{P}}seudo \underline{\textbf{L}}abeling (\textbf{FedCoPL}). Specifically, clients estimate and upload their pseudo label distribution, and the server adjusts and redistributes them to avoid global imbalance among classes. Moreover, we introduce a partial prompt aggregation protocol for effective collaboration and personalization. In particular, visual prompts containing general image features are aggregated at the server, while text prompts encoding personalized knowledge are retained locally. Extensive experiments demonstrate the superior performance of our FedCoPL compared to baseline methods. Our code is available at \href{https://github.com/krumpguo/FedCoPL}{https://github.com/krumpguo/FedCoPL}.