TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Models

AI summary

Overview

• Research area: Federated learning and efficient adaptation of pre-trained vision-language models (VLMs), specifically one-shot, training-free methods.
• Technical level: Advanced. The paper combines hierarchical Bayesian modeling, Gaussian Discriminant Analysis, LLM-based prompt augmentation, and a generalization-bound argument for multimodal fusion.
• Scope: The paper proposes TOFA, a framework that adapts CLIP-style VLMs across federated clients in a single communication round without any gradient-based training on the client or server.

What This Paper Is About

Adapting large pre-trained VLMs to a client's downstream task usually requires many rounds of communication between clients and a server, which is expensive and exposes the system to attacks. The authors ask whether the whole adaptation can be done in one shot and without training anything, while still handling the severe non-IID data distributions that make federated learning hard. TOFA answers this by combining a Bayesian visual pipeline with a globally aligned textual pipeline and fusing their predictions per sample.

Key Contributions

1. The authors state they are the first to propose a training-free one-shot adaptation method for VLMs in the federated learning setting.
2. An one-shot visual pipeline that learns personalized, class-specific prompt (prototype) distributions over visual representations using a hierarchical Bayesian model with global information as the prior.
3. A textual pipeline that extracts robust text augmentations through global alignment of client-side importance scores for LLM-generated prompts.
4. A sample-adaptive modality weight calibration mechanism that fuses personalized visual representations with robust text representations to handle data heterogeneity, validated by extensive experiments across nine datasets and various federated settings.

Main Findings

• Few-shot CLIP datasets (16-shot, 10 clients): TOFA reaches 91.23 on OxfordPets, 95.78 on Flowers102, 85.49 on Food101, 94.58 on Caltech101, and 71.68 on DTD. It consistently beats all one-shot baselines on all five datasets and exceeds CoOp, PromptFolio, and PromptFL on all except Food101.
• Hardest dataset (DTD): On DTD, where the authors note training-free methods struggle, TOFA (71.68) still outperforms every one-shot baseline and shows only a slight gap to multi-round methods such as PromptFolio (65.04) and PromptFL (50.46).
• CIFAR under Dirichlet heterogeneity: With 100 clients and Dir(β = 0.3) partitioning, TOFA scores 93.18 on CIFAR10 and 76.63 on CIFAR100, compared with FedAvg (75.10 / 42.52), Zero-Shot CLIP (87.71 / 64.92), CoOp (93.11 / 74.83), and PromptFL (92.30 / 73.67).
• Feature shift benchmarks: On DomainNet (6 clients, one domain each) TOFA averages 93.05, and on Office-Caltech10 (4 clients) it averages 98.69. This surpasses most methods requiring multiple rounds of training and lands within 2% of the optimal prompt-based FL baseline.
• Training-free baselines struggle on domain shift: Zero-Shot CLIP averages 50.50 on DomainNet and 19.50 on Office-Caltech10, and CLIP-GDA averages 66.71 and 98.15 respectively, showing that existing training-free methods do not hold up under feature shift.
• Effect of the global-information coefficient α: Testing α from 0 to 1 across the CLIP datasets shows the optimal value varies by dataset, but weighting global information more heavily (α ≥ 0.75) achieves near-optimal performance. The authors adopt α = 1 for the reported experiments.
• Effect of shot count: Varying shots over [1, 2, 4, 8, 16] shows accuracy improving with more shots, with stable results starting from the 8-shot setting.
• Inter-modality ablation: The visual-only and text-only variants both score lower than the fused TOFA model, which the authors attribute to the fusion preventing overfitting on local data. The paper presents this as a figure and does not report the individual numbers in the text.

Methodology in Plain English

TOFA treats the pre-trained CLIP model as frozen and derives its predictions from statistics rather than training.

1. Visual pipeline (personalization). Each client encodes its local images with CLIP's visual encoder and summarizes them as per-class statistics. The server combines these into global class-specific prototype distributions using a hierarchical Bayesian model with an uninformative prior. It sends the global prototypes back, and each client uses them as an informative prior to derive its own personalized class prototypes. The paper shows this has a closed-form posterior (Inverse-Wishart for the shared covariance, Gaussian for the class means), so no iterative optimization is needed. Classification then uses Gaussian Discriminant Analysis over the resulting personalized distributions. A scalar α ∈ [0, 1] controls how strongly the global likelihood influences the local posterior.
2. Textual pipeline (robustness). Each client uses a local LLM to generate dataset-aware, class-specific descriptions, alongside the manual prompt "A photo of a class". Each client scores how well each augmented prompt separates its own class from the others. The server then computes an importance score for each prompt by comparing it against the manual prompt across clients, similar in spirit to a KL divergence, and turns these scores into weights via a softmax with temperature τ_t = 0.5.
3. Adaptive multimodal fusion. For each image, a mixing coefficient η(z), computed from the softmax confidences of the visual and textual predictors, decides how much weight each modality receives. A theorem in the paper bounds the generalization error of such a fused classifier and shows that the error is minimized when the mixing weight tracks which modality is more accurate for that sample.
4. One round, no training. Only aggregated statistics and prompt scores travel between clients and server, in a single exchange. The method requires no gradient-based model optimization and no additional training resources on either side.

Why This Matters

• Research impact: The paper opens a direction — training-free, one-shot federated VLM adaptation — that the authors state had no prior work. It shows that a Bayesian posterior formulation plus prompt scoring can substitute for the multi-round prompt-learning pipelines that dominate federated VLM research.
• Real-world applications:
  • Personalizing image classification on mobile devices where users cannot afford local training or repeated uploads.
  • Cross-silo medical or enterprise image classification where a single communication round limits exposure and communication cost.
  • Multi-domain deployment (e.g., sketches, paintings, photos) where each client covers a different visual domain, as in the DomainNet and Office-Caltech10 experiments.
  • Environments with capability-limited servers that cannot run federated training jobs.
• Industry relevance: Lower communication and no training overhead make VLM adaptation more practical for resource-constrained fleets, and limiting client-server exchange to one round reduces the attack surface that the authors cite as a drawback of iterative federated training.

Future Directions

• The paper does not explore the effect of the number of generated text prompts M beyond the LLM augmentation step, nor how the quality of the local LLM affects robustness.
• The privacy analysis is deferred to Appendix B.1, which is not included in the provided content, so the formal privacy guarantee of transmitting only statistics remains an open point for readers of the main text alone.
• Since results are deterministic and the authors explicitly do not report statistical variation, the stability of the differences against the strongest multi-round baselines across repeated partitions is not established.
• Extending the framework beyond classification to other VLM tasks, and testing how the adaptive fusion weight behaves when one modality is badly miscalibrated, are natural next steps.

Target Audience

Researchers and practitioners working on federated learning, parameter-efficient VLM adaptation, or prompt learning, especially those interested in reducing communication rounds and removing training requirements. It is also relevant to engineers deploying CLIP-style models on resource-constrained client fleets who need a practical, single-round alternative to iterative federated fine-tuning.

Authors’ abstract

Efficient and lightweight adaptation of pre-trained Vision-Language Models (VLMs) to downstream tasks through collaborative interactions between local clients and a central server is a rapidly emerging research topic in federated learning. Existing adaptation algorithms are typically trained iteratively, which incur significant communication costs and increase the susceptibility to potential attacks. Motivated by the one-shot federated training techniques that reduce client-server exchanges to a single round, developing a lightweight one-shot federated VLM adaptation method to alleviate these issues is particularly attractive. However, current one-shot approaches face certain challenges in adapting VLMs within federated settings: (1) insufficient exploitation of the rich multimodal information inherent in VLMs; (2) lack of specialized adaptation strategies to systematically handle the severe data heterogeneity; and (3) requiring additional training resource of clients or server. To bridge these gaps, we propose a novel Training-free One-shot Federated Adaptation framework for VLMs, named TOFA. To fully leverage the generalizable multimodal features in pre-trained VLMs, TOFA employs both visual and textual pipelines to extract task-relevant representations. In the visual pipeline, a hierarchical Bayesian model learns personalized, class-specific prototype distributions. For the textual pipeline, TOFA evaluates and globally aligns the generated local text prompts for robustness. An adaptive weight calibration mechanism is also introduced to combine predictions from both modalities, balancing personalization and robustness to handle data heterogeneity. Our method is training-free, not relying on additional training resources on either the client or server side. Extensive experiments across 9 datasets in various federated settings demonstrate the effectiveness of the proposed TOFA method.