BLEND: Balancing Personalization vs. Generalization in Federated Vision-Language Models

AI summary

Overview

Research area: Federated learning applied to vision-language models (specifically CLIP), with a focus on parameter-efficient fine-tuning and the trade-off between adapting to each client's local data and preserving the ability to recognize unseen classes.

Technical level: Intermediate. The paper assumes working familiarity with CLIP's dual-encoder architecture, adapter/low-rank fine-tuning, contrastive similarity-based classification, and the standard federated learning loop of local SGD plus periodic server aggregation.

Scope in one sentence: The paper proposes BLEND, an architecture-and-loss design for federated CLIP fine-tuning that splits trainable parameters into globally aggregated vision/text adapters and a client-local vision projection adapter, then blends their outputs through a fusion loss and an anchor loss to balance personalization against generalization.

What This Paper Is About

Federated vision-language models (FedVLMs) let many separate clients collaboratively fine-tune a shared VLM such as CLIP without pooling their data. The central problem is that pulling every client toward a shared global model helps the model recognize classes it has never seen (generalization), but it also erases the client-specific knowledge that makes the model accurate on that client's own classes (personalization). BLEND's goal is to design the fine-tuning architecture, the aggregation rule, and the loss function so that both objectives improve simultaneously rather than one being traded away for the other.

Key Contributions

1. BLEND framework for FedVLMs. A fine-tuning strategy that augments CLIP with three trainable module groups: global vision adapters, global text adapters, and a personalized vision projection adapter.

2. Selective aggregation scheme. Only the global vision and text adapters are transmitted to the server and aggregated across clients; the personalized projection adapter stays local on each client and is never communicated. This creates an explicit split between aggregatable parameters and personalized/non-aggregatable parameters.

3. Dual-objective loss. A representation fusion loss that optimally combines the generalized and personalized vision embeddings, plus a running anchor loss that constrains the generalized vision representation to stay aligned with the text representation produced through the frozen zero-shot CLIP projection heads, limiting overfitting to local data.

4. Empirical validation. Extensive experiments are reported to show BLEND consistently outperforms state-of-the-art FedVLM baselines and achieves a superior balance between personalization and generalization, captured jointly by a harmonic mean (H-Mean) metric.

Main Findings

• The vision and text branches of CLIP should not be personalized equally. The paper's two guiding intuitions are that CLIP's vision branch yields lower dataset-level recognition performance than the text branch (so it carries more generalizable representations and needs local adaptation), while the text branch already exhibits higher dataset-level recognition (so further personalization risks overfitting). BLEND therefore applies asymmetric adaptation, with stronger client-specific adaptation on the visual side.

• An information-theoretic argument backs the asymmetry. Proposition 3.1 states that for a scene's latent semantics S, its visual observation V, and a textual description T generated from V, the Markov chain S → V → T holds and therefore I(S;T) ≤ I(S;V), following from the data-processing inequality (Theorem 2.8.1 in the cited reference). A text caption cannot carry more scene information than the image it came from — the paper's example is an image of a brown dog running on grass facing left with a red ball nearby versus the caption "an image of a dog."

• Two supporting empirical observations from prior work. The paper cites Schrodi et al., who identified an information imbalance in CLIP-like models where captions are sparse and images richer, and Yang et al., who observed textual representations can exhibit stronger dataset-specific behavior than vision representations.

• Global adapters alone improve generalization but leave personalization gains on the table. Adding the global vision/text adapters to the pre-trained CLIP improves generalization to unseen classes by capturing cross-device knowledge, and also yields personalization gains, but the periodic aggregation prevents them from fully preserving client-specific representations.

• The personalized projection adapter alone hurts generalization. Used in conjunction with the global adapters, the local vision projection adapter significantly improves personalization, but relying on it alone leads to a significant degradation in the model's generalization.

• BLEND achieves the best joint balance. Fusing the G-head and P-head outputs and training with the dual loss yields the highest level of personalization while simultaneously maintaining superior generalization, reflected in the highest harmonic mean (H-Mean) value among the compared fine-tuning strategies.

• Comparative baselines. BLEND is positioned against prompt-based, adapter-based, and federated VLM methods including CoOp, ProVP, MetaPrompt, TCP, MoP, DPC, Tip-Adapter, CLIP-Adapter, MMA, PromptFL, FedOTP, PromptFolio, FedPGP, FedPHA, pFedMoAP, FedDat, and TriplePlay. Among these, FedPGP, pFedMMA, and PromptFolio are identified as the few that also target both personalization and generalization.

• Reported experimental setup, not results. Experiments were run on a server with 3× L40S GPUs, 1TB of RAM, and 2× Intel 8562Y CPUs, using 10 clients following the setup of the cited reference, with the comparison averaged over 10 datasets. The specific accuracy or other performance numbers are not included in the provided paper content, which is truncated mid-way through the experiments section.

Methodology in Plain English

The starting point is a standard CLIP model: an image goes through a vision transformer, and each class label is turned into a text description that goes through a text transformer. Matching an image to a class is done by cosine similarity between the two resulting embeddings.

BLEND modifies this in three ways.

First, it inserts small trainable adapter modules into every transformer block of both the vision and text encoders. Each adapter is a bottleneck: a down-projection to a small rank, a ReLU nonlinearity, and an up-projection back to the original dimension, with the result added residually to the transformer block's output. These adapters are called global because every client trains them and the server periodically averages them across clients, letting them soak up knowledge shared across devices.

Second, it splits the vision projection head into two branches. The G-head uses CLIP's original, frozen pre-trained vision projection head on the classification token, preserving the generalizable representation. The P-head adds a small trainable low-rank adapter to that same classification token before pushing it through the same frozen projection, producing a personalized representation. Only this projection adapter is kept private to the client and never sent to the server, because image data varies far more across clients than text prompts or labels do. The final visual embedding is a weighted average of the two branches' outputs, with a trade-off parameter γ.

Third, it changes the loss. The fusion loss is the ordinary cross-entropy classification loss computed on the blended visual embedding, which pressures the two branches to combine in a way that is accurate on the client's classes. The running anchor loss is a second cross-entropy loss computed using only the generalized vision representation against the same text embeddings. Because both the G-head and the text projection head are the frozen zero-shot CLIP heads, this anchor keeps the general branch tethered to CLIP's original alignment. It is called "running" because it is not a fixed target — it shifts as the global adapters change both sides of the alignment. The total local loss is a weighted sum of the two, controlled by λ.

Training then proceeds in the usual federated way: each client runs J local SGD steps, sends only the global adapter parameters to the server, the server averages them weighted by each client's share of total data, and each client resumes from the synchronized global adapters plus its own retained personalized adapter.

Why This Matters

Impact on research. Most prior FedVLM work targets either personalization or generalization; the paper argues only a few studies (FedPGP, pFedMMA, PromptFolio) target both, and that existing approaches lean on techniques such as optimal transport and singular value decomposition. BLEND's claim to distinction is simplicity: a systematic global-local decomposition of which CLIP elements to fine-tune, which to aggregate, and which to keep local, paired with a loss function explicitly designed to exploit that decomposition. If the reported results hold, it sets a new reference point for the personalization-generalization trade-off in federated VLM fine-tuning and offers a template for asymmetric treatment of encoder branches that could transfer to other multimodal architectures.

Real-world applications.

• Privacy-sensitive multi-institution image classification, such as hospitals or clinics each holding different imaging modalities and label sets, where a shared model must still recognize rare conditions not present at any single site.
• Federated photo or content organization on personal devices, where per-device models should learn a user's own categories without losing the ability to label general objects.
• Distributed retail or manufacturing inspection, where each site has its own product catalog but the system must generalize to new products introduced later.
• Cross-organization robotics or autonomous systems, where visual conditions and object vocabularies differ by deployment site but a shared backbone must handle novel scenes.

Industry relevance. The method is parameter-efficient, since only low-rank adapter matrices are trained and only the global subset is communicated, which matters for bandwidth and on-device compute. It also slots onto an off-the-shelf CLIP checkpoint without architectural retraining from scratch, and the code is released publicly.

Future Directions

• Quantifying the trade-off knobs. The paper defers several controlled studies to ablations: the adapter ranks r_x, r_w, and r_p; which transformer layers to inject adapters into; the fusion weight γ; the anchor weight λ; and whether to fine-tune the text projection head. The full sensitivity picture is not available in the provided content.
• Scaling the federation. Client count is fixed at 10 in the main setup, with higher client counts deferred to an ablation, leaving open how BLEND behaves under much larger or highly heterogeneous client populations.
• Data heterogeneity. The related work frames FedVLMs around non-IID data, but the visible content does not report how BLEND performs as the degree of non-IID-ness varies across clients.
• Beyond CLIP. The paper explicitly adopts CLIP as the deployed VLM, following comparable works, so it is an open question whether the asymmetric vision/text personalization principle carries over to other vision-language architectures and to generative rather than contrastive objectives.

Target Audience

Researchers and practitioners working on federated learning, parameter-efficient fine-tuning, and vision-language models, particularly those interested in the personalization-versus-generalization trade-off under decentralized data. It will also be useful to engineers who want to deploy CLIP-based recognition across multiple sites or devices without sharing raw images, and to readers who want a concrete example of how an information-theoretic argument (the vision branch carries at least as much scene information as the text branch) can drive an architectural design decision.

Authors’ abstract

Federated vision-language models (FedVLMs) represent one of the emerging frontiers of machine learning, aiming to integrate federated learning within the fine-tuning pipelines of vision-language models (VLMs). In this work, we introduce BLEND, a new framework designed to jointly enhance the personalization and generalization capabilities of FedVLMs. BLEND proposes a selective VLM parameter fine-tuning and aggregation strategy that consists of (i) global vision and text adapters shared across clients, and (ii) a personalized vision projection adapter tailored to each client. BLEND further introduces a novel loss function that blends the outputs of the global and personalized adapters to achieve personalization, while simultaneously promoting generalization through a principled anchor design implemented via the projection heads of the zero-shot VLM. Extensive experiments demonstrate that BLEND consistently outperforms state-of-the-art baselines, highlighting its ability to effectively balance personalization and generalization across diverse system settings. (Code is available at: https://github.com/payamsiabd/BLEND.git)