Plug-and-Play Parameter-Efficient Tuning of Embeddings for Federated Recommendation

AI summary

Overview

Research area: Federated Recommendation (FR) — privacy-preserving, distributed training of recommender systems — with a focus on parameter-efficient fine-tuning (PEFT) of item embeddings and communication cost.

Technical level: Advanced. The paper assumes familiarity with federated aggregation, item embeddings, low-rank adaptation, vector quantization (RQ-VAE), hashing, and differential privacy (CDP/LDP).

Scope: The paper proposes a model-agnostic, plug-in framework that freezes pre-trained full item embeddings and transmits only small compressed embeddings during federated training, evaluated across four FR backbones and three public datasets.

What This Paper Is About

In Federated Recommendation, clients keep their raw data local and share only model parameters with a server. Item embeddings — which grow linearly with the number of items and often dominate the model's parameter count — therefore dominate communication cost. Existing compression approaches (low-rank factorization, hashing, quantization) reduce this cost but typically lose recommendation accuracy and become unstable across different FR models. This paper asks whether a lightweight, plug-in embedding design can cut transmitted embedding parameters while maintaining or improving accuracy across diverse FR backbones.

Key Contributions

1. A lightweight, model-agnostic FR framework that applies PEFT to item embeddings, reducing the size of updated embedding parameters throughout the FR process while improving recommendation performance.
2. Three compressed embedding strategies (LoRA, Hash, and RQ-VAE) tailored to different FR settings and datasets, two of which the authors report consistently outperform the full-embedding baseline.
3. Introduction of RQ-VAE as a novel PEFT strategy for FR, encoding items into compact codebooks plus quantized semantic codes; the authors report it achieves superior performance compared to the other PEFT approaches within the framework in many cases.
4. An accompanying open-source implementation (https://github.com/young1010/FedPEFT) and analysis of communication, storage, computation, and representation capacity, including experiments under differential privacy.

Main Findings

• PEFT beats compressed-only embeddings in most settings: Across the averaged results of five runs (Table 2), compressed embeddings alone occasionally win (for example, the best results on FedPerGNN) but degrade noticeably elsewhere, such as the PFedRec-ML1M setting. PEFT embeddings perform strongly and consistently across FedMF, FedNCF, and PFedRec; even on FedPerGNN, P-RQ-VAE outperforms full embeddings.

• Large gains on FedMF: On FedMF, P-LoRA and P-RQ-VAE substantially improve over full embeddings. As an example, on ML1M with FedMF, full embeddings reach N@10 33.98 and H@10 58.44, while P-LoRA reaches N@10 37.98 and H@10 59.79.

• RQ-VAE is stable and competitive: P-RQ-VAE delivers stable, competitive results and sometimes exceeds full embeddings — in the FedMF-Industrial setting it achieves the best overall performance (N@10 10.82, H@10 18.10, versus full embeddings at N@10 9.13, H@10 16.48).

• Strategy effectiveness varies by model and dataset: P-LoRA is robust across multiple settings and clearly leads in FedMF-ML1M, but underperforms on FedPerGNN-Software and FedPerGNN-Industrial. P-Hash strategies do best in FedNCF-Industrial and PFedRec-Industrial, especially combined with SENet, but drop notably on FedMF.

• Hash-only compression is unstable: For compressed embeddings only, C-Hash shows poor and unstable performance in most cases, which the authors attribute to its reliance on random hashing; C-LoRA and C-RQ-VAE generally outperform C-Hash but still lag behind full embeddings, with exceptions in FedMF and FedPerGNN.

• SENet helps only with an MLP backbone: Among PEFT hash variants, SENet brings gains in MLP-based models such as FedNCF and PFedRec (P-Hash(S) outperforms P-Hash), but in embedding-only models such as FedMF and FedPerGNN, SENet actually leads to a performance drop. For compressed embeddings only, SENet improves performance in most settings.

• Communication and storage characteristics differ by strategy: LoRA has no representation collision (direct learnable embeddings) but communication O(k_L·(n+k)) that scales linearly with item count; RQ-VAE achieves O(d_R·l) communication and O(d_R·l+n) storage with a representation space of (d_R)^l, though collision resistance depends heavily on server-side pre-training quality; hash methods give O(d_H) communication and O(d_H+h) storage with representation capacity C(d_H+h−1, h), but need a larger hash table d_H than RQ-VAE's d_R·l codebook budget for comparable capacity. Full embeddings cost O(k·n) communication and storage with representation n. Hash(S) adds O(h²) computation.

• All PEFT strategies cut communication, but the trade-off favors LoRA and RQ-VAE: Figure 3 shows all PEFT embedding strategies substantially reduce communication overhead versus full embeddings, with hash-based methods achieving the greatest reduction overall (though not a significant reduction relative to other strategies); considering accuracy, LoRA and RQ-VAE generally offer the more favorable trade-off in most cases.

• Moderate latent sizes work best for LoRA: For PFedRec, larger k_L increases trainable parameters and communication but does not consistently improve accuracy. On ML1M, N@10 and H@10 peak at k_L = 4 (39.48 and 61.35, with 60.3 KB communication) — surpassing the full-embedding baseline (38.63, 60.48, 482.4 KB) — while k_L = 5 or 6 degrades below full embeddings. On Industrial, all k_L values beat full embeddings, with the best again at k_L = 4 (9.62, 16.97, 81.4 KB versus full embeddings 8.90, 15.56, 651.1 KB).

• Bigger RQ-VAE codebooks are not always better: Increasing codebook size d_R and codebook count l does not always improve performance; performance at d_R = 512 is lower than at d_R = 256, and performance begins to decline when l = 6. The authors suggest overly large codebooks or too many quantization levels introduce redundancy that can weaken relevance between semantically similar items.

• Hash sensitivity: For FedNCF, Hash(S) with SENet consistently outperforms Hash with Mean; performance improves with increasing h (especially from h = 1 to h = 2), but gains become marginal at h = 3 or 4. Larger d_H does not consistently help — in many cases smaller d_H is competitive at lower communication cost.

• Differential privacy behavior: Using the Laplace mechanism with scale δ, on the Industrial dataset both full and PEFT embeddings show minimal degradation under LDP as δ increases, while under CDP performance declines with increasing δ; PEFT methods including LoRA and RQ-VAE consistently outperform full embeddings. On ML1M, a similar LDP trend appears, with RQ-VAE even improving as δ increases; under CDP, RQ-VAE degrades significantly while full embeddings stay relatively stable, and LoRA shows increased H@10 at higher δ — suggesting LoRA is more robust under CDP and RQ-VAE better under LDP.

Methodology in Plain English

The server first converts item attributes (for example movie metadata or Amazon item descriptions) into input embeddings using the pre-trained sentence-t5 encoder with dimension k_p = 768. An autoencoder with fully connected encoder layers [768,512,256,128,32] and a reversed decoder is trained unsupervised to reconstruct these inputs, and its 32-dimensional latent vector becomes the initial full item embedding.

These full embeddings are distributed to clients and trained under standard FR for a brief warm-up — fewer than 20 of the 1,000 global rounds — because the authors observe that limited updates already yield noticeable gains at negligible extra average communication per round. After that, the full embeddings are frozen on every client, and only a small "compressed" component is optimized locally and uploaded. The server aggregates and redistributes only that compressed component.

Three compressed strategies are provided. In LoRA, a small per-item table A of dimension k_L plus a low-rank matrix B (initialized to zeros) are trained, and the item's effective embedding is the frozen full embedding plus B(a_i); both A and B are tuned and transmitted. In Hash, the server picks h universal hash functions and a shared table H of size d_H; each item's compressed vector is the mean (Hash) or a SENet-attention-weighted sum (Hash(S), with hidden expansion ratio r_h = 16) of its h hashed vectors, and only the shared table (plus the two small SENet matrices when used) is trained and uploaded. In RQ-VAE, item attributes are encoded and quantized into a semantic code — a tuple of l indices, one per codebook of size d_R, obtained by repeatedly quantizing the residual of the previous level; the semantic codes are pre-trained on the server and frozen, the codebooks are initialized with the same distribution as the full embedding rather than with the pre-trained centroids, and only the codebooks are trained and uploaded. RQ-VAE uses loss L(x) = L_recon + L_rqvae with β = 0.25, and initializes codebooks via K-means on first-batch residuals to reduce collisions.

Experimental setup: three datasets (ML1M, Amazon Software, Amazon Industrial), four backbones (FedMF, FedNCF, FedPerGNN, PFedRec), metrics Hit Ratio and NDCG at K = 10 and 20, client sampling ratio 10%, 2 local epochs per round, 1,000 global rounds, user embeddings kept local and never uploaded. Pre-training rounds are chosen from {10³, 10⁴, 10⁵, 10⁶} with learning rate from {1e-3, 1e-4}. Implementation uses FuxiCTR, PyTorch, and NVIDIA GeForce RTX 3090 GPUs, with results averaged over five runs.

Why This Matters

Impact on research. The paper reframes embedding overhead — not just model computation — as the central communication bottleneck in federated recommendation, and shows that a fine-tuning-style treatment of embeddings can be transplanted into existing FR methods without architectural surgery. It also introduces RQ-VAE, previously used mainly in generative retrieval, as a PEFT mechanism for FR, and provides a comparative framework (communication O(), storage O(), computation O(), representation capacity) for reasoning about compression strategies.

Real-world applications:

• Mobile and edge recommendation where item catalogs are large and clients pay for bandwidth per training round.
• Cloud-edge collaborative recommender services for e-commerce, where sending full item embedding tables to millions of devices is infeasible.
• Privacy-regulated personalization in domains such as healthcare or finance, where raw interaction data cannot leave the device and even model updates may be perturbed with differential privacy.
• On-device recommendation on storage-constrained hardware, where the compressed table also lowers local storage — a complementary benefit the authors note.

Industry relevance. The framework is plug-and-play and model-agnostic, so it can be layered onto deployed embedding-based FR systems rather than replacing them. The paper reports a concrete efficiency-accuracy trade-off example (PFedRec on ML1M: 60.3 KB per client with k_L = 4 versus 482.4 KB for full embeddings, with higher N@10 and H@10), which is the kind of operating point that matters for production federated deployments.

Future Directions

• Resolving the "no free lunch" pattern: the authors state that no single strategy consistently outperforms others across all datasets and FR models, leaving strategy selection (or automatic adaptation) unresolved.
• Exploring hybrid approaches that jointly optimize performance, communication efficiency, and client storage cost, since PEFT targets communication while compressed embeddings also reduce local storage — a trade-off the authors explicitly flag as promising but unexamined.
• Improving the robustness of RQ-VAE's collision resistance, which the paper says depends heavily on the quality of server-side pre-training, and understanding why performance degrades at large d_R (512) and at l = 6 due to redundancy.
• Better characterizing the differential privacy trade-offs, given the paper's finding that LoRA is more robust under CDP while RQ-VAE performs better under LDP.

Target Audience

Researchers and practitioners working on federated learning, privacy-preserving recommendation, and parameter-efficient fine-tuning; engineers deploying cloud-edge recommender systems with bandwidth or storage constraints; and readers interested in vector-quantization and hashing-based embedding compression, including its interaction with differential privacy. Readers without a background in federated optimization or embedding compression will find the framework conceptually accessible, but the evaluation and analysis sections assume an advanced level of familiarity.

Authors’ abstract

With the rise of cloud-edge collaboration, recommendation services are increasingly trained in distributed environments. Federated Recommendation (FR) enables such multi-end collaborative training while preserving privacy by sharing model parameters instead of raw data. However, the large number of parameters, primarily due to the massive item embeddings, significantly hampers communication efficiency. While existing studies mainly focus on improving the efficiency of FR models, they largely overlook the issue of embedding parameter overhead. To address this gap, we propose a FR training framework with Parameter-Efficient Fine-Tuning (PEFT) based embedding designed to reduce the volume of embedding parameters that need to be transmitted. Our approach offers a lightweight, plugin-style solution that can be seamlessly integrated into existing FR methods. In addition to incorporating common PEFT techniques such as LoRA and Hash-based encoding, we explore the use of Residual Quantized Variational Autoencoders (RQ-VAE) as a novel PEFT strategy within our framework. Extensive experiments across various FR model backbones and datasets demonstrate that our framework significantly reduces communication overhead while improving accuracy. The source code is available at https://github.com/young1010/FedPEFT.