Exploring Cross-Client Memorization of Training Data in Large Language Models for Federated Learning

AI summary

Overview

• Research area: Privacy and training-data memorization in large language models (LLMs), specifically within federated learning (FL) settings. The paper sits at the intersection of federated learning, LLM privacy auditing, and cross-sample memorization measurement.
• Technical level: Intermediate. Readers need familiarity with federated learning basics (FedAvg, clients, communication rounds), LLM fine-tuning, and decoding strategies, but the core idea is conceptually simple.
• Scope: The paper proposes a framework that adapts centralized-learning cross-sample memorization measurement to federated learning, then uses it to quantify intra-client versus inter-client memorization across four task types and to study which training and inference factors change memorization rates.

What This Paper Is About

Existing FL memorization studies rely mainly on canary injection — inserting artificial out-of-distribution phrases to see if the model repeats them — and measure one sample at a time. That approach assumes a memorized text can only be triggered by a prompt from the same sample, which hides the risk that a prefix from one client could cause the global model to leak a suffix belonging to a different client.

This paper generalizes the definition of in-distribution memorization from centralized learning to FL, introduces a pairwise memorization ratio between clients, and measures how much the global model memorizes intra-client data (harm-exposed) versus inter-client data (harmful), along with the factors that influence each.

Key Contributions

1. Problem formulation: Reframes subtle memorization detection in FL by adapting cross-sample memorization assessment methods originally designed for centralized learning, and introduces a formal definition of in-distribution FL memorization in which a prefix from client Cj can induce memorization of a suffix from client Ck.
2. Framework and metrics: Designs a framework that computes a pairwise memorization ratio (MR j→k) and derives aggregate metrics — MR_Intra (harm-exposed), MR_Inter (harmful), MR_TotalCL and MR_TotalFL (total memorization ratios in centralized versus federated settings). Code is released publicly.
3. Empirical measurement across clients: Conducts a study quantifying intra-client versus inter-client memorization on four tasks — summarization, dialog, question answering, and classification — with 3 FL clients per task.
4. Factor analysis: Examines which factors influence FL memorization, including decoding strategy, prefix length, federated algorithm, model size, and number of communication rounds, plus follow-up analyses of input/output length and of which suffixes are most memorized.

Main Findings

• FL models do memorize training data, mostly within the same client. MR_Intra is consistently higher than MR_Inter across all evaluated tasks: summarization 0.342% versus 0.046%, dialog 1.533% versus 1.446%, QA 1.450% versus 0.813%, and classification 0.000% versus 0.000%.
• Classification shows no memorization at all in the default FedAvg setting, which the authors attribute to generated text being shorter than the PAN2014 match threshold of at least 50 characters.
• No clear trend when going from centralized to federated training. In summarization MR_TotalCL was 0.558% versus MR_TotalFL 0.433%; in dialog 3.417% versus 3.992%; in QA 2.150% versus 2.917%; in classification 0.000% versus 0.000%. This contrasts with prior work (Thakkar et al., 2021) that reported reduced memorization in FL, which the authors attribute to that work measuring out-of-distribution memorization via canary injection.
• No observed correlation between memorization and task performance in either the CL or the FL setting.
• Decoding strategy matters. Top-p produced the highest rates: dialog MR_Intra 3.792% / MR_Inter 2.996% and QA 2.567% / 1.438%, compared with top-k (dialog 1.533% / 1.446%; QA 1.450% / 0.813%) and temperature (dialog 1.267% / 1.442%; QA 1.283% / 0.750%). Summarization rates were lower overall (top-p 0.525% / 0.050%; top-k 0.342% / 0.046%; temperature 0.475% / 0.067%).
• Shorter prefixes induce more memorization. At prefix length 10 vs. 100: summarization MR_Intra 0.508% vs. 0.208% and MR_Inter 0.188% vs. 0.004%; dialog 2.108% vs. 1.408% and 1.992% vs. 1.150%; QA 1.525% vs. 1.125% and 1.383% vs. 0.429%. Prefix length 30 (the default) sits between these: summarization 0.342% / 0.046%, dialog 1.533% / 1.446%, QA 1.450% / 0.813%.
• Federated algorithm matters. FedProx yielded higher memorization than FedAvg on every task: summarization 0.942% / 0.138% versus 0.342% / 0.046%; dialog 1.892% / 1.879% versus 1.533% / 1.446%; QA 3.675% / 2.146% versus 1.450% / 0.813%. FedProx also produced nonzero memorization on classification (MR_Intra 0.011%, MR_Inter 0.000%) where FedAvg produced 0.000% / 0.000%. The authors conclude memorization should be evaluated independently per algorithm.
• No strong association with model size or communication rounds. Results for these are reported in the appendix (Sections G.6 and G.7), and the authors state they observed no clear trend.
• Generated text length tracks training input and output lengths. Median generated lengths for Qwen2.5-3B were 10 tokens for summarization (input 184, output 15), 59 for dialog (input 86, output 108), 78 for QA (input 325, output 47), and 1 for classification (input 38, output 2). For Llama3.2-3B: 6 (input 182, output 15), 66 (84, 107), 26 (313, 47), and 2 (37, 2). This short generation on classification limits extractability.
• Some clients' suffixes are more memorizable than others. In the dialog task, suffixes from Group 3 were memorized more than others regardless of which client supplied the prefix (e.g., Group3→Group3 at 1.950% versus Group2→Group1 at 1.150%), suggesting dataset characteristics drive memorization.
• Memorization is mostly paraphrase-level. The per-category breakdown shows verbatim rates near zero in most cells, with the larger contributions coming from paraphrase categories — for example, Qwen2.5-3B dialog MR_Intra of 1.533% splits into 0.000% verbatim, 0.067% idea, 0.258% paraphrase (p > 0.5), and 1.217% paraphrase (p < 0.5).

Methodology in Plain English

The authors take an existing centralized-learning recipe for detecting cross-sample memorization and adapt it so it works between clients rather than within a single dataset.

Each training sample is split into a fixed-length prefix (the prompt) and a suffix (the text whose leakage is being tested). To estimate the memorization ratio from client Cj to client Ck, the framework samples a subset of prefixes from Cj and a subset of suffixes from Ck (each of size n = 4K, where n is much smaller than either pool). It then:

1. Feeds each sampled prefix into the fine-tuned global model to generate text.
2. Indexes the sampled suffixes with Elasticsearch and retrieves the top n′ = 10 suffixes most similar to the generated text.
3. Compares the generated text against those top-10 suffixes using the PAN2014 plagiarism detector, which acts as the discriminative function and reports three similarity types: verbatim, paraphrase, and idea-level. A minimum match threshold of at least 50 characters is used.
4. Counts a prefix as memorization-inducing if the detector returns True for any of the top-10 suffixes in any of the three categories, and estimates MR j→k as the fraction of prefixes that trigger True.

From these pairwise ratios, the aggregate metrics are computed: MR_Intra as a weighted sum of same-client ratios, MR_Inter as a weighted average over other clients, and MR_TotalFL / MR_TotalCL as the fraction of the union of all memorization-inducing prefixes over the union of all prefixes.

Experimental setup: four tasks (summarization with ArxivAbstract, dialog with HealthCareMagic patient-doctor dialogues, QA with PubMedQA, classification with PubMed 200k RCT), each partitioned across 3 FL clients in a non-IID way, with 30,000 sampled instances per client and a 27,000/3,000 train-test split. Classification and other partitions use a Dirichlet distribution with α = 5.0. The main model is Qwen2.5-3B, with additional results for Llama3.2 and GPT2. Training used LLaMA Factory with a 2e-4 learning rate, bfloat16, and batch size 64. The default configuration is Qwen2.5-3B, prefix length 30, top-k decoding, and 3 FL rounds under FedAvg. Decoding variants used k = 40, p = 0.8, and temperature = 1.0. Model-size experiments covered Qwen2.5-0.5B/1.5B/3B and Llama-3.2-1B/3B (494M, 1.54B, 3.09B, 1.24B, and 3.21B parameters), with 1, 3, and 5 communication rounds. Training used one NVIDIA A100 GPU with 4,000 GPU hours; applying the memorization framework took 1,600 GPU hours and 1,600 CPU hours.

Why This Matters

The paper shows that federated learning does not neutralize memorization of LLM training data, and that leakage can cross client boundaries — a prefix held by one participant can surface a suffix belonging to another. Because existing FL privacy audits lean on canary injection, they likely underestimate this risk on realistic in-distribution data. The framework gives FL researchers a direct way to quantify cross-client leakage and reveals that the measured level depends on inference choices (decoding, prefix length) and on the federated algorithm, meaning the same data can look safer or riskier depending on configuration.

Real-world applications:

• Healthcare and medical dialog: Two of the four datasets are medical (patient-doctor dialogues from HealthCareMagic and PubMedQA), where leaked suffixes could expose patient conditions or clinical text.
• Clinical-trial and biomedical text processing: The PubMed 200k RCT classification task represents abstracts of randomized controlled trials, where memorization could surface unpublished or restricted study text.
• Academic writing and publishing: The ArxivAbstract summarization task frames memorization as plagiarism risk when abstracts are used as training data.
• Cross-silo federated deployments generally: Any multi-organization FL setup (hospitals, banks, institutions) where prefixes may be partially public and suffixes are private.

Industry relevance: Teams deploying FL for sensitive LLM fine-tuning can use the released framework as a privacy audit before shipping a global model, and can tune decoding strategies, prefix-length assumptions, and algorithm choice based on the measured leakage.

Future Directions

• Explain why intra-client memorization exceeds inter-client memorization. The authors note that a theoretical investigation into this gap is an open and interesting direction, since their evidence is empirical.
• Identify what makes some clients' suffixes more memorizable. Group 3 suffixes in the dialog task were memorized more than others; the paper calls for research into which specific dataset properties drive this and how they interact with the FL setting.
• Improve discriminative detectors. PAN2014 can only measure fine-grained memorization in English and produces misleading results on incoherent model output (misclassifying repetition as idea-level memorization), so it is an apparent limitation of any PAN2014-based method.
• Extend beyond the factors that showed no clear trend. Model size and communication rounds showed no strong association with memorization, which the authors report but do not explain.

Target Audience

This paper is most useful to federated learning and privacy researchers, LLM practitioners who fine-tune models across multiple data holders, and engineers evaluating memorization risk in privacy-sensitive deployments. It also benefits readers already familiar with centralized-learning memorization metrics who want to understand how those methods transfer to decentralized training, and reviewers or policy-focused readers assessing whether FL alone is sufficient protection for sensitive training data.

Authors’ abstract

Federated learning (FL) enables collaborative training without raw data sharing, but still risks training data memorization. Existing FL memorization detection techniques focus on one sample at a time, underestimating more subtle risks of cross-sample memorization. In contrast, recent work on centralized learning (CL) has introduced fine-grained methods to assess memorization across all samples in training data, but these assume centralized access to data and cannot be applied directly to FL. We bridge this gap by proposing a framework that quantifies both intra- and inter-client memorization in FL using fine-grained cross-sample memorization measurement across all clients. Based on this framework, we conduct two studies: (1) measuring subtle memorization across clients and (2) examining key factors that influence memorization, including decoding strategies, prefix length, and FL algorithms. Our findings reveal that FL models do memorize client data, particularly intra-client data, more than inter-client data, with memorization influenced by training and inferencing factors.