What Gradients Add to Text Leakage in Split Language Models, Counted per Token and per Document

AI summary

Overview

Research area: Security and privacy of split learning for large language models (cs.CR), specifically privacy leakage from activations and gradients crossing the client-server split.

Technical level: Intermediate. The paper assumes familiarity with transformer layers, gradients, tokenisation and basic statistics (bootstrap intervals, R², ROC AUC), but its attack is described step by step and its central result can be read without implementing anything.

One-sentence scope: The paper measures how much the returning gradient adds to text reconstruction from activations in split-trained language models, counted both per token and per whole document, and tests whether the layer at which an offloaded run starts changes model quality or leakage for a fixed run length.

What This Paper Is About

In split learning, a client runs the first layers of a language model on its own hardware and sends the server only activations, one vector per token, so the raw text never leaves. During training the server sends gradients back. Both signals are computed from the private text, so an observer at the split may be able to invert them.

The paper's central problem is that prior work shows gradients and activations can each be inverted, but does not say how much the gradient adds once activations alone already give away most of the text, or whether that answer depends on how success is counted. A second question asks whether, for a server-trained run of fixed length, it matters where in the model that run starts.

Key Contributions

1. How to score defences. The authors show that per-token and per-document scores can rank the same defence very differently, so a leakage report needs both.
2. What gradients add. They measure how much the returning gradients help an attacker that already reads activations well, on documents never used to train the model or the attacker. On GPT-2 the gradients add 3.17 points of token recovery and nearly triple the share of documents rebuilt exactly, and noise or rounding applied to the observed gradient leaves most of this gain in place.
3. Where offloading starts. On GPT-2 and Qwen3-0.6B, they show that when the server trains only part of the model, the layer where that part starts affects both model quality and leakage, even when the run's length is fixed. Knowing the start helps predict new training runs only under the 8-bit defence.
4. Everything released. They release the code, the exact inputs, the study plans fixed in advance and the results of every run for both experiments.

Main Findings

• Activations alone already leak almost every token. The activation-only attacker, holding only the publicly released weights of the client's layers, recovers 94.20–97.81% of tokens at both split depths and with both copies of the client's layers. Because inference sends activations only, these rates also describe an attacker watching inference traffic.

• Gradients add three points of token recovery. At depth six with the public copy and candidate budget k = 32, the joint attacker recovers 97.38% of tokens against the activation-only attacker's 94.20%. The paired difference is +3.17 percentage points with a 95% crossed-bootstrap interval of [+2.72, +3.64], clearing the two-point pass mark fixed in advance. Of the tokens the activation-only attacker gets wrong, the gradients correct about 55%.

• Whole-document recovery nearly triples. At the main setting, exact 32-token document recovery rises from 13.71% to 37.77%, a 2.75-fold increase. The largest gain is with the trained copy at depth six, from 20.23% to 58.55%. Across the four depth/copy combinations the pairs are: depth 3 public 43.13% → 55.51%, depth 3 trained 51.41% → 66.13%, depth 6 public 13.71% → 37.77%, depth 6 trained 20.23% → 58.55%.

• The first token caps exact recovery. The first position gets no gradient help, so it is often wrong: in the released predictions at the main setting it is wrong in 2,408 of 5,120 activation-only reconstructions and 2,362 of 5,120 joint reconstructions. In about a third of these cases (820 and 806) the predicted token differs by nothing more than a leading space and looks identical in print. These errors alone keep whole-document recovery below about 54% for both attackers.

• Perturbing the observed gradient changes little. At depth six with the public copy, token recovery is 97.31% under noise of scale σ = 0.1 and 97.33% under eight-bit rounding, against 97.38% with the true gradient. Stronger perturbations cost more but leave most of the gain: 96.83% at σ = 1.0 and 96.38% under four-bit rounding. Both controls that strip the gradient's information do worse than no gradient at all (shuffled gradient 93.91%, random candidate 93.89%, versus 94.20% for activation-only).

• Secret mixup nearly removes exact documents but not tokens. Under secret mixup (weight λ = 0.75 on the true activation) the attacker still recovers 83.01–90.67% of tokens, yet its best whole-document recovery is 1.48%. With the public copy at k = 32, whole-document recovery is 0.31% at depth three and 0.04% at depth six.

• Secret mixup protects names, not numbers. Name recovery falls to 47.85–66.15% from 67.36–97.44% without a defence. Numbers are protected far less (93.94–96.50% recovered, from 97.64–100.00%) and identifiers 90–100% (from 100.00–100.00%). Names are the largest annotated class, present in 249 of 256 documents, while the identifier class contains two spans.

• Secret mixup costs model quality; replica-median does not. Secret mixup raises held-out perplexity by 31.8% at depth three and 26.9% at depth six. Replica-median costs almost nothing (perplexity 36.05 → 36.51–36.62) and leaves token recovery within two-thirds of a percentage point of the undefended attacks, while still changing whole-document recovery by up to 7.7 points (SipIt with the public copy at depth three falls from 43.77% to 36.11%).

• Where the server-trained run starts matters, mostly. In the second experiment, the start explains variation beyond length in seven of eight tests (Table 5). The one failure is GPT-2 leakage under the 4-bit noisy defence (partial R² = 0.21, p = 0.019, above the corrected threshold of 0.0125). Under the 8-bit defence the start accounts for almost all leftover variation in both outcomes for both models (partial R² of 1.00 for GPT-2 utility and Qwen3 utility, 0.99 for GPT-2 leakage, 0.91 for Qwen3 leakage).

• Only 8-bit placements generalise to unseen seeds. Fitting on seeds 11–66 and evaluating on seeds 77 and 88, adding the start to a length-based predictor lowers held-out RMSE from 1.5698 to 0.7594 for utility and from 0.3042 to 0.1337 for the leakage index under the 8-bit defence. Under the 4-bit noisy defence the falls are smaller (0.2154 → 0.2018 and 0.0783 → 0.0735) and neither reaches the specified 10%.

Methodology in Plain English

The attacker is a passive observer at the split. It never changes what either side computes, never sees the private text, the server's layers or the next-token labels, and probes only the final trained model.

Main experiment setup. GPT-2 (12 layers, 124M parameters) is split after layer 3 or layer 6 (L ∈ {3, 6}). Inputs are 32-token passages from a pinned WikiText-2 copy, divided into 32 public calibration documents the attacker may use, 128 documents that train the client, and 256 evaluation documents the attacker must rebuild. The attacker gets either the publicly released weights of the client's layers or the client's trained copy.

The attack, in four steps. (1) Pass the 32 public documents through the attacker's copy of the client's layers and fit a ridge regression mapping each activation to its token's embedding, called the ridge fit. (2) At each position, list the k tokens whose embeddings are closest to the ridge fit's output by cosine similarity, with k ∈ {8, 16, 32}. (3) The joint attacker replaces half that list with tokens suggested by the gradient; since the gradient comes from the next-token loss, the gradient at one position carries a signal about the token that follows it. The first position gets no gradient help. (4) A beam search of width two builds the text left to right, recomputing activations through the attacker's copy of the client's layers and keeping the partial texts that best match what was observed. Both attackers check the same number of partial texts, so the only difference is which candidates they try.

Controls and alternatives. A shuffled-gradient control gives the joint attacker a different document's gradient; a random-candidate control fills the gradient half of the list with random tokens; a backward-only heuristic uses the gradient alone; and SipIt is implemented by the authors and run with 32, 64 and 128 iterations per token.

Scale and statistics. The main experiment runs 40 attack configurations (28 undefended, 12 defended) across both depths and both copies against ten trained models (victim seeds) with two attack seeds, giving 3,200 attack evaluations; six activation-only configurations are also counted as inference, adding 480, for 3,680 in total, each on the same 256 documents. The main test uses a crossed bootstrap resampling documents and trained models as separate axes (10 victim runs × 256 documents, 100,000 replicates, fixed seed) and passes only if the interval's lower end exceeds two points. A further 47 token comparisons use a Bonferroni correction of α = 0.05/47. The main difference is computed at 3.1714 points before rounding.

Second experiment. The server trains a run of s consecutive layers starting at layer b, written [b, b+s). The GPT-2 grid crosses lengths s ∈ {2, 4, 6} with starts b ∈ {0, 1, 2, 3}, two defences and eight victim seeds, giving 192 trained models, plus eight paired all-local runs — 200 in total, all completed. Each run takes 128 AdamW steps with batch size one and learning rate 5×10⁻⁵ in one pass over the training documents. The two defences both compress what crosses the split: an 8-bit defence rounding activations and gradients to eight bits, and a 4-bit noisy defence rounding to four bits with the returned gradient clipped at 1.0 and noise added at 0.1 times each document's gradient RMS, in the order clip, noise, round. Utility is the extra held-out cross-entropy over the paired all-local model, ΔU = L_segment − L_all_local. Leakage uses an adapted vanilla model inversion (VMI) attack that decodes each activation to the nearest token by cosine similarity using the ridge fit, plus a property attack that sums the size of the returned gradient per document to guess whether a document contains token ID 286, 357 or 764. Both are folded into a normalised leakage index I = max(0, a_VMI − a_control, max_j 2|AUC_j − 0.5|). A test passes if partial R² ≥ 0.10 and a permutation test (10,000 shuffles of start labels within each seed and length) gives p < 0.0125, with 98.75% percentile intervals from 10,000 bootstrap draws. The Qwen3-0.6B replication uses 28 layers, the same lengths, starts, defences and eight victim seeds, and the same 128 steps at 5×10⁻⁵; the share of layers trained uses 28 as its denominator.

Why This Matters

Impact on research. The paper argues that a leakage report must give both per-token and per-document numbers, because the same defence can look strong under one and weak under the other. It also pushes against judging privacy defences by exact-copy counts alone: secret mixup stops almost all exact documents while 83–91% of tokens remain recoverable. The second experiment reframes partial offloading as a design choice with privacy consequences, not just a compute choice.

Real-world applications.

• Auditing deployed split-learning systems where a client keeps text on-device and offloads layers to a third-party server.
• Comparing privacy defences before deployment: the paper's perplexity-versus-leakage framing lets an operator see what secret mixup's 27–32% perplexity cost buys, and that replica-median costs almost nothing while still lowering whole-document recovery by up to 7.7 points.
• Choosing where to place the boundary between client and server, since the start of a server-trained run changes both model quality and leakage even for fixed length.
• Assessing whether transmitted gradients need protection at all: strong noise (σ = 1.0) and four-bit rounding left most of the gradient's gain in place, so degrading the gradient is not by itself an adequate defence here.

Industry relevance. The paper's bottom line for practitioners is that what a split model sends should be treated as being as sensitive as the text itself, not merely as an intermediate representation.

Future Directions

• Explaining why the start matters. The paper establishes that the starting layer changes utility and leakage in most tests, and that the effect generalises to held-out seeds only under the 8-bit defence, but the mechanism is not resolved. A predictive model for placement would let operators pick a split point without retraining.
• Defences that reduce token recovery, not just exact documents. Secret mixup leaves 83–91% of tokens recoverable and replica-median leaves token recovery within two-thirds of a point of undefended attacks. A defence that lowers the token floor as well would change the picture.
• Protecting names and numbers equally. Secret mixup protected names far more than numbers (93.94–96.50% of numbers and 90–100% of identifiers still recovered) and the identifier class contains only two spans, so span-level conclusions for identifiers rest on very little data.
• Extending past two models and one split boundary setting. The results cover GPT-2 at two split depths and Qwen3-0.6B in the partial-offloading grid; the paper notes that Qwen3's replication needed more numerical precision after a first run at 32-bit precision failed, and the supplied text ends mid-sentence in this section, so the remaining details of that replication are not reported in the content provided.

Target Audience

Researchers working on privacy attacks and defences for distributed or split training; engineers and architects choosing where to place the client-server boundary in a split-learning deployment; and privacy auditors or reviewers who need to know why a single leakage number is insufficient. The paper is written so that a reader with intermediate machine-learning background can follow the attack and the statistics without reproducing the implementation, and the released study plans, inputs and per-run results make it directly usable by practitioners who want to check the claims.

Authors’ abstract

Split learning lets a client train a language model on a server without sending its text. The client runs the first layers itself and sends the server only their output, a vector of numbers for each token. During training, the server sends gradients back. We show that an observer at the split can rebuild most of the client's text from this traffic, and we measure how much the gradients help. On GPT-2, an attacker who holds only the publicly released weights of the client's layers recovers 94.20% of tokens from the activations alone and 97.38% when it also sees the gradients, 3.17 percentage points more 95% interval [2.72, 3.64]. Counted by document, the difference is much larger. The attacker rebuilds 13.71% of 32-token documents exactly without the gradients and 37.77% with them, because a document only counts when every token is right. How we count also changes how good a defence looks. Secret mixup, which blends each outgoing vector with a decoy, stops the attacker from rebuilding almost any document exactly, yet the attacker still recovers 83-91% of tokens. In a second experiment on GPT-2 and Qwen3-0.6B, where the server trains only a run of consecutive layers, the layer at which the run starts changes both model quality and leakage, even when the run's length is fixed. We recommend reporting leakage both per token and per document, and treating what a split model sends as being as sensitive as the text itself.