Privacy Failure in Split-LLM Training, The Returned Gradient Nullifies the Decoys

AI summary

Overview

Research area: Systems security and privacy auditing for split learning / split-LLM training (cs.CR), touching gradient-inversion attacks, privacy evaluation methodology, and differential-privacy-style auditing.

Technical level: Advanced. The paper assumes familiarity with split learning, gradient inversion, Wilson confidence bounds, Bonferroni correction, and cross-entropy as a proper scoring rule.

Scope: A single-system security case study of a two-node split-LLM training pipeline whose privacy gate passed while the backward wire (the returned output gradient) was never declared as a privacy surface.

Note: the supplied paper content is truncated inside Section 7.1 ("A selected audit of split-LLM evaluations"), so the external-audit discussion and Section 8 (scope, limitations, verification trail) are summarized only to the extent they appear above.

What This Paper Is About

In split learning, a trusted local node (TLN) holds the private data and the loss, and sends protected activations to an untrusted cloud node (UCN) that holds a middle stack of transformer layers; the TLN then returns the output gradient so the cloud side can train. The paper shows that in this implementation, mixing 48 decoy ("chaff") rows into every 80-row frame to hide the 32 real rows fails: because the private loss ignores the decoys, their returned gradients are exactly zero, and the pattern of zeros tells the cloud exactly which rows were real. The goal is a verified diagnosis of that defect plus a channel-explicit, calibrated evaluation protocol that would have caught it.

Key Contributions

1. A verified structural disclosure diagnosis. The paper documents that, with the output gradient returned unprotected, the zero-support pattern of each gradient frame exactly reproduces the real-versus-decoy split — 4,096/4,096 frames, row-level agreement 1.000, on each of nine seeds.
2. A calibrated evaluation protocol applied to the case. It instantiates three audit disciplines: declare every channel, calibrate every metric against a deliberately injected known leak, and define a pre-set gate statistic (Bonferroni-adjusted Wilson upper-95 excess over a constant baseline, failing above +1.0 pp) with shuffled-label negative controls.
3. A bounded content-inference result. A pre-set frequent-token probe on the frame contents yields a pooled paired effect of +0.92 pp (95% interval [0.74, 1.09], τ = 0.23, I² = 77%) across nine seeds, with the gradient's own margin over an oracle-partitioned forward frame at most +0.18 pp on five of the six exploratory seeds.
4. A replication on a deployable configuration and a mitigation. Six defended, gradient-open cells across two datasets pass the forward privacy gate and the utility gate yet fail once the returned gradient is included (joint-view gate +1.44 to +2.18 pp, 6/6), and per-row gradient clipping (C = 0.01) plus Gaussian noise (σ = 0.35C) closes the leak for roughly 0.01 nats of held-out cross-entropy.

Main Findings

• The forward gate passed while the backward wire leaked. The original evaluation instrumented only the forward wire and passed its gate; the backward wire was outside the declared adversary view and was never scored.
• The zero-support partition is exact and deterministic. Decoy gradients are identically zero because the loss truncates before the decoys, so the split is disclosed on 4,096/4,096 frames per run on every seed, with row agreement 1.000 — not a sampling artefact.
• The content effect is modest and bounded. Across the nine committed seeds the gradient-arm paired effect over the constant baseline ranges from +0.6456 pp (seed 48) to +1.5008 pp (seed 49); the shuffled-label negative controls sit at |N| ≤ 0.08 pp and recover nothing. The constant baseline sits near 5 to 6% for this model and corpus, so +1 pp corresponds to roughly one extra token per hundred.
• The gradient contributes the partition, not the content. Comparing gradient and oracle-partitioned forward arms seed by seed, the gradient's margin is at most +0.18 pp on five of six exploratory seeds (+0.0011 to +0.0395 pp on four of them) and reaches +0.64 pp only on seed 42.
• The leak survives a configuration that passes both gates. On all six defended, gradient-open cells (three seeds on each of two datasets) in the four-layer mitigation topology, the run passed the forward privacy gate and the utility gate, yet the joint view broke the +1.0 pp gate set in advance.
• Per-row clipping and noising removes the signal. With per-row gradient clipping at C = 0.01 and Gaussian noise at σ = 0.35C, all scored arms sit at floor, support reads 0/1,024 frames (agreement 0.400 at the 32/80 base rate), and utility costs about 0.01 nats.
• The naked control restores the signal. The same topology with the defence off reads +33.9 to +34.0 pp on the joint view, and +9.5 to +10.7 pp on the forward gate, so the defended near-floor readings reflect the defence rather than a blind instrument.
• The cross-Gram feature carries nothing. Because the forward activation matrix X and returned gradient G share the per-request rotation, their product XGᵀ cancels it; the implemented cross-tensor probe reads at floor on every captured cell (defended gate statistic +0.31 to +0.86, paired effect −0.29 to +0.46; naked paired effect −0.78 to −0.22).
• The structural signal's conversion to a token advantage is attack-specific. The effect is detected at 12 and 11 delegated layers and at floor at 8 and 6; detected at D = 64 and D = 96 and at floor at D = 128. Increasing exposure from 40k to 100k steps (2.5×) did not amplify the per-row effect in that run (+0.9302 to +0.9066 pp). The partition classification itself remains exact at every depth and width.
• Metric calibration matters. In the dose–response sweep, token_top1 has a sharp onset between coverage 0.04 and 0.06, rare_token_top1 is the most sensitive (first responding at 0.04, full recovery at 1.0), token_cross_entropy is dose-insensitive until the injected leak dominates, and membership_auc is flat and dose-insensitive (AUC − 0.5 spans 0.058 to 0.159), so it was falsified as a channel and retained only as a probe-generalisation diagnostic.
• Five attack classes were never measured. These are accumulated_history, stateful_remote_state, timing_metadata, membership_property, and response_side; active_perturbation is listed as constructible but unexecuted. The paper states the system is therefore not shown to be safe.
• The effect appears on a second corpus but is not shown to be dataset-independent. On the public WikiText-2 corpus the gradient alone breaks the gate on two of three seeds (+1.175 and +1.219).

Methodology in Plain English

The researchers take one concrete split-LLM setup — Qwen3-0.6B with a split after layer 14 and resumption at layer 26 in the main configuration, and 21/26 in the mitigation runs — and study it as a security case rather than proposing a new general method. Every training frame carries 32 real rows plus 48 decoys, and the defence applies clipping at C = 1.0 with Gaussian noise σ = 0.35C plus a per-request rotation and permutation and a latent width of D = 64.

Before scoring anything, the team declares which channels the attacker observes and which metrics apply to each. They then calibrate each metric by injecting a known token leak at controlled dose (coverage × amplitude) and checking where the metric starts to respond — so a metric that has never detected a planted leak is not allowed to certify its absence. Finally they set the decision rule in advance: an attack result is the top-1 token accuracy of a probe minus a constant baseline that always guesses the most frequent evaluation token, with the gate failing only when the Bonferroni-adjusted Wilson upper-95 excess exceeds +1.0 pp. Shuffled-label twins serve as negative controls, and uncertainty is bootstrapped over frames within a run and hierarchically over seeds.

They run nine seeds in total across three groups — 42–47 exploratory, 48–50 replication, 51–53 mitigation — and capture either 4,096 frames per run (main and replication) or 1,024 (mitigation). To check the leak is not an artefact of a configuration nobody would deploy, they repeat it in a shallower four-layer topology that passes the utility gate, and then apply per-row gradient clipping and noising to see whether the leak closes.

Why This Matters

Impact on research. The paper argues the failure is not a new discovery of gradient leakage — that is established in gradient-inversion and split-learning work — but an evaluation failure. A channel absent from the declared adversary view is exempt from the gate by construction, and a threshold calibrated against degenerate floor cells can pin the statistical floor and hide a signal that is deterministic and exact. The calibration results show that reconstruction metrics need not agree on privacy risk, so no single threshold fits all metrics.

Real-world applications (as contexts the work speaks to):

• Split-LLM and split-learning deployments where a data owner rents cloud compute and assumes activations-only sharing is private.
• Mobile and edge training pipelines that offload middle layers to a cloud node to save local compute.
• Privacy auditing and compliance workflows that need positive controls and negative controls before a system is declared clean.
• Deployments in regulated data domains (for example healthcare or finance) where a privacy gate result is used as evidence in a review.

Industry relevance. Any organization whose privacy claim rests on an evaluation that enumerates channels must first check the enumeration is complete; here the omitted channel carried a deterministic signal, and the mitigation (per-row gradient clipping at C = 0.01 with Gaussian noise) closed it at roughly 0.01 nats of held-out cross-entropy. The paper is explicit that this does not make the system safe: five classes of attack, including those accumulating observations across training steps, were never measured.

Future Directions

1. Measure the unmeasured channels. The accumulated_history and stateful_remote_state families — attacks that pool observations across training steps — have no positive control and are the most direct open question, alongside timing_metadata, membership_property, and response_side.
2. Extend beyond the two implemented emitters. The paper states other cross-tensor attacks remain open, and that stronger inversion attacks can recover prompts from deep LLM states, so the observed depth threshold is not evidence that depth generally provides privacy.
3. Test at larger scale and longer budgets. The reported runs are 2,000-step diagnostics on the 0.6B model, and each shape threshold is a single-seed reading on seed 42; the claim that a configuration passes both gates attaches to the four-layer topology, not the eleven-layer main configuration.
4. Revisit the data split and the external evaluations. The runner tokenizes the flat corpus into disjoint sequential fixed-width blocks rather than a document-level split, and Section 7 audits three recent split-LLM evaluations — a natural place to apply the same channel-enumeration and calibration demands.

Target Audience

Security and privacy researchers working on split learning, split-LLM training, gradient inversion, and privacy auditing; ML engineers who deploy or review split-training systems and rely on privacy gates; and reviewers or auditors who need to check whether an evaluation enumerated every observable channel and calibrated each metric against a planted leak before accepting a pass. Readers focused on model quality or training efficiency will find less here, since the paper explicitly frames itself as a systems-security case study rather than a general methodology.

Authors’ abstract

We present a systems-security case study of a two-node split-LLM training system whose privacy evaluation passed while leaving an observable channel untested. The Trusted Local Node (TLN) sends protected activations to the Untrusted Cloud Node (UCN), the UCN returns its output, and TLN, holding the private loss, returns the output gradient. The frame the UCN receives mixes real rows with decoys, and the loss ignores the decoys. Their gradients are exactly zero, so the pattern of zeros reveals which rows were real. We measure it with a protocol fixed in advance: a leak injected at known strength to prove the instrument can see one, a shuffled-label control to prove it does not report absent leaks, and a threshold set before the runs. Across nine seeds, the zeros identified the real rows on every frame, 4,096 of 4,096 per run. An attack on the frame contents recovered about one extra token per hundred over a constant-guess baseline (+0.65 to +1.50 percentage points); the shuffled controls recovered nothing. A second set of runs repeated this on a configuration that keeps model quality within budget, so the finding is not confined to a setting nobody would deploy. On both datasets, every such run passed the forward-channel privacy check and the quality check, yet failed that same check once the returned gradient was included. Clipping and noising each row of the gradient closed the leak for about 0.01 nats of held-out cross-entropy. The system is not thereby safe: five classes of attack, including those accumulating observations across training steps, were never measured.