LLM-Attack Suite: A Unified Red-Team Framework for Privacy Evaluation of Split and Distributed LLM Systems

AI summary

Overview

Research area: Security and privacy of machine learning systems (cs.CR), specifically the privacy of split learning / distributed LLM inference and training, with a methodological emphasis on reproducible red-team evaluation.

Technical level: Advanced. The framing is conceptual and readable, but the paper assumes familiarity with split learning, activation/gradient inversion attacks, ridge decoders, beam search, bootstrap intervals, and Wilson bounds.

Scope (one sentence): The paper presents LLM-Attack Suite, a framework that standardizes how privacy attacks against split and distributed LLM systems are recorded, compared, scored and audited, and uses a controlled GPT-2 124M study plus corpus, defence and implementation studies to show that evaluation choices change the conclusion.

What This Paper Is About

In split learning, a client keeps the private text and runs the first layers of a model, sending intermediate activations to a server; during training the server returns gradients. Both messages carry information about the private text, and prior work has reconstructed text from gradients, from activations, or from both. The problem is that these results are hard to compare, because attackers differ in what messages they see, which model weights they hold, which corpus they use, and how recovery is measured — and a defence can look effective simply because the attack already fails without it. The paper's goal is to make a privacy evaluation an explicit, checkable experiment: one that records the attacker's access and budget, reuses the same captured messages across compatible attacks, demands a linked undefended reference before any defence can pass, and reports which attacks remain untested.

Key Contributions

1. A common evaluation procedure. An evaluation is defined as a tuple ℰ = (T, V, O, K, B, C, M, G) covering the target model/split/defence/checkpoint, the attacker's position and privileges, its observations, its knowledge, its calibration/optimisation/search budgets, the corpus and tokenisation, the measured outcome, and the controls, uncertainty calculation and decision rule. Each module declares its supported modes, input requirements, threat metadata and execution function.

2. Checkable privacy results. Capture is separated from attack execution: a split capture stores forward messages and (for training) returned gradients with session and ordering metadata, compatible attacks replay the same observations, and predictions are saved for independent re-scoring. Scoring uses dataset-specific baselines, a linked undefended control, and a coverage report that marks applicable threats as covered, partial, unmeasured or out of scope. The registry records validation evidence for all 64 modules, including three orchestration helpers.

3. Evidence for the procedure from a controlled reconstruction experiment. Using GPT-2 124M split after layer three or six on pinned 32-token WikiText-2 passages, the study isolates the effect of giving the attacker returned gradients while matching the search (32 candidate tokens per position, beam width two).

4. Separate evidence from reference conformance and measurement-code testing. A BiSR port is compared against its archived reference, and 25 pre-selected development-test functions (37 parameterised cases) are rerun to check metrics, score calculations, control eligibility, observation boundaries, solver fidelity, resampling and evidence validation — deliberately reported separately from measured attack effectiveness.

Main Findings

• Gradients add far more in whole documents than token averages suggest. At depth six with public client layers, token recovery rose from 94.20% to 97.38% (a gain of 3.17 percentage points, 95% interval [2.72, 3.64]), while exact-document recovery rose from 13.71% to 37.77%. The prespecified primary test passed: the interval lay above the two-percentage-point threshold fixed before testing. All 47 secondary contrasts favoured the joint attack after Bonferroni correction.

• The gain depends on gradient information, not on replacing half the candidates. Shuffled-gradient and random-candidate controls recovered 93.91% and 93.89% respectively, against 49.98% for the backward-only heuristic.

• Small perturbations leave reconstruction largely intact. Gaussian gradient noise at 0.1 times the per-document root mean square left recovery at 97.31%, and eight-bit quantisation at 97.33%.

• Configuration matters as much as the attack. At the same undefended depth-six boundary, exact-document recovery across attack configurations varies from 1.70% to 58.55%, with differences in observation, knowledge and effort. Table 1 runs from 94.20%/13.71% (depth 6, public layers, activation-only) to 98.72%/66.13% (depth 3, trained layers, joint); each entry aggregates 5,120 document-level scores from the same 256 held-out documents across ten trained models and two attack seeds.

• Defences shift the failure mode rather than removing it. Secret mixup left 83.01–90.67% token recovery while limiting exact-document recovery to at most 1.48%; name-span recovery remained 47.85–66.15% across 716 annotated spans in 249 documents (annotations fixed before the run, with AI assistance for names — described as semantic annotations, not human gold labels). Geometric-mean perplexity rose by 31.8% at depth three and 26.9% at depth six. Replica median, an integrity defence that sends activations unchanged, kept token recovery within two-thirds of a point of its undefended comparator.

• Corpus choice changes both recovery and its baseline. The same linear inversion decoder without beam search recovered every token of templated synthetic clinical notes (baseline 6.25%), 64.94% of source code (baseline 20.31%), and 63.25% of WikiText-2 (baseline 2.64%), giving excesses of 93.75, 46.78 and 62.79 percentage points. The clinical result characterises regular synthetic templates and does not measure recovery from clinical records. All three undefended conditions failed the token gate.

• Knowing the defence enables white-box leakage through compression. The validated white-box path recovered 18.85–68.59% mean tokens across stable-compressor depth profiles despite reducing each 768-dimensional activation to 64–256 numbers; co-trained bottlenecks showed 23.52–67.68% across their separate final-state protocol. Twenty checks validated the gradient path, work accounting and capture matching, and all co-trained captures replayed exactly.

• Utility costs can be severe. In a 500-step extension, a fixed random projection suppressed the declared, projection-unaware inversion attacker below its majority baseline but also largely prevented useful learning: cross-entropy remained 7.42–7.75 against 4.39–4.44 without a defence.

• Longer contexts and noise shift recovery, but not to zero in every case. On 64- and 128-token code and Wikipedia passages and 32-token dialogue, undefended inversion recovery ranged from 76.56% to 90.22%; Gaussian activation noise with σ = 15 reduced it to 9.96–55.05%.

• No single attack wins everywhere. The BiSR port matched the archived reference on all 3,072 paired predictions. In the adapted comparison, refined BiSR recovered 93.21% at depth six with public layers versus 97.38% for the joint attack, a paired difference of −4.17 points with interval [−4.79, −3.53]; with trained layers the difference was −0.02 points, interval [−0.63, 0.58]. Of 15 prespecified secondary contrasts, ten intervals lay below zero, one above, and four crossed zero.

• The measurement code behaved as specified on the selected cases. All 37 selected cases matched expectations with no failures or skips; hiding scorer-only text from the predictor left predictions byte-identical and supplying that text to the predictor was rejected; removing or weakening a required positive control prevented a candidate pass; and increasing perfectly correlated document columns from two to 500 preserved the interval [0, 1] rather than treating copies as independent evidence.

• The paper is explicit about the limits of its own gate. The Wilson upper bound uses z = 2.5392, a fixed convention originating in a five-percent error budget across nine known-plaintext probe attempts, and is not a simultaneous confidence guarantee over all 64 modules; token correlation limits a literal binomial interpretation, so the gate is an operational screening convention rather than a distribution-free privacy guarantee.

Methodology in Plain English

The suite treats an evaluation as a recorded experiment rather than a one-off script. First, the target model is run and its split traffic is captured once — activations for every token, plus gradients during training — and stored with metadata. Attacks then replay those same stored messages, so comparisons are paired: the same checkpoint, the same documents, the same captured tensors. Each attack declares what it can see (activations only, or activations plus gradients), what weights it holds (a public pretrained copy of the client layers, or the client's trained copy including fine-tuned embeddings), and its fixed budget.

Reconstruction itself works in four stages. The attacker fits a linear ridge map from public activations back to token embeddings; lists the 32 tokens whose embeddings point most nearly in the same direction as the mapped activation; the joint attacker keeps 16 of these and adds 16 candidates derived from the negated gradient at the previous position (whose target is the next token); finally a width-two beam search reads left to right, re-running candidate prefixes through the client layers and keeping the two whose recomputed activations best match what crossed the split. Controls swap in another document's gradient, or replace gradient-derived candidates with random tokens, to test whether the gradient's information is doing the work.

Scoring compares predictions against held-out text with a separate scorer. The gate asks how much text the strongest tested attack recovers above a simple baseline (the most common token in the attacker's calibration text), using a versioned one-sided Wilson upper bound. A candidate defence passes only if its largest excess is at most one percentage point, it has at least three random-seed runs, and a linked undefended reference from the same attacker family itself shows excess above one percentage point; otherwise the verdict is INSUFFICIENT_EVIDENCE. The framework also reports a "floor" — the positive excess even a baseline-level attack shows at finite sample size — so a failed gate must be read alongside the estimate and interval.

Statistics use bootstrap resampling of trained models and documents independently with 100,000 replicates and seed 93281, plus Bonferroni correction for the 47 secondary contrasts. The campaign contains 3,200 training measurements and 480 explicitly reused activation-only inference measurements, 3,680 records in total, all re-verified against manifests. Evidence handling checks files against retained hashes, recomputes metrics from saved predictions, and regenerates predictions from pinned inputs; optional hash-chained journals reveal edits but do not authenticate an untrusted producer.

Why This Matters

Impact on research. Split-learning privacy results have been reported under heterogeneous attacker assumptions, corpora and metrics, which makes "defence X reduces leakage to Y%" hard to interpret. This paper argues, and demonstrates, that changing a single protocol component — adding gradients, swapping Wikipedia for code, granting the attacker trained layers — changes the conclusion, so results should be reported with the attacker, corpus and scoring rule named, with controls attached, and with untested attacks labelled as untested. The emphasis on retaining predictions so reviewers can re-score without retraining the victim model is a concrete step toward auditable privacy claims.

Real-world applications:

• Privacy review of split inference deployments where a user's phone or browser holds the first layers and a remote server holds the rest.
• Split fine-tuning pipelines, where returned gradients are the leak channel that a forward-message-only privacy check would miss.
• Compression and bottleneck defences for on-device or bandwidth-limited serving, evaluated jointly on leakage and utility rather than leakage alone.
• Regulatory or internal audit documentation, where a verdict must name its tested settings and list what was not tested.

Industry relevance. The framework's separation of implementation checks from measured effectiveness, its versioned scoring policies, and its refusal to aggregate different protocols into one success rate map onto how security assurance is usually documented. Vendors shipping split or distributed LLM features can reuse the same capture-and-replay structure with their own models, and the reported utility costs (for example, the perplexity increases with secret mixup and the learning degradation under a fixed random projection) show why leakage numbers alone are a poor basis for a defence decision.

Future Directions

• Extending beyond GPT-2 124M and the stated corpora, lengths and budgets. The paper says its strongest quantitative reconstruction comparisons concern GPT-2 124M, and that longer contexts, new targets and additional attack budgets can extend the evidence through the same procedure.
• Adding attacks to retained captures. Because captures and predictions are stored, new modules can enter a frozen comparison through the same input/output contract; the coverage report would then shrink its list of untested threats. The paper notes that with a fixed protocol and existing measurements, adding an attack can only increase the envelope or leave it unchanged.
• Replacing the operational gate with an inferential one. The authors flag that the Wilson-bound gate is a screening convention, not a distribution-free guarantee, and that token correlation limits a binomial reading — an open question is what statistical construction would give a defensible guarantee across many modules.
• Broadening agentic and cross-surface evidence. The live agentic tier qualified a Qwen3-0.6B planner/tool integration in ten fixed settings and twenty trials but blocked all six sanctioned peer-request trials, identifying over-blocking rather than general agentic security; the authors state this does not establish general agentic security, and that membership, timing and agentic outcomes remain separate from the text-reconstruction gate.

Target Audience

Privacy and security researchers working on split learning, federated and distributed LLM systems, and machine-learning privacy evaluation methodology; red-team and ML-safety engineers who need a reproducible structure for comparing attacks and defences; reviewers and auditors who must judge whether a reported privacy claim is supported; and defence designers evaluating compression, noising or decoy-based protections who need leakage and utility measured together. Readers wanting an introduction to inversion attacks themselves will find the paper assumes prior familiarity, though Supplementary Section S2 walks through the attack mechanics step by step.

Authors’ abstract

Split language models keep input text on a client but send intermediate representations to a server. Attacks can reconstruct text from these messages. Comparing privacy evaluations is difficult when attackers see different messages, know different model weights or use different datasets and recovery measures. We present LLM-Attack Suite, a framework that records these choices and lets researchers check, compare and extend an evaluation. Compatible attacks reuse recorded messages, and their predictions are saved for independent scoring. The scoring rule requires a linked undefended reference before a defence can pass. Reports also identify attacks that remain untested. In a controlled GPT-2 experiment, adding gradients returned during training increased token recovery from 94.20% to 97.38% and exact-document recovery from 13.71% to 37.77%. The 95% interval for the token gain lay above the two-percentage-point threshold fixed before testing. This experiment is shared with a companion study; here it tests how the evaluation responds to a change in attacker access. Further studies measure how corpus choice and compression defences affect recovery, compare an adapted published attack with its reference implementation, and test the measurement code. The framework reports implementation checks separately from measured attack effectiveness. Each privacy claim should state the attacker, dataset and scoring rule, report its controls and identify untested attacks.