Pick Your Poison: Learning to Select Poison Sets for Stronger LLM Backdoor Attacks

AI summary

Overview

Research area: Machine learning security — specifically backdoor data poisoning attacks against large language models, framed as a combinatorial set-selection problem.

Technical level: Advanced. The paper assumes familiarity with influence functions, finetuning, attack success rate metrics, and oracle-budgeted optimization.

Scope: The paper shows that which poisoned examples an attacker selects matters as much as how many, and introduces a learned, oracle-budgeted method (SAILS) for finding strong poison sets across LLaMA-3-8B, SmolLM-360M, Qwen3-4B, and API-only Kimi-K2.5 settings.

What This Paper Is About

Backdoor poisoning attacks insert a small number of trigger-response examples into a model's finetuning data, so the trained model misbehaves when the trigger appears. Standard evaluations fix the number of poisoned examples and pick them at random, implicitly assuming the particular examples do not matter.

This paper shows that assumption is false: across three LLaMA-3-8B backdoor settings, holding the model, clean data, trigger, target behavior, and poison count fixed, held-out attack success ranges from 3% to 80% depending only on the poison set chosen. The goal is to find strong poison sets under a limited budget of expensive finetune-and-evaluate runs.

Key Contributions

1. Formalization of poison selection as oracle-budgeted set optimization. The authors define the problem over a candidate pool with a poison-set size, note the combinatorics (900 candidates with a budget of 9 gives roughly 10^21 poison sets), and show both empirically and theoretically that pointwise influence proxies can be suboptimal — aggressive optimization of a single pointwise-additive proxy can even reduce true attack success.

2. SAILS (Set-level Audit-Informed Iterative Learned Selection). A propose–score–audit framework: train a set scorer on oracle-labeled poison sets, propose and cheaply score large numbers of candidate sets, audit only a small top-ranked shortlist with expensive oracle queries, and retrain on audited results. The authors prove a shortlist-regret bound (Theorem 1) showing that as long as the scorer ranks one strong set high enough to be audited, SAILS returns a near-optimal set.

3. Empirical gains across primary and extended settings. SAILS improves held-out attack success by 30 percentage points on average over the strongest influence baselines across three LLaMA-3-8B settings, and the pipeline extends to code-generation, agentic, and API-only backdoors.

4. Transfer and access flexibility. A scorer trained on cheaper small-scale finetune-and-evaluate runs transfers to full-scale finetuning, and selected poison sets partially transfer across architectures without re-optimization.

Main Findings

• Poison-set choice dominates vulnerability: Across three LLaMA-3-8B backdoor settings, with model, clean data, trigger, target behavior, and poison count all fixed, different poison sets from the same candidate pool yield held-out attack success ranging from 3% to 80%. Random-set evaluation therefore substantially underestimates worst-case risk.

• SAILS beats influence baselines on mini benchmarks: At an oracle budget of B = 1500, SAILS achieves 72% / 92% / 67% ASR on refusal / command / compliance (average 77%), outperforming the best B = 10 influence method by +30 percentage points on average. With |D| = 3000 scorer labels it reaches 78% / 91% / 74% (average 81%), exceeding oracle-greedy construction on refusal and compliance at comparable budget. Oracle greedy itself scores 76% / 97% / 61% (average 78%).

• Baseline specifics: TRAK (B = 10) averaged 31%, TRAK + representer 40%, gradient cosine 23%, gradient dot product 26%, bilevel influence 30%, and random selection (mean, B = 1500) 24%. Random oracle best-of-B reached 56% average. SAILS also outperformed TRAK + representer at the same oracle budget by +21 percentage points on average.

• Low false-trigger rate: Across all settings and methods, the false-trigger rate on clean (untriggered) inputs remained below 5%.

• Budget regimes on SmolLM-360M: At low budget (B < 100), TRAK is competitive (36% at B = 10); at medium budget (100 < B < 1000), SAILS with iterative refinement achieves the strongest ASR among methods compared, reaching 68% at B ≈ 370 on a 50K pool versus roughly 58% on a 900-item pool, while random oracle best-of-B plateaus at roughly 44% in both pools; at high budget (B > 5000), oracle-guided RL with GRPO peaks at 74% but needs about 6,500 evaluations. SAILS reaches about 92% of that observed ASR at 1/18th the oracle cost.

• Mini-to-full scorer transfer works: A scorer trained only on mini-scale labels (roughly 1500 evaluations, 50 epochs) selected poison sets for full-scale finetuning (100 epochs, larger corpus, higher k) reaching 80% / 69% / 58% ASR on refusal / command / compliance — beating random mean by +41 percentage points and the strongest evaluated TRAK-greedy construction by +20 percentage points on average, with poison-set sizes growing from 4 to 9, 5 to 9, and 2 to 5 respectively.

• Partial cross-model transfer: SAILS-selected sets (optimized on LLaMA-3-8B only) were evaluated on 9 unseen target models without re-optimization. SAILS best-of-10 outperformed random best-of-10 on 6/9 targets for refusal/compliance and 4/9 for command.

• Code-generation backdoor (Qwen3-4B): On NL2SH-ALFA with k = 12, a pool of 1000 NL queries, and 200 clean bash pairs, SAILS achieved 67% ASR at B = 1200 versus 47% for random best-of-B and 28% for TRAK (B = 10). The attack injects a malicious curl pipe when a file path is rewritten to /home/anthropic/.

• Agentic backdoor (WebShop): With Qwen3-4B, full-parameter finetuning (30 epochs), k = 2 multi-turn poison trajectories from a 200-item pool, and a ModernBERT scorer trained on 750 random oracle labels, SAILS achieved 91% ASR versus 84% random best-of-750, evaluated on 100 held-out sneaker goals.

• API-only finetuning (Kimi-K2.5): Using the Tinker API on a model with 1T parameters and 32B active, with k = 2 poison examples, 200 clean pairs, and an oracle budget of B = 200, SAILS reached 72% ASR, versus 16% mean ASR for random selection and 46% for random oracle best-of-B.

• Robustness to mini-batch SGD: Re-evaluated under mini-batch SGD, SAILS retained higher mean ASR than the influence baseline in each condition, though outcomes varied across seeds.

Methodology in Plain English

The authors treat poison selection as a search problem with a scarce measurement instrument. The instrument is the oracle query: a full finetune-and-evaluate run that returns the attack success rate (or the triggered loss) for one specific poison set. These runs are expensive — the paper notes they take hours per set — so the attacker may only afford a few hundred.

Rather than ranking individual examples and assuming their effects add up, SAILS learns a function that scores whole sets. It works in rounds:

• Seed: Sample random poison sets, run the oracle on them, and train an initial set scorer on those labels.
• Propose: Generate N candidate k-sized sets.
• Score: Use the current scorer to rank all N candidates cheaply.
• Audit: Pick a shortlist of m candidates (much smaller than N) using epsilon-greedy acquisition and run the expensive oracle on each. The oracle, not the scorer, makes the final choice.
• Refine: Add the newly measured labels to the training data and retrain the scorer.

The default scorer is a DistilBERT encoder with a regression head. Sets are represented as text: the examples in a set are sorted by pool index and concatenated with separators. This means the scorer needs no access to the victim model's weights, gradients, or hidden states, and can score novel combinations of examples.

A key design point the authors emphasize is why this beats pointwise influence: pointwise proxies drop the interaction terms between examples, so they cannot penalize redundancy (two examples whose joint effect falls below the sum of their individual effects) or exploit complementarity (joint effect above the sum). A learned set scorer can capture both.

To make labeling cheaper, the authors train the scorer on small-scale (mini) finetune-and-evaluate runs and reuse it to rank candidates for more expensive full-scale finetuning, then audit the shortlist with the full-scale oracle. This exploits the fact that finetune-and-evaluate runs are much cheaper for a small model — on SmolLM-360M each query takes about one minute.

Hyperparameters used throughout unless stated otherwise: initialization with |D_0| = 500 randomly sampled oracle-labeled sets, N = 500 candidate sets scored per round, m = 10 audits per round, and epsilon = 0.2 (20% of the audited shortlist sampled at random).

A theoretical result, Theorem 1, bounds shortlist regret — the utility gap between the best proposed set and the best audited set — by two terms: how much the best proposed set is underestimated by the scorer, plus the smallest overestimation among audited sets. If the best proposed set is itself audited, the regret is zero. This bound motivates the refinement step, since enlarging the candidate pool N with a fixed audit budget m creates a Goodhart effect where overestimated sets can displace stronger ones.

Why This Matters

Impact on research: The paper challenges a standard evaluation shortcut. If random poison-set sampling underestimates worst-case vulnerability by the margins reported here, then published robustness numbers may systematically understate risk. The paper also argues that improving pointwise influence estimators can reduce precision loss but cannot, by definition, reduce additivity loss — redirecting effort toward set-level methods. The code is released at a public repository.

Real-world applications:

• Auditing third-party finetuning data: Modern models are routinely finetuned on public instruction sets, crowd-worker data, and user interactions. The paper's framing gives auditors a way to estimate worst-case vulnerability rather than average-case vulnerability on such data.
• Code-generation security: The Qwen3-4B code-generation result — a file-path trigger that causes injection of a malicious curl pipe into generated bash commands — is directly relevant to coding assistants that translate natural language into shell commands.
• Autonomous agent safety: The WebShop setting shows an agent can be poisoned into silently purchasing a specific high-priced item when a user asks for sneakers, regardless of the user's stated preferences. This maps onto real shopping, browsing, and tool-using agents.
• Evaluation of API-only finetuning services: The Kimi-K2.5 result shows selection works without access to weights, gradients, or activations, which matters for hosted finetuning APIs where the provider, not the customer, holds model internals.

Industry relevance: The findings apply directly to any organization that finetunes on externally sourced data, or that offers finetuning-as-a-service. Since SAILS operates in an oracle-only regime — needing only scalar feedback from finetune-and-evaluate runs — it is a practical threat model as well as a practical auditing tool, and it does not require the attacker to modify clean data, the victim architecture, or the training algorithm.

Future Directions

• Defense and detection: The paper focuses on attack-side selection. A natural next step is whether the same set-level scoring machinery can be inverted to identify and filter high-risk poison sets before training, or to build poison-set-aware defenses.

• Wider cross-model and cross-scale transfer: Cross-model transfer succeeded on 6/9 targets for refusal/compliance and 4/9 for command. The paper does not report an analysis of which content-level properties drive this partial generalization, or how to improve it.

• Extending the score-signal and encoder choices: The framework supports alternative encoders and input representations — Ridge and GNN predictors using victim-model hidden states are mentioned as requiring white-box access — but the paper's default is DistilBERT on concatenated text. The space of scoring architectures and set representations is left largely open.

• Broader threat surfaces: The paper reports settings spanning refusals, command injection, harmful compliance, code generation, shopping agents, and API-only finetuning. Additional trigger mechanisms, target behaviors, and domains are not explored here, and the paper's truncated content indicates further robustness experiments (for instance, mini-batch SGD and clean-data effects) that are only partially described.

Target Audience

This paper is most useful to machine learning security researchers studying data poisoning and backdoor attacks; red-teamers and safety evaluators who audit finetuning pipelines and third-party training data; practitioners building or operating finetuning-as-a-service platforms, coding assistants, or tool-using agents; and ML researchers working on data selection, influence functions, and set-level learning. Readers will get the most from it with prior familiarity with finetuning workflows, attack success rate as an evaluation metric, and the basics of influence-based data attribution.

Authors’ abstract

Backdoor poisoning attacks add poisoned examples to otherwise-clean finetuning data, pairing a trigger with a target behavior that the model learns to produce when the trigger appears. Existing evaluations typically fix the number of poisoned examples and sample them at random from a candidate pool. We show that this can severely underestimate worst-case vulnerability: across three LLaMA-3-8B backdoor settings, holding the model, clean data, and poison count fixed, attack success ranges from 3% to 80% depending only on which poison set is chosen. We formalize poison selection as oracle-budgeted set optimization and introduce SAILS (Set-level Audit-Informed Iterative Learned Selection), which learns a set scorer from a few hundred finetune-and-evaluate runs, ranks millions of candidate sets, and audits only a small shortlist. SAILS improves held-out attack success by 30 percentage points on average over the strongest influence baselines, transfers from small-scale to full-scale finetuning, and extends to code-generation, agentic, and API-only backdoors.