Hail to the Thief: Exploring Attacks and Defenses in Decentralised GRPO

AI summary

Overview

Research area: Security and adversarial robustness of decentralised reinforcement learning for Large Language Model post-training, specifically Group Relative Policy Optimization (GRPO).

Technical level: Advanced. The paper assumes familiarity with reinforcement learning post-training, policy gradients, log-probability-based generation strategies, and distributed training synchronisation.

Scope (one sentence): The paper formalises two flavours of decentralised GRPO training, introduces the first poisoning attacks against them, and evaluates two defenses against those attacks.

What This Paper Is About

GRPO is well suited to decentralised training because nodes only need to exchange completions as strings rather than gradients or model weights. That same openness creates an opening for malicious participants: a dishonest node can share carefully crafted completions that satisfy the reward function while smuggling in unwanted behaviour. This paper asks how badly such an adversary can corrupt honest models, and whether filtering incoming completions can stop it.

Key Contributions

1. A formalisation of decentralised GRPO-style training in two configurations. Vertical dRL, where nodes generate completions for locally chosen prompts, and horizontal dRL, where nodes generate portions of the completions for globally selected prompts. The paper also distinguishes homogeneous model networks (all devices hold the same weights) from heterogeneous ones.

2. The first adversarial attacks against decentralised GRPO-style training. Four attacks are presented — TEXT (text injection), DoS (denial of service via long completions), MATH (equation manipulation), and CODE (malicious code injection) — and tested across vertical and horizontal settings and across text, math, and coding domains.

3. Two defense mechanisms. A Logit defense that checks whether incoming completion tokens fall within the generator's sampling distribution, and a Judge defense that uses an auxiliary LLM to label completions and multiply the verifiable reward by a binary judge reward.

4. An ablation on malicious participation. An analysis of how many poisoned completions are needed within a group, theoretical and empirical, including variation of the malicious participation rate.

Main Findings

• Attack success can be near-total and fast. Across attacks, an adversary achieves attack success rates of up to 100% in as few as 50 iterations (abstract). Within 20 iterations, an attacker can poison upwards of 60% of the completions produced by benign models, and reach as high as 100% during training.

• TEXT is the most effective attack. Text injection reaches an ASR of almost 100% in as few as 40 iterations. The target phrase used in experiments is "Hail to the Thief" (HTTF).

• DoS more than doubles output length. The denial-of-service attack, which teaches models to produce unnecessarily lengthy but conceptually correct chain-of-thought completions, more than doubles the generated token length. It is built by repeatedly querying the trained model (80 completions per question) and submitting the longest sequence per question that has the highest reward.

• MATH plateaus near 90%. The equation manipulation attack (targeting equations involving 2 and 2, e.g. 2+2=5) stagnates at around 90% ASR. The authors attribute this to models sometimes producing wrong equations such as 2+2≠4 or 2×2≠4 that also do not equal 5, which are counted as unsuccessful.

• CODE is the weakest attack but still substantial. Malicious code injection, which imports an attacker-owned library implementing basic math operations, achieves the lowest ASR of the four but still succeeds at over 50%.

• Defenses stop nearly everything except DoS. The Logit defense is sufficient against TEXT and CODE but provides no benefit against DoS and does not prevent MATH. The Judge defense perfectly prevents TEXT and CODE; the MATH attack succeeds up to iteration 50 but benign models then "unlearn" the behaviour. Neither defense stops DoS.

• Chain-of-thought matters greatly for the judge. Using CoT prompting, the Qwen-14B judge accepted 3.75% of TEXT attack completions, versus 86% with a no-CoT prompt. For MATH, the CoT judge accepted 1.54% versus 19.44% without CoT. Honest high-quality completions were accepted at 74.75% (CoT) versus 95% (no CoT). With CoT prompting, Qwen-7B accepted 38% of TEXT poisonings and Qwen-32B accepted 37%; both accepted over 83% of honest completions.

• Vertical is easier to attack at low participation. In a horizontal TEXT attack, ASR degrades sharply when malicious participation is very low. In a vertical MATH attack, there is minimal degradation in ASR at lower malicious participation, because the attacker controls all completions for a prompt.

• Roughly half the group is the optimal poison ratio. The theoretical analysis suggests the optimal number of malicious completions per group is 0.5, though even at 0.2 the attack should succeed. The relative effect analysis is plotted for group sizes of 12, 16, 24, and 32.

• Oracle answers are not required. Without access to ground-truth answers, attackers can generate N candidate solutions per prompt and use majority voting to predict the correct answer, then submit a modified response. Attacks still work under this constraint.

• KL-divergence and reward-outlier filtering are insufficient. Retaining the KL term does not prevent the attacks, and filtering completions by outlier reward mistakenly conflates reward variability with malice.

• Self-judging fails. Using the trained models themselves to judge incoming completions performed poorly; the game-theoretic optimal strategy for an agent is to accept every completion to maximise reward.

Methodology in Plain English

The authors set up a simulated decentralised training world of independent nodes running GRPO post-training on shared data with a shared reward model. A portion of the nodes (25% in the default configuration, i.e. 3 benign models and 1 malicious one, with 4 participants total) are dishonest and share manipulated completions during the all-gather step.

The core insight the attacks exploit is that GRPO assigns a single scalar advantage to an entire completion, which then boosts or punishes all the tokens in that completion. If a poisoned completion earns a high reward — because it still has correct formatting and the right final answer — the model learns the malicious content alongside the genuine solution. Attackers therefore keep the verifiable reward intact while injecting unwanted material into the <think> section or the code.

Experiments use math reasoning on the GSM8k dataset and coding on the OpenMathInstruct dataset, with Qwen-2.5 1.5B as the base model for math and Qwen2.5-Coder-1.5B for coding. Training uses a batch size of 32 prompts, 12 generations per prompt, the Adam optimizer, a maximum sequence length of 1024 tokens, and a learning rate of 2×10⁻⁵, on H100s connected via InfiniBand. Each experiment is repeated five times with different seeds, and ASR is tracked over 100 iterations of training. The paper separately verifies the attacks on larger Qwen-2.5-3B models.

The Logit defense exploits homogeneity: when all nodes hold the same model, a benign node can run an incoming completion through a single forward pass and check that every token is plausible under the generation strategy (top-k with k=50 and top-p with p=1.0 in these experiments). The Judge defense instead calls on an auxiliary Qwen-2.5-14B-Instruct model to assess each gathered completion, producing a binary reward that multiplies the verifiable reward.

Why This Matters

The paper shows that decentralised GRPO, which is attractive precisely because it requires very little communication, carries a cheap and highly effective poisoning risk. A small fraction of malicious participants can change the behaviour of honest models without reducing their measured reward, and current defenses against this class of attack are incomplete.

Research impact: it opens an adversarial-robustness agenda for decentralised RL, an area the authors note has had few attacks studied relative to federated learning, and it provides an initial attack/defense benchmark. The released code at https://github.com/gensyn-ai/HTTT makes the attacks and defenses reproducible.

Real-world applications affected:

• Crowdsourced or permissionless post-training networks, where nodes contribute completions rather than gradients, and where dishonest participants cannot be assumed away.
• Cost-sensitive inference deployments, where the DoS attack inflates token generation length and therefore serving cost without producing incorrect answers that would trigger a correctness-based filter.
• Code-generation assistants, where malicious code injection teaches models to import an attacker-owned library that performs basic mathematical operations — a payload that can be activated after deployment.
• Judge-based moderation and reward pipelines, which this paper shows are vulnerable to false negatives and depend on prompt design (CoT versus no CoT) and judge model choice.

Industry relevance: the paper notes that existing systems already employ decentralised training in various forms (LlamaRL, decentralised SAPO), and that no active monetarily incentivised decentralised RL system exists to the authors' knowledge. The implication is that threat modelling should be built in before such systems become economically attractive to attack.

Future Directions

• Adaptive attacks against the defenses. None of the presented attacks assumed a defense was deployed. The authors flag jailbreak-style attacks that search for prefixes or suffixes able to pass the judge's gavel as future work.
• Bitwise reproducibility as a hardening mechanism. If the generation seed is public and all models are identical, another node can validate a generation exactly. The authors leave integration of reproducible operations to future work, noting the Logit defense becomes perfect under these conditions.
• Prompt optimisation for the judge. The optimal prompt for judging completions is described as a research problem in its own right, and the current prompts are standard rather than tuned.
• Modular, task-specific defenses. Because the attack space is excessive, the authors argue against catch-all defenses and propose instead task-specific measures such as a reasonable max-token limit to counter DoS.
• Judge availability. For some tasks or languages, an adequate judge model may simply not exist, which limits the Judge defense.

Target Audience

Researchers and engineers working on decentralised or federated training of LLMs, reinforcement learning post-training pipelines, and adversarial machine learning. It is most useful to readers with prior exposure to GRPO, policy gradients, and distributed training synchronisation, though the attack descriptions and results are written clearly enough for a technically comfortable newcomer to follow.

One caveat: the supplied paper content is truncated partway through Appendix B.2, so the full detail of that ablation and any later appendices is not available here; several appendix results (B.1, B.3, B.4, B.5, C.2, C.3, D.1–D.3) are referenced in the text but their contents are only partially reported in the available content.

Authors’ abstract

Group Relative Policy Optimization (GRPO) has demonstrated wide adoption in the post-training of Large Language Models (LLMs). In GRPO, prompts are answered by the model and preferred behaviour is learnt via reinforcement learning. Owing to the small communication volume, GRPO is inherently suitable for decentralised training as the prompts can be concurrently answered by multiple nodes and these completions are exchanged in the form of strings. In this work, we explore the robustness of decentralised GRPO by presenting the first adversarial attacks and countermeasures. We present a diverse set of attacks where malicious nodes poison benign models by sharing their poisoned completions. We demonstrate these attacks on math and coding tasks and show that an adversary can achieve attack success rates of up to 100% in as few as 50 iterations. Moreover, to mitigate the attacks, we propose two defense mechanisms that check logit probabilities of completions or utilize an LLM judge to filter completions. The defenses prevent all but the DoS attack that causes unnecessarily lengthy but conceptually correct completions. The code of both attacks and defenses can be found at: https://github.com/gensyn-ai/HTTT.