Diffusion Guided Adversarial State Perturbations in Reinforcement Learning

AI summary

Overview

• Research area: Adversarial robustness in reinforcement learning (RL), specifically state-perturbation attacks on vision-based agents, and the use of diffusion models as generators of adversarial inputs.
• Technical level: Advanced. The paper assumes familiarity with Markov Decision Processes, denoising diffusion probabilistic models (DDPMs), classifier and classifier-free guidance, and standard RL robustness defenses.
• Scope: The paper proposes SHIFT, a policy-agnostic, diffusion-based attack that generates realistic, history-aligned but semantically altered image states to defeat state-of-the-art defenses in RL.

What This Paper Is About

Reinforcement learning agents that take raw image input can be misled by small, carefully chosen perturbations to their observations. The paper argues that the defenses currently considered state of the art only look robust because existing attacks are constrained to stay inside an l_p ball around the true image, which means they can barely change the decision-relevant meaning of the scene even when the budget is relatively large. The goal of this work is to build an attack that breaks out of that constraint by generating perturbed states that are semantically different from the true state while still looking realistic and consistent with the agent's recent history, so they evade detection.

Key Contributions

1. A diagnosis of existing attacks and defenses. The authors show that the effectiveness of current defenses, including diffusion-based ones, stems from a fundamental weakness of l_p norm-constrained attacks, which cannot alter the semantics of an image input even under a relatively large perturbation budget. They note that Korkmaz [32] was the only prior study to identify this limitation for state perturbations in RL, but that its high-sensitivity direction attacks use simple image transformations targeting visually significant rather than domain-specific semantics.

2. New formal characterizations of stealthy semantics-aware attacks. The paper defines valid states (states reachable under some stationary policy), realistic states (states within a projection-distance threshold δ1 of the valid set), semantics-changing states, history-aligned states, and trajectory-faithful states (bounded by threshold δ2). Because these conditions are computationally expensive to verify, the authors provide practical approximations: reconstruction error from an autoencoder-based anomaly detector for realism, Wasserstein distance for history alignment, and SSIM and LPIPS for trajectory faithfulness.

3. A guided diffusion attack framework. SHIFT combines classifier-free guidance conditioned on history to generate history-aligned next states, policy guidance using the victim's state-action value function Q^π to steer toward states that induce low-value actions, and autoencoder guidance to restore realism. Theorem 3.6 shows that policy guidance and classifier-free guidance can be combined in the reverse process without interfering, because the classifier-free noise term depends only on the history τ_{t−1} while the policy gradient term depends only on Q^π.

4. Two attack instantiations and an empirical evaluation. SHIFT-O conditions on the true history to produce perturbations aligned with the real trajectory, while SHIFT-I conditions on the victim's observed (perturbed) history with the last action dropped, luring the agent into a self-consistent but imaginary trajectory. Both share the same pretrained diffusion model, policy guidance, and realism guidance; they differ only in history conditioning. The attack is policy-agnostic, requires no retraining across victim policies, and applies to both value-based and policy-based RL. Code is released in a public GitHub repository.

Main Findings

• Defenses are only apparently robust. The authors state that the current defenses withstand prior attacks because those attacks cannot change decision-relevant semantics, not because the policies are genuinely robust to semantics-aware perturbations.
• SHIFT breaks all known defenses. Evaluation reports that the attack effectively breaks existing defenses, including the most sophisticated diffusion-based ones, and significantly outperforms existing attacks while being more perceptually stealthy.
• Large reward drops. The paper reports that SHIFT lowers agents' cumulative reward in various environments by more than 50%, as shown by episode rewards in Pong, Freeway, Doom, and AirSim. For example, in Table 1 the undefended DQN baseline earns 21.0 ± 0.0 in Pong, 34.1 ± 0.1 in Freeway, 75.4 ± 4.4 in Doom, and 40.3 ± 0.5 in AirSim, while under SHIFT-O with attack frequency 1.0 DQN earns −21.0 ± 0.0 in Pong, 1.2 ± 0.8 in Freeway, −354.0 ± 8.2 in Doom, and 9.8 ± 2.3 in AirSim. Against DMBP, the strongest defense reported in the visible table, SHIFT-O-1.0 yields −14.0 ± 4.9 in Pong and 22.0 ± 0.7 in Freeway.
• Partial attacks remain effective. SHIFT is only permitted to attack a fraction ξ of time steps; results are reported for attack frequency 1.0 and 0.25, showing that attacking less often still degrades performance substantially against several defenses.
• Better stealth metrics. Compared with prior attacks, SHIFT shows lower reconstruction error, lower Wasserstein-1 distance, and lower LPIPS, and higher SSIM.
• Diffusion states are closer to true states. In the Freeway environment, states generated by the history-conditioned diffusion model have a smaller average l2 distance to the true states than states produced by PGD, even with small budgets.
• A trilemma in sequential adversarial perturbations. The paper argues that current methods optimize only two of three critical properties: semantic-altering, historically-aligned, and trajectory-faithful. SHIFT-O favors trajectory faithfulness and semantic alteration at the cost of subtle historical inconsistency from abrupt object disappearance; SHIFT-I favors historical alignment and semantic alteration at the cost of deviating from the true trajectory.
• Qualitative examples. In AirSim, PGD with an l∞ budget of 15/255 and a 3-degree counterclockwise rotation leave decision-relevant semantics intact, while SHIFT-O removes pedestrians and bicycles at a crosswalk and SHIFT-I makes the car appear to have already crossed.

Methodology in Plain English

The authors treat the attack as a generation problem rather than a noise-optimization problem. During a training stage, they collect trajectories from a clean environment using a well-trained reference policy, then train a conditional diffusion model that predicts the next state given a window of past true states and actions. They train it with classifier-free guidance, randomly dropping the history conditioning so the model learns both conditional and unconditional distributions, which lets a guidance scale at sampling time control how strongly the generated state must agree with the given history. A separate autoencoder is trained only on clean states so that its reconstruction error can serve as a realism score.

At test time, the attacker observes the true state and the victim's policy. Starting from noise, the reverse diffusion process is modified in three ways: classifier-free guidance pulls the sample toward the given history, policy guidance adds a gradient term that pushes the sample toward observations that make the victim take low-value actions under the true state, and autoencoder guidance nudges the sample back toward the manifold of clean states. The attacker decides when to attack using a state importance weight, defined as the gap between the maximum and minimum Q-values over actions. An attack is injected when the current weight falls in the top ξ percentile of all previous weights and the cumulative number of attacks is still below t·ξ. When no attack is injected, SHIFT-O passes the true state through, while SHIFT-I keeps generating states conditioned on the perturbed history and true actions, without policy guidance, to keep the imaginary trajectory internally consistent. Although the theory is presented using DDPM, the implementation uses the EDM formulation, which needs fewer reverse steps and enables real-time attacks, building on prior work that applied EDM-based conditional diffusion to Atari world modeling.

Why This Matters

• Impact on research: The paper reframes robustness in vision-based RL as a semantics problem rather than a pixel-budget problem. It argues that l_p-bounded threat models understate the vulnerability of agents, and it introduces evaluation criteria (semantic alteration, historical alignment, trajectory faithfulness) plus quantitative proxies (Wasserstein-1, SSIM, LPIPS, reconstruction error) for measuring them.
• Real-world applications:
  • Autonomous driving, where an attacker could erase pedestrians or bicycles from a camera image, or make a crosswalk appear already crossed, pushing the vehicle toward unsafe acceleration.
  • Robotics and embodied AI, where manipulated camera observations could cause a manipulator or mobile robot to take unsafe actions.
  • Industrial automation, where image-based control policies could be misled without any obvious visual corruption that a human operator would notice.
  • Security-sensitive and safety-critical RL deployments, including sensor or communication-channel manipulation scenarios the paper references.
• Industry relevance: The attack is policy-agnostic and does not require retraining for each victim policy, and it can be run with few enough reverse diffusion steps for real-time use, which makes it a practical red-teaming tool for teams deploying RL in perception-driven systems. The finding that diffusion-based defenses can be defeated also signals that hardening such defenses is an active engineering problem, not a solved one.

Future Directions

• Closing the trilemma. The authors state that finding an attack that better satisfies semantic alteration, historical alignment, and trajectory faithfulness than current methods remains an open problem.
• Stronger detectors. The paper notes that SHIFT-O's abrupt object disappearance can disrupt temporal coherence and that this could in principle be detected by an agent with sufficient domain knowledge and advanced detection capabilities, though it shows this is a nontrivial task for complex environments.
• More robust policies. The authors conclude that the results indicate the importance of developing more robust policies, implying defenses that reason about semantics rather than pixel budgets.
• Scaling and generalization. The framework requires a separate diffusion model per environment and is evaluated in Pong, Freeway, Doom, and AirSim; extending the approach to other environments, and to the illusory-attack style at image scale, remains open given that prior illusory attacks do not scale to high-dimensional image inputs.

Target Audience

This paper is most useful to adversarial machine learning and RL security researchers, authors of robust RL defenses, and practitioners deploying vision-based RL in autonomous driving, robotics, or industrial settings who need to understand realistic threat models. Readers need a working knowledge of MDPs, diffusion models, and policy-gradient or value-based RL to follow the derivations, though the qualitative examples and high-level framing are accessible to a broader safety-engineering audience.

Authors’ abstract

Reinforcement learning (RL) systems, while achieving remarkable success across various domains, are vulnerable to adversarial attacks. This is especially a concern in vision-based environments where minor manipulations of high-dimensional image inputs can easily mislead the agent's behavior. To this end, various defenses have been proposed recently, with state-of-the-art approaches achieving robust performance even under large state perturbations. However, after closer investigation, we found that the effectiveness of the current defenses is due to a fundamental weakness of the existing $l_p$ norm-constrained attacks, which can barely alter the semantics of image input even under a relatively large perturbation budget. In this work, we propose SHIFT, a novel policy-agnostic diffusion-based state perturbation attack to go beyond this limitation. Our attack is able to generate perturbed states that are semantically different from the true states while remaining realistic and history-aligned to avoid detection. Evaluations show that our attack effectively breaks existing defenses, including the most sophisticated ones, significantly outperforming existing attacks while being more perceptually stealthy. The results highlight the vulnerability of RL agents to semantics-aware adversarial perturbations, indicating the importance of developing more robust policies.