SEBA: Sample-Efficient Black-Box Attacks on Visual Reinforcement Learning

AI summary

Overview

Research area: Adversarial machine learning and reinforcement learning security — specifically black-box adversarial attacks against agents that learn control policies from raw pixels (visual RL).

Technical level: Advanced. The paper assumes familiarity with MDPs, Q-learning/TD updates, GANs, and world models.

Scope: The paper introduces SEBA, a framework that generates visually subtle adversarial perturbations against image-based RL agents using only query access to the victim policy, and evaluates it on five pixel-based MuJoCo continuous-control tasks and three Atari games.

What This Paper Is About

Visual RL agents learn to control robots and other systems directly from camera images, but small, carefully crafted pixel changes can make them fail. Most prior adversarial attacks on RL assume either low-dimensional vector states or discrete action spaces (like Atari), and those that work on images tend to need enormous numbers of environment interactions.

SEBA targets the harder setting: continuous-action, image-based control under a strict black-box constraint, where the attacker can query the victim policy for actions but cannot see its gradients, parameters, or internals. The goal is to degrade the victim's cumulative reward while keeping perturbations imperceptible and keeping environment/victim queries low.

Key Contributions

1. A formulation of sample-efficient black-box attacks on visual RL in continuous control, which the authors state is the first work to investigate visual RL attacks in continuous control tasks (prior visual attacks were limited to discrete-action Atari).

2. The SEBA framework, combining a shadow Q (critic) model that estimates cumulative reward under perturbation without access to the victim's internals, a GAN-based generator/discriminator pair for perceptually realistic perturbations, and a learned world model that produces synthetic rollouts to cut real-environment queries.

3. A two-stage alternating optimization scheme that freezes the generator/discriminator while updating the shadow critic, then freezes the critic while updating the GAN, to avoid the instability that arises when both are trained jointly.

4. Comprehensive evaluation on pixel-based MuJoCo (DrQ-SAC victims) and Atari (Rainbow victims), covering attack effectiveness (reward), imperceptibility (FID), query efficiency, component ablations, and targeted attacks that steer individual action dimensions into a chosen range.

Main Findings

• Strongest rewards degradation on MuJoCo continuous control: SEBA produced the lowest rewards on all five MuJoCo tasks among all compared methods (image-space and adapted vector-state attacks). On Cheetah Run it reached 1.61 ± 3.97 versus 150.72 ± 49.52 for PGD; on Reacher Hard it reached 0.3 ± 0.9 versus 592.64 ± 116.13 for OPTIMAL. Clean (unattacked) rewards ranged from 718.15 ± 26.79 (Walker Run) to 944.28 ± 42.26 (Walker Walk).

• Best imperceptibility in the reported comparisons: SEBA recorded the lowest FID in the MuJoCo tables (62.43), compared with 110.97 (PGD), 78.05 (C&W), 118.01 (Square), 115.46 (Critic-Based), 106.34 (MAD), 97.55 (PA-AD), and 93.04 (OPTIMAL).

• Zero victim queries at attack execution time: SEBA reports Atk. Vic (per-step) = 0, whereas SimBA needed 400 and Square 202 queries per step in Table 1, and Critic-Based needed 20 and MAD 11 in Tables 1–3.

• Large reduction in training-time interaction: SEBA used 160K total environment queries and 800K victim queries during training on MuJoCo, versus roughly 4M environment and 4M victim queries for PA-AD and OPTIMAL.

• Competitive on discrete-action Atari: with Rainbow victims, SEBA reduced Freeway reward from 34 to 10, Pong from 21 to 3, and Alien from 8858 to 982, outperforming Critic-Based, MAD, and OPTIMAL, while PA-AD achieved the strongest overall degradation (e.g., 6 on Freeway, −21 on Pong, 443 on Alien) using full gradient access. SEBA also had the lowest Atari FID (81.7) and used 80K environment and 400K victim training queries versus 2M/2M for PA-AD and OPTIMAL. The Section 4 overview describes the Atari transfer as using PPO agents, while Section 4.3 and Table 3 specify Rainbow as the victim agent; the provided content does not reconcile this.

• Ablations isolate each component's role: removing the discriminator (-D) raised FID from 62.43 to 97.18 with only a small change in reward; removing perturbed states in Stage 1 (-Noise) caused the largest performance drop (for example, Walker Walk went from 35.74 to 118.11); removing the world model (-WM) gave comparable or slightly stronger attacks but raised environment queries from 160K to 800K.

• Targeted attacks succeed at high rates: SEBA reached 96.6% success on Cheetah Run (dimension 4 of 6, target [0.3, 0.5]) versus 60.8% for PGD and 68.1% for Critic-Based; 91.3% on Walker Walk versus 37.2% and 53.2%; 95.6% on Walker Run; 93.2% on Reacher Hard; and 98.8% on Hopper Stand.

Methodology in Plain English

The attacker is a generator network that takes a clean image observation and outputs a small pixel perturbation, clipped so that no pixel moves by more than 8/255 and the result stays within [0, 1]. Because the attacker cannot see the victim's gradients, it needs a substitute signal. SEBA builds a "shadow" Q model — a learned critic trained with temporal-difference updates on transition tuples collected by running the real victim policy on perturbed observations. This shadow critic estimates the cumulative reward the victim would earn, and the generator is trained to push that estimate downward.

To keep perturbations visually plausible, a discriminator tries to tell clean images from perturbed ones, and the generator is rewarded for fooling it — a standard GAN setup. The generator loss combines the discriminator term (for imperceptibility) with a term that minimizes the shadow critic's value (for attack strength), scaled by a weight of 1.

Training alternates between two stages: first, freeze the GAN and update the shadow critic on rollouts in the perturbed environment; second, freeze the critic and update the GAN using its stable guidance. To avoid paying for every interaction, the authors train an IRIS-style world model — a discrete image tokenizer plus an autoregressive Transformer that predicts future latent tokens and rewards — and use it to generate synthetic transitions. Each real interaction is paired with H = 4 synthetic transitions, which the paper says reduces real environment queries by roughly 1/H; periodic real queries are interleaved to correct model drift. Key settings include 200K world-model updates, 20 total training iterations, phase lengths of 5K each, and evaluation over 10 random seeds. For targeted attacks, the generator objective is modified to maximize the shadow critic subject to the victim's chosen action dimension falling inside a target interval.

Why This Matters

Impact on research: The paper argues that prior adversarial RL work was largely confined to vector states or discrete-action Atari, where the policy behaves like a classifier. By showing that a generator guided by a learned critic can attack high-dimensional continuous-action visual control efficiently, it opens a harder testbed for robustness research. The finding that PA-AD and OPTIMAL degrade sharply when the perturbation becomes a 3 × 84 × 84 ≈ 2 × 10^4-dimensional "action" also gives a concrete diagnosis of why RL-based attackers scale poorly to pixel space.

Real-world applications (drawn from the domains the paper cites):

• Robotic manipulation: pixel-driven policies used in manipulation could be silently degraded by small visual changes in the scene.
• Autonomous navigation: a navigation agent relying on camera input could be pushed off course or into failure by subtle observation-level perturbations.
• Visual control systems: general image-based controllers, including those used in embodied AI, need robustness evaluation under limited-access threat models.
• Safety auditing of pretrained agents: with no access to weights, auditors can still probe a deployed model's failure modes, which matches realistic deployment conditions where model internals are proprietary.

Industry relevance: The query-efficiency results matter operationally. SEBA requires zero victim queries at attack time and far fewer training-time interactions (160K vs 4M environment queries on MuJoCo; 80K vs 2M on Atari), which lowers the cost of red-teaming and makes robustness evaluation feasible where simulator rollouts or API calls are metered, slow, or restricted. The code is released at https://github.com/tairanhuang/seba.

Future Directions

• Reconciling the Atari victim setup: the Section 4 overview refers to PPO agents while Section 4.3 and Table 3 specify Rainbow; clarifying which policy was used, and reporting the other, would strengthen the cross-domain claim.
• Closing the gap to white-box PA-AD on Atari: PA-AD still achieves stronger degradation with gradient access, so improving black-box transfer in discrete-action settings remains open.
• Reducing world-model dependence and drift: the (-WM) ablation shows the world model mainly buys query efficiency rather than attack strength, and the paper notes periodic real queries are needed to correct model drift — how to minimize those corrections is unresolved.
• Beyond the tested benchmarks: the paper reports MuJoCo, Atari, ablations, and targeted attacks, but the additional results for other visual RL attacks, varying hyperparameters, and implementation details are said to be in Supplementary Material that is not included in the available content.

Target Audience

Researchers and practitioners in adversarial machine learning, reinforcement learning security, and embodied AI who are evaluating or defending image-based control policies. It is most useful to readers already comfortable with RL objectives, TD learning, and GAN training; newcomers will find the conceptual story accessible but the notation dense. Security engineers tasked with red-teaming deployed vision-based controllers, and RL researchers studying robustness under black-box threat models, are the primary beneficiaries.

Authors’ abstract

Visual reinforcement learning has achieved remarkable progress in visual control and robotics, but its vulnerability to adversarial perturbations remains underexplored. Most existing black-box attacks focus on vector-based or discrete-action RL, and their effectiveness on image-based continuous control is limited by the large action space and excessive environment queries. We propose SEBA, a sample-efficient framework for black-box adversarial attacks on visual RL agents. SEBA integrates a shadow Q model that estimates cumulative rewards under adversarial conditions, a generative adversarial network that produces visually imperceptible perturbations, and a world model that simulates environment dynamics to reduce real-world queries. Through a two-stage iterative training procedure that alternates between learning the shadow model and refining the generator, SEBA achieves strong attack performance while maintaining efficiency. Experiments on MuJoCo and Atari benchmarks show that SEBA significantly reduces cumulative rewards, preserves visual fidelity, and greatly decreases environment interactions compared to prior black-box and white-box methods. The code is available at https://github.com/tairanhuang/seba online.