Adversarial Attack on Black-Box Multi-Agent by Adaptive Perturbation

AI summary

Overview

Research area: Security and reliability evaluation of multi-agent reinforcement learning (MARL) systems, specifically adversarial attacks against multi-agent systems (MAS) under a strict black-box threat model.

Technical level: Advanced. The paper assumes familiarity with multi-agent Markov Decision Processes, soft actor-critic (SAC) reinforcement learning, generative adversarial imitation learning, and gradient-based adversarial example generation (C&W attacks).

Scope: This paper proposes AdapAM, a learning-based framework that adaptively selects one victim agent and one anticipated malicious action per time step, then uses proxy agents trained by generative adversarial imitation learning to produce observation perturbations that induce the victim to execute that malicious action in a black-box MAS.

What This Paper Is About

Multi-agent systems, where several agents cooperate using decentralized decision-making and shared information, are increasingly deployed in domains such as unmanned aerial vehicles, industrial robots, and auto-driving, which makes their security worth testing. Existing attack frameworks on MAS are limited: many assume white-box access to internal networks or action-control authority over target agents, and others attack all agents or a fixed agent, sacrificing stealthiness or effectiveness. AdapAM's goal is to degrade a victim MAS's performance effectively while perturbing only a single, adaptively chosen agent and remaining hard to detect.

Key Contributions

1. A novel black-box adversarial attack framework for MAS that learns an adaptive policy to select the adversary agent and the anticipated malicious action at each time step.
2. A proxy-agent-based perturbation generation method, in which proxy agents approximate the victim MAS to supply the white-box information (gradients, logits) needed to generate perturbations that would otherwise be unavailable in a black-box setting.
3. An experimental evaluation of attack performance and stealthiness across eight multi-agent tasks, showing AdapAM outperforms four state-of-the-art and commonly used baselines, along with demonstrations of the effects of adversary and malicious action selection.

Main Findings

• Best attack performance overall: Across eight environments and different perturbation rates, AdapAM "achieves the best attack performance," measured by reductions in Reward and Win Rate, compared against four baselines. Table 1 reports results at a 100% perturbation rate.
• Complete degradation in smaller systems: In environments where the number of agents is less than 10, AdapAM drives the Win Rate down to 0% and outperforms all baselines.
• Two exceptions: In SMAC-bane_vs_bane and SMAC-27m_vs_30m, AdapAM is slightly weaker than MASafe. The paper attributes MASafe's performance there to it being the only baseline that perturbs all agents, while AdapAM perturbs only one, and notes AdapAM "does not fall far behind."
• Stronger against robust systems: When attacking the Robust Target MAS (trained with the robustness-enhancing method ROMANCE), AdapAM achieves a more significant attack effect in all cases than when attacking the Normal Target MAS.
• Smallest perturbations: Table 2 reports L-infinity magnitude of perturbations (lower is better). AdapAM records 0.10 to 0.19 across the eight environments, compared with 0.31 to 0.40 for MASafe, 0.20 to 0.29 for AMCA, and 0.15 to 0.23 for Lin. AMI was not measured this way because it directly manipulates agent actions rather than adding perturbations.
• Hardest to detect: Table 3 reports detection F1 scores (lower is better) using the abnormal-action detection method of Kazari et al. 2023. AdapAM scores 0.36 to 0.71, versus 0.82 to 0.92 for MASafe, 0.57 to 0.76 for AMCA, 0.65 to 0.81 for AMI, and 0.61 to 0.79 for Lin.
• Stealthiness trade-off exposed: Although MASafe's attack performance is nearly comparable to and in one case better than AdapAM's, the paper reports MASafe's stealthiness is the worst, because it applies perturbations to all agents and does not account for stealthiness.

Methodology in Plain English

AdapAM operates under a strict black-box setting: the attacker can only observe the victim MAS's observations and actions, and can only perturb observations, not commands. The framework has two parts.

First, an adaptive selection policy decides, at each time step and given the current environmental state, which agent to victimize and which malicious action that agent should be pushed to take. Rather than learning both choices jointly (which the authors say creates too large a search space and makes convergence difficult), the policy is split hierarchically into two classifiers: one that picks the victim agent from the state, and one that picks the malicious action from the state concatenated with a one-hot vector of the chosen agent. The policy is trained with SAC, using two critic networks and two target critic networks and a target-network weight parameter of 0.005. The attacker's reward is defined as the negative of the global reward, so the policy is optimized to reduce the victim MAS's reward. Critic and policy updates follow a soft Bellman residual and an expected KL-divergence objective, with a weighting parameter alpha balancing exploration.

Second, proxy-based perturbation generation supplies the white-box information that the black-box setting lacks. For each agent, a proxy policy is trained with Multi-Agent Generative Adversarial Imitation Learning (MAGAIL), pairing a discriminator that distinguishes the target agent's observation-action pairs from the proxy's with a generator that becomes the proxy policy. Once the proxy approximates the target agent's observation-to-action mapping, the attacker uses the proxy's gradients and logits with the C&W attack technique to search for a perturbed observation that makes the target agent output the desired malicious action. The search minimizes the distance between the original and perturbed observation (measured with Euclidean distance) plus an objective term, defined over the second-to-last layer's logits, that measures whether the input produces the target action. The paper states this method can misguide the target agent with a 100% success rate.

Experiments use three benchmarks with eight environments in total: StarCraft Multi-Agent Challenge (four environments), Google Research Football (two environments), and Multi-Agent Particle Environments (two environments). Four baselines are compared: MASafe, AMCA, AMI, and Lin. Normal Target MAS policies are trained with QMIX; Robust Target MAS policies are trained with ROMANCE. Perturbation rate is defined as the percentage of time steps in each episode at which the attack is performed; Table 1 reports the 100% case. MPE-spread and MPE-reference do not define win or loss, so only reward is reported there.

Why This Matters

Impact on research. The paper argues that prior MAS attack work relies on unrealistic assumptions (white-box access, action-control authority) or makes poor stealthiness/effectiveness trade-offs by attacking all agents or fixed agents. AdapAM reframes the problem as two learned subproblems, victim-and-action selection and perturbation generation, and shows that attacking a single adaptively chosen agent can approach or match the effect of attacking every agent while adding much smaller and less detectable perturbations. It also contributes an evaluation protocol that measures stealthiness both by perturbation magnitude and by detection F1 score.

Real-world applications:

• Unmanned aerial vehicles and drone swarms, which the paper cites as a domain where MARL algorithms are playing increasingly significant roles.
• Industrial robots, another cited deployment domain for multi-agent decision-making.
• Auto-driving, where the paper notes that in real-world multi-agent autonomous driving a driver can seize control to override an autonomous agent's outputs.
• Camera-based agents, since the paper states the deployment-phase observation manipulation can be achieved by adding a patch as a perturbation to an agent's camera.

Industry relevance. Organizations deploying cooperative multi-agent policies (robotics fleets, vehicle platoons, distributed control) need to know how much a single compromised observation channel can degrade a whole team, and whether existing detection systems would catch it. The reported detection F1 scores for AdapAM being the lowest among the compared methods suggest that current abnormal-action detection may not reliably flag this style of attack.

Future Directions

• Scaling to larger and more diverse scenarios. The paper's own stated future work is extending AdapAM beyond the eight evaluated environments; notably, the two environments where AdapAM did not achieve optimal performance (SMAC-bane_vs_bane and SMAC-27m_vs_30m) are described as having a large number of agents.
• Designing robust defense MAS. The authors state they plan to use insights from AdapAM to design multi-agent systems that are robust against this class of attack, which the current paper does not provide.
• Improving detection. Because AdapAM reports the lowest detection F1 scores under the compared detector, an open question is whether stronger abnormal-action detectors can be built without losing the stealthiness advantage, and how detection would perform at the lower perturbation rates shown in the reward/win-rate sweeps.
• Closing the remaining performance gap. AdapAM's single-agent perturbation still trails MASafe's all-agent attack in two high-agent-count environments; whether better victim-selection or action-selection policies can close that gap under the same stealthiness constraint remains unresolved.

Target Audience

This paper benefits researchers and practitioners in multi-agent reinforcement learning security and robustness evaluation, adversarial machine learning, and AI safety testing. It is most useful to readers who already understand MARL training pipelines (QMIX, SAC), imitation learning, and adversarial example generation, and who want a threat model and evaluation methodology for black-box attacks that perturb observations rather than actions. Engineers responsible for deploying or red-teaming cooperative multi-agent policies in robotics, autonomous driving, or distributed control are a secondary audience, though the paper's technical machinery is dense for non-specialists.

Authors’ abstract

Evaluating security and reliability for multi-agent systems (MAS) is urgent as they become increasingly prevalent in various applications. As an evaluation technique, existing adversarial attack frameworks face certain limitations, e.g., impracticality due to the requirement of white-box information or high control authority, and a lack of stealthiness or effectiveness as they often target all agents or specific fixed agents. To address these issues, we propose AdapAM, a novel framework for adversarial attacks on black-box MAS. AdapAM incorporates two key components: (1) Adaptive Selection Policy simultaneously selects the victim and determines the anticipated malicious action (the action would lead to the worst impact on MAS), balancing effectiveness and stealthiness. (2) Proxy-based Perturbation to Induce Malicious Action utilizes generative adversarial imitation learning to approximate the target MAS, allowing AdapAM to generate perturbed observations using white-box information and thus induce victims to execute malicious action in black-box settings. We evaluate AdapAM across eight multi-agent environments and compare it with four state-of-the-art and commonly-used baselines. Results demonstrate that AdapAM achieves the best attack performance in different perturbation rates. Besides, AdapAM-generated perturbations are the least noisy and hardest to detect, emphasizing the stealthiness.