DRG-MAPPO: Hierarchical Dynamic Role-Graph Multi-Agent Reinforcement Learning for Cooperative Air Combat

AI summary

Overview

Research area: Multi-agent reinforcement learning (MARL) applied to cooperative multi-UAV air combat, combining graph neural network relational modeling with hierarchical reinforcement learning and dynamic role assignment.

Technical level: Advanced. The paper assumes familiarity with Dec-POMDPs, centralized training with decentralized execution (CTDE), PPO clipping, generalized advantage estimation, graph attention networks, and hierarchical policy optimization.

Scope: The paper proposes DRG-MAPPO, a hierarchical MARL framework that builds a graph-structured battlefield observation, assigns tactical roles through a high-level policy with temporal commitment, and executes discrete maneuver and weapon actions through a role-conditioned low-level policy in a 2v2 beyond-visual-range (BVR) simulation.

What This Paper Is About

Existing MARL approaches to cooperative air combat use flat architectures that treat the raw observation as a single feature vector and do not explicitly represent relationships among battlefield entities or the tactical roles agents should play. DRG-MAPPO addresses this by decomposing each agent's observation into a graph of self, ally, enemy, and missile nodes, then splitting decision-making into a high-level role policy (e.g., "leader" and "supporter") and a low-level maneuver policy conditioned on that role. The goal is to improve coordination quality, interpretability, and training stability in highly dynamic engagements.

Key Contributions

1. A hierarchical MARL framework for air combat that decouples tactical role assignment from low-level maneuver control, so that each agent's discrete actions are conditioned on an explicitly assigned role rather than emerging implicitly.

2. A graph-attention relational module that constructs a directed entity graph per agent at each timestep and extracts self-level, agent-level, and team-level relational features, plus a target-priority auxiliary task intended to foster behaviors such as focus-fire without reward shaping.

3. A temporal commitment mechanism that re-samples roles only at fixed intervals, preventing destabilizing role oscillation and enforcing behavioral consistency within each commitment window.

4. Empirical evaluation in a high-fidelity simulation reporting an 87% peak win rate and an average 69.9% win rate in a round-robin cross-play tournament against MARL baselines, alongside ablation and qualitative tactic visualizations.

Main Findings

• Peak win rate: DRG-MAPPO reaches a peak win rate of 87% over 2500 training episodes in the 2v2 scenario.
• Learning dynamics: All methods show a similar cold-start bottleneck during episodes 0–800; DRG-MAPPO begins to significantly outperform baselines after approximately 1250 episodes.
• Baseline comparison: MAPPO+GAT and HAPPO yield moderate improvements by adding graph structures or hierarchical decomposition alone, but fall short of the integrated DRG-MAPPO architecture. Standard MAPPO lacks the structural inductive biases for entity reasoning, while QMIX and IPPO perform worst, indicating that value decomposition and independent learning are insufficient for tight tactical coordination in adversarial BVR engagements.
• Average reward: Initial rewards are approximately -450 across methods, primarily from boundary breaches and missile attrition. Curves surpass the zero threshold around episode 400. DRG-MAPPO converges to roughly 950, while mid-tier methods plateau between 500 and 650.
• Cross-play generalization: In a round-robin tournament where each pair of policies competes for 200 independent episodes, DRG-MAPPO achieves an average win rate of 69.9%, outperforming all baselines head-to-head. Minor deviations from 0.5 on the diagonal are attributed to simulation stochasticity.
• Ablation: Four core modules are identified as jointly responsible for performance and tactical efficiency: graph-based modeling, dynamic role assignment, the target-priority auxiliary task, and temporal commitment. Removing temporal commitment is associated with high-frequency role oscillations, tactical inconsistency, and severe performance fluctuations.
• Emergent tactics: Visualized 2v2 episodes show a bait-and-flank tactic in which the Supporter deliberately maintains a forward trajectory to draw enemy fire while the Leader executes a lateral flanking maneuver; a coordinated focus-fire behavior in which both agents converge on a single adversary and launch missiles from complementary angles; and a dynamic transition of the Supporter from defensive to offensive posture as threat diminishes.

Methodology in Plain English

The researchers model the air combat problem as a Dec-POMDP with N cooperative agents, then build on MAPPO with parameter sharing: a shared actor acts on local observations, while a centralized critic sees the global state during training.

Instead of feeding each agent one long state vector, they convert the observation into a small directed graph. Nodes fall into four categories — self, ally, enemy, and missile — with features such as position, velocity, pitch/roll/yaw, remaining missiles, relative distances and angles, closing speed, aspect angle, and missile threat flags. Because the categories have different feature sizes, all node features are zero-padded to a common dimension and combined with a learned entity-type embedding before being projected through a two-layer MLP.

A scaled dot-product graph attention encoder then runs over the fully connected entity graph, producing query, key, and value projections, an attention matrix, and layer-normalized outputs with a feed-forward block. From the final layer, the authors derive three representations: a graph-refined self representation, a mean-pooled agent representation summarizing local battlefield topology, and a mean-pooled team representation shared across agents.

A high-level role policy maps these relational features plus a global context vector to a discrete role in {1, ..., K}, which is then converted into a learned role embedding rather than a one-hot vector. A temporal commitment mechanism re-samples the role only when the timestep is divisible by a commitment horizon T_role; otherwise the previous role persists. A low-level actor then selects one of 12 discrete actions covering maneuvers (heading and altitude changes, acceleration, deceleration, S-shaped oscillation, notch evasion) and missile launch, conditioned on the observation, graph features, global context, role embedding, and a one-hot agent identity.

To encourage cooperation without hand-crafted coordination rewards, an auxiliary head predicts a probability distribution over enemy targets by softmax over a two-layer MLP applied to the self, agent-level, and role features. Labels come from a heuristic priority score combining distance to the enemy, own and enemy radar lock status, and the number of friendly missiles already tracking that enemy; the lowest-scoring enemy is the primary target. This head shares the graph attention encoder and is trained with cross-entropy loss.

Training uses the Adam optimizer with a joint objective summing clipped actor and role losses, value MSE, auxiliary cross-entropy, and entropy bonuses for both policy levels. Role gradients are computed only at re-sampling intervals, using a distinct clipping coefficient. Rewards combine a team-shared combat term (win/loss, kill/death), a dense advantage term based on antenna train angle and relative distance, a threat-avoidance penalty based on incoming missile proximity, and a fixed penalty for leaving the operational boundary. The authors state that cooperative behaviors such as focus-fire are not explicitly rewarded.

Why This Matters

Impact on research: The work argues that combining structured relational modeling with explicit hierarchical role assignment addresses two limitations of flat MARL architectures — the inability to capture time-varying topological relations among battlefield entities, and the absence of tactical role modeling that leads to ambiguous task allocation. It offers a template for injecting domain structure into MARL without over-engineering the reward function.

Real-world applications:

• Multi-UAV swarm coordination in contested airspace, including tactics such as flanking, bait-and-switch, and cooperative suppression.
• Air combat training and simulation, where interpretable role labels ("leader", "supporter") and visualized tactics can support human review and instruction.
• Autonomous escort or defensive escort missions where one platform draws threat while another maneuvers for advantage.
• Target allocation and fire coordination in multi-agent weapon engagement, using the target-priority auxiliary head to reduce redundant missile expenditure.

Industry relevance: Defense and aerospace organizations developing autonomous air combat systems currently rely on expert rules, optimization approaches, or hand-crafted tactical indicators, which the paper notes struggle to scale in highly dynamic engagements. A learned policy that generalizes across opponents (as suggested by the 69.9% cross-play average) is directly relevant to simulation-based acquisition, doctrine development, and human-machine teaming studies. The C++ simulation with GTacview visualization used here reflects the tooling typical of defense simulation environments.

Future Directions

• Scalability: The authors state that future work will explore scaling the hierarchical paradigm to larger-scale swarm confrontations; the current evaluation covers only a 2v2 BVR scenario.
• Beyond 2v2 validation: Whether the reported win rate, reward plateau, and cross-play generalization hold in larger, messier engagements is not reported in this paper.
• Hyperparameter and architecture sensitivity: The role count K, the commitment horizon T_role, hidden dimensions, and the auxiliary loss and reward coefficients are referenced symbolically without reported values, leaving their sensitivity unexplored in the content provided.
• Opponent diversity: Generalization is tested against MARL baselines and a rule-based policy in the visualization; robustness against adaptive or human-designed adversaries beyond these is not reported.

Target Audience

This paper is most useful to MARL researchers working on hierarchical reinforcement learning, graph-based multi-agent relational modeling, or CTDE algorithm design; to aerospace and defense engineers building autonomous air combat and UAV coordination systems; and to graduate students or advanced practitioners who already understand PPO, Dec-POMDPs, and attention mechanisms and want a concrete example of combining graph encoders with hierarchical role assignment in an adversarial simulation.

Authors’ abstract

Multi-Agent Reinforcement Learning (MARL) has emerged as a pivotal paradigm for complex decision-making in autonomous systems and air combat. While MARL has demonstrated significant potential in air combat, achieving sophisticated tactical coordination remains a non-trivial challenge. This difficulty is largely attributed to two primary limitations: (1) the absence of structured relational modeling hinders agents from capturing complex, time-varying interactions among battlefield entities; and (2) conventional flat architectures often lack the capability to explicitly model tactical roles, leading to ambiguous task allocation in highly dynamic environments. To address these challenges, we propose Hierarchical Dynamic Role-Graph Multi-Agent Proximal Policy Optimization (DRG-MAPPO), a novel MARL framework that integrates graph-based relational modeling with dynamic role assignment. Specifically, DRG-MAPPO constructs a graph-based representation of battlefield interactions and leverages graph attention mechanisms to extract critical relational features among allies, enemies, and threats. Subsequently, a high-level policy employs a dynamic role assignment mechanism to determine tactical responsibilities (e.g., ``leader'' and ``supporter''). Conditioned on these roles and encoded graph-relational features, a low-level policy executes discrete maneuver actions, facilitating the joint optimization of tactical strategy and collaborative execution. Furthermore, a target-priority auxiliary task is designed to foster the emergence of behaviors such as focus-fire. Experimental results demonstrate that DRG-MAPPO achieves a state-of-the-art win rate of 87%, suggesting that our framework effectively balances relational modeling, interpretability, and optimization stability for cooperative air combat.