CHDP: Cooperative Hybrid Diffusion Policies for Reinforcement Learning in Parameterized Action Space

AI summary

Overview

Research area: Deep reinforcement learning (DRL) for hybrid (parameterized) action spaces, combining diffusion-based generative policies, multi-agent reinforcement learning (MARL), and vector-quantized representation learning.

Technical level: Advanced. The paper assumes familiarity with diffusion models, diffusion Q-learning (DQL), parameterized action MDPs, vector quantization, and actor-critic training.

One-sentence scope: The paper proposes CHDP, a two-agent cooperative framework in which a discrete diffusion policy and a continuous diffusion policy jointly produce hybrid actions, supported by a sequential update scheme and a Q-guided codebook, and reports state-of-the-art success rates on eight hybrid-action benchmarks.

What This Paper Is About

Many real-world control problems require an action that is a pair: a categorical choice plus a set of continuous parameters (for example, picking which tool to use and then how much force to apply). Existing methods struggle with this hybrid action space for two reasons: their policies are unimodal (Gaussian or deterministic), so they cannot represent situations where several different action pairs are equally good, and they do not scale when the number of discrete choices grows combinatorially. CHDP's goal is to capture multi-modal hybrid action distributions while remaining scalable in high-dimensional discrete action spaces.

Key Contributions

1. A cooperative two-agent MARL formulation of the hybrid action problem. CHDP treats the single-agent parameterized action decision as a fully cooperative game between a discrete agent (policy π_θd) and a continuous agent (policy π_θc), both built on diffusion policies, so that the expressiveness of diffusion can be used in each action subspace. The continuous policy is conditioned on the discrete action's representation to explicitly model the dependency between the two.

2. A sequential update scheme to prevent policy conflicts. Because simultaneous updates of cooperating policies can conflict, CHDP decomposes joint optimization into a turn-based sequence: the discrete policy is updated first (with the continuous action drawn from the replay buffer as a fixed target), and the continuous policy plus codebook are then updated conditioned on the newly updated discrete policy, with a stop-gradient operator blocking gradients back into the discrete policy.

3. A Q-guided codebook for scalability. Inspired by VQ-VAE, CHDP builds a learnable codebook E_ζ ∈ R^(K × d_e), where K is the cardinality of the discrete action space and each codeword e_k ∈ R^(d_e), and embeds the high-dimensional discrete action space into a low-dimensional latent space. The discrete policy outputs a continuous latent vector that is quantized to its nearest codeword, whose index becomes the discrete action and whose embedding conditions the continuous policy.

4. State-of-the-art empirical results. CHDP reports improvements over the prior leading method of up to 19.3% in success rate on challenging hybrid action benchmarks, and demonstrates a learned multi-modal policy in a qualitative experiment.

Main Findings

• Overall benchmark performance: Across eight standard PAMDP benchmarks — Platform, Goal, Catch Point, Hard Goal, and four Hard Move variants — CHDP reports the highest mean success rate in every environment, improving over the prior state-of-the-art method by up to 19.3%. Results are mean success rates (± one standard deviation) over 5 independent trials, each trial itself the average of its final 5 evaluations.

• Expressiveness on Hard Goal: CHDP reaches 79.5% ± 5.0 on Hard Goal, compared to 60.2% ± 5.0 for HyAR-TD3, 43.0% ± 10.0 for PA-TD3, 1.2% ± 0.7 for HHQN-TD3, 0.0% ± 0.0 for PDQN-TD3 and HPPO.

• Strong results on Platform and Catch Point: CHDP achieves 99.7% ± 0.2 on Platform (HyAR-TD3: 96.6% ± 2.2; PDQN-TD3: 96.7% ± 4.1; PA-TD3: 95.1% ± 3.6; HPPO: 66.3% ± 0.9; HHQN-TD3: 56.7% ± 29.4) and 93.8% ± 0.6 on Catch Point (HyAR-TD3: 86.6% ± 0.9; PDQN-TD3: 89.8% ± 3.2; PA-TD3: 86.7% ± 8.3; HPPO: 55.7% ± 5.2; HHQN-TD3: 23.7% ± 5.5).

• Scalability across growing discrete action spaces: In the Hard Move suite, the discrete space is 2^n (n actuators each on/off), so n=4 gives 16 discrete actions, each with its own continuous parameters, and n=8 gives 2^8 = 256 options. CHDP reports 94.2% ± 1.7 (n=4), 93.9% ± 1.0 (n=6), 90.6% ± 2.2 (n=8), and 79.8% ± 5.4 (n=10), versus HyAR-TD3 at 91.4% ± 2.4, 92.3% ± 0.6, 88.3% ± 1.9, and 69.0% ± 5.6 respectively. Other baselines degrade much more severely at larger n (for example, PA-TD3 falls to 4.6% ± 2.1 at n=8 and 10.3% ± 2.0 at n=10). The paper states CHDP maintains a success rate exceeding 90% for n up to 8.

• Sample efficiency: Learning curves show CHDP rising more steeply than all baselines, converging faster and to higher final performance, notably in Platform, Catch Point, and the Hard Move suite.

• Ablation — diffusion policy matters: Replacing the diffusion policies with deterministic ones drops Hard Goal performance from 81.8% ± 3.5 to 49.8% ± 26.9, and Hard Move (n=6) from 94.4% ± 1.2 to 88.2% ± 5.6. The deterministic variant's 49.8% on Hard Goal still beats several baselines but is far from state of the art.

• Ablation — codebook matters most for scalability: Removing the codebook (replacing it with an argmax selection over raw outputs) in Hard Move (n=6), which has 2^6 discrete actions, collapses performance from 94.4% to 56.1%; on Hard Goal the drop is from 81.8% to 73.3% ± 7.8.

• Ablation — sequential update matters most for coordination: A variant that updates both policies concurrently using replay-buffer data (as in MADDPG) drops Hard Goal from 81.8% to 53.5% ± 14.5, while Hard Move (n=6) only drops from 94.4% to 92.3% ± 4.5.

• Ablation — synergy: Removing both the codebook and the sequential update yields 52.4% ± 13.7 on Hard Goal and 54.6% ± 9.6 on Hard Move (n=6). Ablation defaults are diffusion steps N=15, η=10, and d_e=8, which the paper notes are distinct from the Table 1 configuration.

• Qualitative evidence of multi-modality: In a single-step Hard Move (n=6) setup with the target placed just above the agent's fixed start, CHDP uses at least three distinct actuator combinations across 100 trials: base direction (-0.5, -0.866) at 79.0% frequency with continuous action -0.897 ± 0.089; (0.0, -0.866) at 17.0% with -0.703 ± 0.309; and (0.75, 0.443) at 4.0% with 0.945 ± 0.070. HyAR selects a single mode (base direction (0.75, 0.433)) in 100.00% of trials with a continuous action of 0.445 ± 0.000. The Gaussian baseline HPPO is excluded from this comparison because it failed to solve the task.

Methodology in Plain English

CHDP reframes one hard decision as a two-player team. Instead of a single network that must output a discrete choice and a continuous value at once, two separate diffusion policies divide the labor. The first policy proposes a latent vector for the discrete choice; that vector is matched to its closest entry in a learned "codebook" of embeddings, and the matching index becomes the discrete action. The matched embedding is then handed to the second policy as extra context, so the continuous parameters are generated knowing which discrete choice was made.

The codebook exists to keep the discrete problem small. Rather than learning over an enormous set of raw discrete actions, the discrete policy learns in a compact latent space of codewords, which limits the damage from combinatorial blow-up.

Training proceeds in a fixed order rather than simultaneously. First the discrete policy is improved using a diffusion loss plus a term that rewards higher Q-values, with the continuous action taken as a fixed, detached value from the replay buffer so it acts as a stable target. Then the continuous policy and the codebook are updated together using the freshly updated discrete policy's output, with gradients from the Q-function flowing into the continuous policy and into the selected codeword, but a stop-gradient blocking them from flowing back into the discrete policy. Both the codeword embeddings and the discrete policy's latent outputs are therefore steered by the same Q-value signal and align implicitly, and the meaning of each codeword is defined by its downstream usefulness rather than by reconstruction. Critics are trained with a Mean-Squared Bellman Error loss under the Double Q-learning paradigm.

Why This Matters

Impact on research. The paper targets a structural limitation rather than a single benchmark: prior hybrid-action methods have generally traded scalability against expressiveness (for example, HyAR's latent-space scalability with a deterministic policy, versus HyDo's expressive continuous diffusion policy that inherits PDQN-style scalability limits). CHDP argues those two properties can be pursued together by combining a Q-guided codebook with diffusion policies. It also imports ideas from heterogeneous-agent MARL — sequential updates with stop-gradients — into the single-agent hybrid-action setting, and reframes vector quantization as a task-driven, Q-value-guided representation rather than a reconstruction-driven one that requires pre-training.

Real-world applications the paper points to:

• Robotic manipulation, including selecting a specific tool and then regulating force magnitude.
• Non-prehensile manipulation, where the discrete choice is which of many contact points to use, each with its own continuous parameters.
• Autonomous driving, where a movement mode is selected and then speed or similar parameters are tuned.
• Soccer-like and game AI tasks, where a goal can be scored by distinct strategies (the paper's example: a left- or right-foot shot) each with unique continuous parameters such as force and angle.

Industry relevance. Systems that must both choose among many discrete options and fine-tune continuous values are common in robotics, autonomous driving, and game AI. The reported sample efficiency — converging in fewer environment interactions — and the reported ability to remain above 90% success for discrete spaces up to 256 options speak directly to the practical cost of training and to operating in large action libraries.

Future Directions

• Behavior at larger discrete spaces: The paper reports CHDP at 79.8% ± 5.4 for Hard Move (n=10), the largest reported configuration; whether the codebook approach continues to hold as n grows further is not established by the reported results.
• Separating the two ablated mechanisms: The ablations show the codebook is decisive for scalability (Hard Move n=6) while the sequential update is decisive for coordination (Hard Goal). Whether either mechanism can be redesigned to cover both failure modes is unaddressed.
• Why the sequential update helps only in some tasks: The paper hypothesizes that the sequential update matters when the bottleneck is policy coordination and matters little when the bottleneck is action-space dimensionality, but reports no direct test of that hypothesis beyond the two ablation environments.
• Whether the alignment between codeword embeddings and discrete-policy latents can be measured directly: The paper argues the two are implicitly aligned via a shared Q-function, but the reported evidence is performance and behavioral, not a direct measurement of that alignment. The paper does not report the codebook cardinality-to-dimension tradeoff across values of d_e, nor the sensitivity of results to the number of diffusion steps beyond the ablation default of N=15.

Target Audience

Reinforcement learning researchers and graduate students working on hybrid or parameterized action spaces, diffusion-based policies, or multi-agent coordination mechanisms. It is also relevant to practitioners in robotics, autonomous driving, and game AI who must deploy controllers that combine discrete mode selection with continuous parameter tuning and who are evaluating whether diffusion policies and learned action codebooks are worth the added complexity. Readers without a background in diffusion models or actor-critic RL will find the paper demanding.

Authors’ abstract

Hybrid action space, which combines discrete choices and continuous parameters, is prevalent in domains such as robot control and game AI. However, efficiently modeling and optimizing hybrid discrete-continuous action space remains a fundamental challenge, mainly due to limited policy expressiveness and poor scalability in high-dimensional settings. To address this challenge, we view the hybrid action space problem as a fully cooperative game and propose a \textbf{Cooperative Hybrid Diffusion Policies (CHDP)} framework to solve it. CHDP employs two cooperative agents that leverage a discrete and a continuous diffusion policy, respectively. The continuous policy is conditioned on the discrete action's representation, explicitly modeling the dependency between them. This cooperative design allows the diffusion policies to leverage their expressiveness to capture complex distributions in their respective action spaces. To mitigate the update conflicts arising from simultaneous policy updates in this cooperative setting, we employ a sequential update scheme that fosters co-adaptation. Moreover, to improve scalability when learning in high-dimensional discrete action space, we construct a codebook that embeds the action space into a low-dimensional latent space. This mapping enables the discrete policy to learn in a compact, structured space. Finally, we design a Q-function-based guidance mechanism to align the codebook's embeddings with the discrete policy's representation during training. On challenging hybrid action benchmarks, CHDP outperforms the state-of-the-art method by up to $19.3\%$ in success rate.