Extending NGU to Multi-Agent RL: A Preliminary Study

AI summary

Overview

Research area: Multi-Agent Reinforcement Learning (MARL), specifically intrinsic-motivation and exploration methods for sparse-reward cooperative tasks.

Technical level: Intermediate. The paper assumes familiarity with Q-learning, DQN, replay buffers and intrinsic rewards, but its structure is readable for anyone who knows the basics of reinforcement learning.

Scope: A preliminary empirical study that adapts the single-agent Never Give Up (NGU) exploration algorithm to a cooperative multi-agent pursuit–evasion task and tests three design choices for that adaptation.

What This Paper Is About

Sparse rewards make exploration hard in reinforcement learning, and in multi-agent settings the problem is compounded by credit assignment, non-stationarity and the need for coordinated exploration. The NGU algorithm solved several sparse-reward Atari games in the single-agent case by rewarding state novelty, and this paper asks whether that mechanism can be carried over to a team of cooperative agents. The goal is not a new algorithm from scratch, but a faithful, simplified port of NGU's core exploration machinery to multi-agent DQN agents, plus an analysis of how experiences and novelty signals should be shared across the team.

Key Contributions

1. A direct extension of NGU to cooperative multi-agent environments with sparse rewards, where each agent keeps its own Q-network, embedding network, episodic memory and intrinsic reward.
2. An empirical analysis of three design choices for the multi-agent version: shared versus individual replay buffers, sharing episodic novelty across agents via a threshold k, and assigning heterogeneous intrinsic/extrinsic trade-off values of β across agents.
3. Evidence that the multi-agent NGU variant (Multi-NGU) achieves moderately higher returns and more stable learning dynamics than a multi-agent DQN baseline in the simple_tag environment from PettingZoo, with the gains being largest when a shared replay buffer is used.
4. Concrete directions for future work on extending NGU to broader MARL settings and stronger benchmarks.

Main Findings

• Shared replay buffer wins. Multi-NGU with a shared replay buffer gives the best performance and stability across the configurations tested. The authors interpret this as the gains coming from combining NGU's intrinsic exploration with experience sharing.
• NGU beats Multi-DQN in both buffer settings. With individual replay buffers, the Multi-DQN baseline learned slowly and unstably and reached only modest returns, while the NGU-augmented agents showed higher average returns and smoother learning curves. Multi-DQN also benefited from the shared buffer but still lagged behind the NGU variants.
• Novelty sharing is comparable only at k = 1. Sharing episodic novelty across agents performs comparably to standard Multi-NGU when k = 1, but degrades learning for larger k values, with k = 2 and k = 3 underperforming according to Appendix B.
• Curves cross over. Multi-novelty with k = 1 rises faster early in training, but standard Multi-NGU eventually surpasses it and achieves higher long-term returns.
• Heterogeneous β does not help. Varying the intrinsic/extrinsic trade-off across agents (for example {0.1, 0.2, 0.4}) did not improve over a small common value; the heterogeneous variants still beat the baseline but consistently trailed standard Multi-NGU.
• Smaller heterogeneous values are more stable. Among heterogeneous assignments, the set with smaller β values showed more stable learning and higher returns than the larger set (0.2, 0.4, 0.6).
• β = 0.1 was best. A search over β ∈ {0, 0.1, 0.5, 1.0} found β = 0.1 gave the most stable learning and highest average returns.

Methodology in Plain English

The researchers took NGU and kept only the parts that drive exploration, deliberately dropping Random Network Distillation (RND) and Universal Value Function Approximators (UVFA) to reduce complexity and computational cost. What remains is an embedding network that compresses raw observations into a compact representation, an inverse dynamics model trained to predict which action was taken between two consecutive embeddings, and an episodic memory that stores embeddings seen during the current episode. Novelty is measured as a k-nearest-neighbor distance in that embedding space, and it is converted into an intrinsic reward that is added to the extrinsic reward, scaled by a parameter β.

Each pursuer is a DQN agent augmented with these components, and every agent has its own Q-network, embedding network, episodic memory and intrinsic reward. The evader uses the default heuristic policy provided by PettingZoo. Experiments run in simple_tag_v3, where rewards are sparse and shared: when any pursuer tags the evader, all pursuers get the same reward. Two scenarios are compared (no shared replay buffer, and shared replay buffer), and within each one four configurations are run under identical conditions: the Multi-DQN baseline, Multi-NGU, the novelty-sharing variant, and the heterogeneous-β variant. All agents share the same architecture and hyperparameters, every configuration runs for 200,000 timesteps over 15 independent runs with identical random seeds, and results are reported as smoothed returns with a window of 100 and 95% confidence intervals.

Why This Matters

Impact on research. Multi-agent sparse-reward problems are typically attacked with increasingly elaborate architectures that add computational overhead. This paper shows that a simplified version of an existing single-agent exploration method can improve stability and returns in a cooperative MARL task, and it isolates which specific design choices (buffer sharing, novelty sharing, β assignment) actually matter. That is useful evidence for researchers deciding whether added architectural complexity is justified.

Real-world applications (as motivated by the broader interest in the area, not tested in this paper):

• Multi-robot coordination, such as teams of robots searching for or intercepting a target with infrequent success signals.
• Traffic and logistics control, where multiple agents must coordinate and rewards arrive only at the end of a successful outcome.
• Game AI and simulation, including pursuit–evasion and adversarial training scenarios.
• Any cooperative control setting with delayed or sparse feedback where the reward only materializes when a team objective is completed.

Industry relevance. The result that a shared replay buffer produced the best performance speaks directly to practical system design: pooling experience across agents is cheap, and this paper suggests it is the more valuable change compared to sophisticated novelty-sharing schemes. The negative results on heterogeneous β and on larger k values also save practitioners from chasing configurations that did not pay off in these experiments.

Future Directions

• Evaluate more configurations within the simple_tag environment, since this study covers only a single environment and a single learning algorithm (DQN).
• Test the approach on a broader range of multi-agent settings, from simpler coordination tasks to large-scale competitive scenarios, to assess generality and scalability.
• Swap DQN for other algorithms, such as value decomposition methods (VDN, QMIX) or policy-gradient approaches (MADDPG, MAPPO).
• Compare Multi-NGU against stronger benchmarks, including large-scale environments such as the StarCraft Multi-Agent Challenge, for a more comprehensive evaluation of robustness and scalability.

Target Audience

Reinforcement learning researchers and graduate students working on exploration, intrinsic motivation or multi-agent systems, particularly those interested in sparse-reward cooperative tasks. It is also relevant to practitioners who want a practical starting point for adding exploration bonuses to a multi-agent DQN pipeline, since the implementation is publicly available and the paper reports the exact hyperparameters used. Readers seeking a fully developed new MARL algorithm, a large-scale benchmark comparison, or results beyond a single environment will find this to be a preliminary study rather than a final answer.

Authors’ abstract

The Never Give Up (NGU) algorithm has proven effective in reinforcement learning tasks with sparse rewards by combining episodic novelty and intrinsic motivation. In this work, we extend NGU to multi-agent environments and evaluate its performance in the simple_tag environment from the PettingZoo suite. Compared to a multi-agent DQN baseline, NGU achieves moderately higher returns and more stable learning dynamics. We investigate three design choices: (1) shared replay buffer versus individual replay buffers, (2) sharing episodic novelty among agents using different k thresholds, and (3) using heterogeneous values of the beta parameter. Our results show that NGU with a shared replay buffer yields the best performance and stability, highlighting that the gains come from combining NGU intrinsic exploration with experience sharing. Novelty sharing performs comparably when k = 1 but degrades learning for larger values. Finally, heterogeneous beta values do not improve over a small common value. These findings suggest that NGU can be effectively applied in multi-agent settings when experiences are shared and intrinsic exploration signals are carefully tuned.