Hybrid-AIRL: Enhancing Inverse Reinforcement Learning with Supervised Expert Guidance

AI summary

Overview

Research area: Reinforcement learning / inverse reinforcement learning, with applications to imperfect-information game playing (poker).

Technical level: Advanced. The paper assumes familiarity with Markov decision processes, maximum-entropy IRL, generative adversarial training, and KL-divergence-based objectives.

Scope: The paper evaluates Adversarial Inverse Reinforcement Learning (AIRL) on Heads-Up Limit Hold'em (HULHE) poker, finds it inadequate, and proposes Hybrid-AIRL (H-AIRL)—an extension that adds supervised loss terms and stochastic regularization to both the policy and discriminator objectives.

What This Paper Is About

Reinforcement learning requires a well-designed reward function, but in complex domains such as poker the reward is sparse and only arrives at the end of each hand, which makes learning effective strategies hard. Inverse reinforcement learning (IRL) tries to solve this by inferring a dense reward function from expert demonstrations, but the paper shows that the state-of-the-art adversarial IRL method, AIRL, fails to extract a sufficiently informative reward function in the highly stochastic, partially observable HULHE poker setting. The goal of the work is to fix that failure by adding direct supervised guidance from expert data into the adversarial IRL training loop.

Key Contributions

1. A new method, Hybrid-AIRL (H-AIRL). It augments AIRL's policy loss with a supervised term (cross-entropy or KL divergence between the policy's action distribution and the expert's) weighted by a factor α, and augments the discriminator loss with a mean-squared-error term that regularizes the learned reward against the ground-truth environment reward, weighted by a factor β.
2. A stochastic regularization mechanism. Gaussian noise is injected into policy actions along the mini-batch axis, with a per-sample standard deviation that decays monotonically from the first sample (σ_start) to the last (σ_end) in each batch. This is designed to counter the overfitting that occurs when the supervised term makes the policy produce expert-like actions too quickly, leaving the discriminator with a weak adversarial signal.
3. The first application of IRL to poker. The authors state that no prior work has applied inverse reinforcement learning to capture the reward incentives of Heads-Up Limit Hold'em poker, and that this is the first work to fuse a supervised loss component into an IRL framework. The approach is inspired by techniques from conditional generative adversarial networks (Odena et al., 2017), but adapted to IRL.
4. A multi-faceted evaluation protocol. The paper assesses both policy alignment with experts (state-level action alignment) and the quality of the learned reward function itself (by training separate RL agents on the inferred reward), plus a two-dimensional visual analysis of the learned reward in MountainCar and a 1,000,000-tournament head-to-head comparison in poker.

Main Findings

• AIRL fails in HULHE poker. AIRL-DQN competed against RLCard's default DQN agent over 1,000,000 tournaments across 20 random seeds and produced a payoff of −693 ± 34 mbb/h (milli-big-blinds per hand), with p < 10⁻¹⁰, meaning it consistently loses to the baseline.
• H-AIRL-DQN wins in the same setting. It produced a payoff of +96 ± 14 mbb/h with p < 10⁻¹⁰, outperforming AIRL-DQN and performing competitively against the default DQN. The paper notes that professional poker players consider 50 mbb/h a sizable margin. Significance was assessed using the Bonferroni correction with a threshold of p < 0.0025.
• Better and faster expert alignment. Across discrete-action Gymnasium benchmarks (Acrobot, LunarLander, MountainCar, and HULHE poker), the H-AIRL policy achieved substantially better state-level action alignment throughout learning, approached expert behavior more quickly and accurately than AIRL, and showed reduced variance.
• Better or equal RL training on inferred rewards. When the learned reward functions were fed into PPO or DQN agents, H-AIRL-derived rewards gave a better or otherwise equal ability to approach expert learning compared to AIRL-derived rewards across all Gymnasium tasks and poker.
• Reward visualization shows a qualitative difference. In MountainCar, AIRL's learned reward was strongly biased toward "thrust right," with no region of the state space where "no thrust" was most valuable, so the agent never learned the back-and-forth switching needed to build momentum. H-AIRL's reward was well balanced, with appropriate regions for each action, allowing the agent to reproduce the expert's oscillatory strategy.
• Small amounts of supervision help; full supervision hurts. In one-factor-at-a-time sweeps on MountainCar over 10 independent runs, a relatively small policy supervision weight (α ≈ 0.1) gave sufficient guidance and substantial gains over AIRL, but α = 1 eliminated the adversarial game and hurt performance. Similarly, a moderate discriminator supervision weight (β ≈ 0.25) improved the learned reward, but β = 1.0 caused degradation.
• Reward supervision is optional. The poker results were achieved without any reward supervision (β = 0 in that case).
• A trade-off between convergence speed and reward fidelity. The value of α that yields the highest policy return may not coincide with the value that produces the best performance when the inferred reward is used to train a separate RL agent.
• Caveat on the poker data. In online poker datasets, folding actions reveal no information about the player's hand because folded cards remain concealed, so those actions are absent from the data; the IRL agent must focus on the remaining actions (calling, raising, and checking).

Methodology in Plain English

The researchers start from AIRL, which frames IRL as a game between a discriminator and a policy: the discriminator tries to tell expert state-action pairs apart from policy-generated ones, and the policy is trained to maximize the reward the discriminator implies while staying exploratory (entropy-regularized). The authors use AIRL's state-action formulation rather than the state-only variant, because the state-only version requires access to the actual next state, which is not feasible in offline or partially observable domains such as poker.

To build H-AIRL, they keep the adversarial machinery but add two extra learning signals. First, they blend into the policy loss a supervised imitation term that directly pushes the policy's action distribution toward the expert's actions, controlled by a weight α. Second, when ground-truth environment rewards exist for the expert demonstrations, they blend into the discriminator loss a mean-squared-error term that pulls the learned reward toward the true reward, controlled by a weight β (set to 0 when true rewards are unavailable). Third, to stop the supervised term from making the policy too expert-like too fast and starving the discriminator of useful signal, they add decaying Gaussian noise to the policy actions across each mini-batch, so the discriminator sees a spectrum of action quality from strongly perturbed to nearly noise-free.

Training has two stages: first the IRL phase alternating discriminator and policy updates, then an RL phase in which an agent (PPO from Stable-Baselines3 for Gymnasium, or DQN from RLCard for poker) is trained using the learned dense reward. Experts for the Gymnasium tasks came from a PPO agent (MountainCar exceptionally used DQN), and poker demonstrations came from the IRC Poker dataset, which contains millions of real-world poker game state observations including over 1 million state-action pairs for HULHE, covering expert amateurs and professionals, including a world champion. Learning curves report the mean and standard deviation across 10 training runs per algorithm.

Why This Matters

This work shows that a standard, well-regarded IRL algorithm can fail badly in a genuinely hard real-world domain, and it offers a simple, reusable recipe for fixing it. The recipe—mixing a supervised imitation signal into adversarial IRL and regularizing the discriminator with decaying noise—does not require abandoning the adversarial framework, only augmenting it. For the IRL research community, this is a concrete demonstration that adversarial reward inference can be stabilized with direct supervision, and the paper is the first to apply IRL to poker at all.

Real-world applications suggested or implied by the work:

• Poker and other imperfect-information games. Opponents and training agents that infer dense strategic reward signals from logged human play rather than relying on terminal game payoffs.
• Autonomous driving and robotics. Learning reward functions from human demonstration logs where the reward is sparse and delayed, and the environment cannot be faithfully simulated for next-state access.
• Epidemic control. A domain the paper cites as an area where deep RL has shown strong performance and where reward design is difficult.
• Any offline, partially observable decision setting with expert logs. The IRC Poker dataset case generalizes to any domain with large historical interaction datasets and missing information (such as unobservable actions).

Industry relevance: The methods matter to teams that have large corpora of expert behavior logs but no clean reward signal—for example, game studios, simulation and forecasting groups, and companies building decision-making agents from recorded human operators. The code is released at https://github.com/silue-dev/hairl.

Future Directions

• Understanding the α trade-off. The paper's own results show that the α value maximizing policy return need not maximize downstream reward fidelity; a principled way to choose α (or to balance both objectives) remains open.
• Extending the ablation on stochastic regularization. The paper introduces σ_start and σ_end as core hyperparameters and presents sweeps for them (Figure 5c and 5d), but the discussion of these results is not fully available in the text provided here, so the conclusions about noise scheduling remain to be characterized.
• Handling missing information in expert data. In HULHE, folding actions and concealed cards are absent from the dataset, so the agent cannot learn from them. Methods for reasoning about unobserved actions or hidden state in IRL would broaden applicability to real-world logs.
• Reducing reliance on ground-truth rewards. The discriminator supervision term requires ground-truth environment rewards; the poker results were achieved without it, which raises the question of how far H-AIRL can be pushed in fully reward-free settings and in continuous action spaces (where the paper notes a mean-squared-error loss would be used for the supervised policy term).

Target Audience

This paper is most useful to reinforcement learning and imitation learning researchers, particularly those working on inverse reinforcement learning, reward shaping, and adversarial imitation methods. It is also relevant to practitioners applying RL in imperfect-information or partially observable domains with large offline expert datasets—game AI engineers, simulation and control researchers, and teams working on decision-making from human interaction logs. Readers need a solid grounding in MDPs, maximum-entropy IRL, and GAN-style training to follow the derivations, although the high-level contribution and the empirical results are accessible to a broader machine learning audience.

Authors’ abstract

Adversarial Inverse Reinforcement Learning (AIRL) has shown promise in addressing the sparse reward problem in reinforcement learning (RL) by inferring dense reward functions from expert demonstrations. However, its performance in highly complex, imperfect-information settings remains largely unexplored. To explore this gap, we evaluate AIRL in the context of Heads-Up Limit Hold'em (HULHE) poker, a domain characterized by sparse, delayed rewards and significant uncertainty. In this setting, we find that AIRL struggles to infer a sufficiently informative reward function. To overcome this limitation, we contribute Hybrid-AIRL (H-AIRL), an extension that enhances reward inference and policy learning by incorporating a supervised loss derived from expert data and a stochastic regularization mechanism. We evaluate H-AIRL on a carefully selected set of Gymnasium benchmarks and the HULHE poker setting. Additionally, we analyze the learned reward function through visualization to gain deeper insights into the learning process. Our experimental results show that H-AIRL achieves higher sample efficiency and more stable learning compared to AIRL. This highlights the benefits of incorporating supervised signals into inverse RL and establishes H-AIRL as a promising framework for tackling challenging, real-world settings.