Robust Adversarial Reinforcement Learning in Stochastic Games via Sequence Modeling

AI summary

Overview

• Research area: Offline reinforcement learning (RL), sequence modeling with Transformers, adversarial and multi-agent robustness, and stochastic game theory.
• Technical level: Advanced. The paper assumes familiarity with the Decision Transformer, Q-learning, temporal-difference learning, expectile regression, and Nash equilibrium concepts.
• Scope: The paper proposes and empirically evaluates the Conservative Adversarially Robust Decision Transformer (CART), a method that conditions Transformer policies on expected minimax (NashQ) values so that they remain robust against optimal adversaries when state transitions in the game are probabilistic.

What This Paper Is About

Decision Transformer-style methods learn policies by conditioning on a target return, but they are vulnerable to adversarial opponents, and the earlier Adversarially Robust Decision Transformer (ARDT) only computes correct worst-case values when state transitions are deterministic. In stochastic games, where the transition kernel T(s' | s, a, ā) is probabilistic, ARDT becomes over-optimistic and can be fooled by rare, high-payoff trajectories that are unlikely to be reached. CART addresses this by relabeling training trajectories with a NashQ value that explicitly averages over possible next states before taking the max over the protagonist's next action.

Key Contributions

1. The CART framework. The authors introduce the Conservative Adversarially Robust Decision Transformer (CART), which they describe as the first framework designed to enhance the adversarial robustness of the Decision Transformer in adversarial stochastic games.
2. A stage-game formulation that accounts for stochastic transitions. At each stage the protagonist–adversary interaction is treated as a stage game whose payoff is the expected maximum value over subsequent states: Q̄(s, a, ā) = E_{s' ~ T(·|s,a)}[r + V(s')] with V(s') = max_{a'} Q(s', a'). This integrates over the transition probabilities instead of assuming a deterministic next state.
3. NashQ-based conditioning for the Transformer. The conditioning variable for training is z = Q_CART(s, a) = min_{ā} Q(s, a, ā), so the Transformer learns policies that are simultaneously less exploitable and conservative about transition uncertainty.
4. A practical estimation procedure. The minimax and maximization operations are approximated with expectile regression combined with temporal-difference learning, using three alternating objectives: a TD error for the stage-game payoff Q̄, an expectile-regression objective with α → 0 to estimate Q*(s, a) = min_{ā} Q(s, a, ā), and an expectile-regression objective with α → 1 to estimate V*(s') = max_{a'} Q(s', a'). Mean-squared error is used to learn terminal state values.

Main Findings

• Highest worst-case return under varying target returns. In the illustrative two-stage stochastic game, CART achieves the highest worst-case return when conditioning on various large target return values, while ARDT and DT obtain lower returns against an optimal adversary (left-hand side of Figure 3).
• Reported illustrative comparison. The table in Figure 1 lists returns against the optimal adversary as DT = 6, ARDT = 5.7, and CART (ours) = 8.0.
• Best average performance with lowest variation. Across 5 synthetic adversarial stochastic games, with a high target return set during decoding, CART obtains the highest worst-case return with the lowest variation (right-hand side of Figure 3).
• ARDT's failure mode. Because ARDT conditions only on minimax returns, it can be misled by rare and high-payoff trajectories, overestimating the values of state–action pairs and neglecting true transition probabilities, which can cost robustness at high target returns.
• Why CART is less exploitable. By jointly considering payoff and transition stochasticity, CART focuses on feasible strategies that maximize worst-case expected return, producing an adversarially robust policy.
• Illustrative game specifics. In the two-stage game, the worst-case expected returns for the protagonist's actions a0, a1, and a2 are 6, 8, and 8 respectively, making a1 or a2 the robust choices.
• Three-stage game. The authors also evaluate a three-stage stochastic game with two actions per player per stage and a fixed terminal reward observed only at the end of the episode; the performance comparison is presented in Figure 7. The paper text does not report numeric values for this comparison.
• Training-data setup. Data were collected with uniformly random actions for both protagonist and adversary, containing 10^5 trajectories and encompassing all possible trajectories; the reported metric is worst-case return against an assumed optimal adversary.

Methodology in Plain English

The authors work in an offline setting: the learning agent cannot interact with the environment and must learn only from a fixed dataset of previously collected trajectories. They start from the Decision Transformer idea of treating RL as sequence modeling, where the model predicts an action given the recent history plus a desired return, but they replace the usual return-to-go target with a worst-case-aware target.

The central move is to break the game down into single-stage games. At each step, the adversary is assumed to act after seeing the protagonist's action, so the adversary can directly counteract it. Rather than immediately taking the minimum over the adversary's action, CART first averages the payoff over all possible next states according to the transition probabilities, and adds the value of the best action available from that next state. Only then does it take the minimum over the adversary's action. That ordering is what makes the estimate conservative with respect to stochasticity instead of optimistic.

Since scanning a dataset to compute exact minima and maxima is impractical, the authors approximate these operations with expectile regression, a technique that can be tuned toward a lower or upper expectile of a distribution by varying a parameter α. They alternate three losses: one fits the stage-game payoff Q̄ with a temporal-difference error, one fits Q(s, a) using a low expectile (α → 0) to approximate the minimum over adversary actions, and one fits V(s') using a high expectile (α → 1) to approximate the maximum over protagonist actions. Terminal values are learned with mean-squared error. Once this converges, the resulting NashQ values are used as the conditioning variable z when training the Decision Transformer.

Evaluation uses a suite of synthetic stochastic games, including a two-stage game and several variants (a rare payoff adjusted to 100; transition probabilities of 20% and 80% to states 1 and 2; a target return of 8.2 with payoffs of 10 or 8 depending on adversary actions; and a combination of these properties), plus a three-stage game. The authors compare CART against ARDT and Decision Transformers, and note that their DT implementation removes conditioning on past adversarial tokens, following ARDT.

Why This Matters

Robustness of learned policies against an adversary that reacts to the agent is a hard problem, and most sequence-modeling RL work has addressed robustness to environmental stochasticity or corrupted data rather than to adversarial behavior. This paper argues that naively combining minimax reasoning with return-conditioned sequence modeling breaks down once transitions are stochastic, and it offers a concrete fix. The result is more accurate minimax value estimation and better worst-case returns on the games tested, which suggests a route toward sequence-modeling policies that do not simply exploit weak opponents in the training data.

Real-world applications suggested by the setting:

• Competitive and multi-agent systems, where an opponent observes and reacts to an agent's decisions.
• Security and network defense, where an adaptive attacker exploits predictable or over-optimistic policies.
• Poker-like and other strategic games, which the authors explicitly name as a future target (Kuhn and Leduc Poker).
• Any high-stakes decision setting with rare, high-payoff outcomes, where being misled by unlikely favorable events can produce fragile policies.

Industry relevance: the paper targets offline, data-driven policy learning, which fits settings where deploying an agent to explore online is expensive or risky, such as autonomy, robotics, trading, and strategic planning. It also speaks to practitioners who build on Decision Transformer checkpoints and need return conditioning that does not encourage unrealistic outcomes.

Future Directions

• Extend CART to more complex multi-agent and competitive environments, with the authors specifically naming poker variants such as Kuhn and Leduc Poker.
• Test whether CART can mitigate over-optimism, improve policy stability, and handle rare high-payoff events in those richer games.
• Evaluate the method on larger-scale games and longer planning horizons, since the reported experiments are on short-horizon stochastic games.
• Investigate generalizing beyond the synthetic two-stage and three-stage settings and beyond the offline uniform-behavioral-policy data collection regime used in the experiments.

Target Audience

This paper suits researchers and graduate students working on offline RL, decision transformers, and sequence modeling for sequential decision-making, as well as those interested in game theory and adversarial or multi-agent robustness. It is most useful to readers already comfortable with value-based RL and Transformer-based policy learning, since the contribution rests on NashQ learning, temporal-difference errors, and expectile regression. Practitioners applying Decision Transformer variants in competitive settings may also benefit, particularly those concerned with policies that look strong in training data but fail against a strong or adaptive opponent.

Authors’ abstract

The Transformer, a highly expressive architecture for sequence modeling, has recently been adapted to solve sequential decision-making, most notably through the Decision Transformer (DT), which learns policies by conditioning on desired returns. Yet, the adversarial robustness of reinforcement learning methods based on sequence modeling remains largely unexplored. Here we introduce the Conservative Adversarially Robust Decision Transformer (CART), to our knowledge the first framework designed to enhance the robustness of DT in adversarial stochastic games. We formulate the interaction between the protagonist and the adversary at each stage as a stage game, where the payoff is defined as the expected maximum value over subsequent states, thereby explicitly incorporating stochastic state transitions. By conditioning Transformer policies on the NashQ value derived from these stage games, CART generates policy that are simultaneously less exploitable (adversarially robust) and conservative to transition uncertainty. Empirically, CART achieves more accurate minimax value estimation and consistently attains superior worst-case returns across a range of adversarial stochastic games.