CausalXRL: Explainable Reinforcement Learning through Causal Graph Reasoning

AI summary

Overview

Research area: Explainable reinforcement learning (XRL), specifically causal inference and causal-graph-based explanation for model-free RL agents.

Technical level: Intermediate. The paper uses formal causal-inference machinery (Rubin Causal Model, BART, CATE), but the core ideas and the two-level explanation structure are describable without heavy math.

Scope: CausalXRL is a post-hoc framework that learns causal dependencies between human-interpretable state representations in order to produce global causal graphs and situation-specific local causal models of a trained RL agent's behavior, evaluated quantitatively on four benchmark environments and qualitatively with 8 RTS domain experts in MicroRTS.

What This Paper Is About

Deep reinforcement learning agents can outperform human experts, but their decision-making is opaque, which limits trust and makes debugging hard. Existing causal XRL methods such as the Action Influence Model (AIM) bind causal edges directly to actions and require either a predefined causal structure from domain experts or expensive conditional independence tests (CITs), which makes them impractical in multi-unit settings like real-time strategy games. CausalXRL aims to remove both constraints: it learns causal structure automatically from data over interpretable state representations rather than actions, and does so at a much lower computational cost.

Key Contributions

1. A three-stage XRL pipeline. CausalXRL is organized into Representation Extraction (mapping raw experience tuples into expert-specified interpretable representations), Causal Dependency Inference (Rubin Causal Model + BART to estimate treatment effects), and Explainable Causal Models (global causal graphs and local causal models).

2. An action-free causal formulation for multi-agent environments. Instead of binding causal edges to actions as AIM does, CausalXRL operates over human-interpretable representations, which the authors argue avoids the deterministic edges that arise in multi-unit RTS games and generalizes beyond single-agent settings.

3. A scalable structure-learning algorithm. Causal discovery uses estimated Conditional Average Treatment Effects with percentile-based significance checks (a link is added if P₂.₅ of the CATE samples is above 0 or P₉₇.₅ is below 0), with a worst-case complexity of O(n²) versus O(n^{k+2}) for CIT-based methods.

4. A two-part evaluation, including an expert study. A quantitative comparison of behavior-prediction fidelity and structure-learning runtime on four benchmarks (CartPole, LunarLander, PongDuel, MicroRTS-Lite), plus a qualitative think-aloud study with 8 RTS experts using an interactive browser interface on MicroRTS. Code is released at https://github.com/yanmluk/causal-xrl.

Main Findings

• Behavior prediction is competitive in simple environments and clearly better in complex ones. Against linear models (LM) and decision trees (DT), CausalXRL reached 91.4% accuracy on CartPole (LM 88.5, DT 94.1), 88.7% on LunarLander (LM 81.3, DT 85.1), 90.2% on PongDuel (LM 81.5, DT 88.3), and 86.0% on MicroRTS-Lite (LM 75.3, DT 80.3). CausalXRL did not beat DT on CartPole.

• F1-scores follow the same pattern. 91.6 on CartPole (LM 90.0, DT 94.6), 86.6 on LunarLander (80.2 / 84.7), 90.7 on PongDuel (81.1 / 88.4), and 85.9 on MicroRTS-Lite (74.5 / 82.2).

• Structure learning is far faster than CIT baselines. Average runtime over five runs to build the global causal graph: 0.1 s vs PC 1.7 s and FCI 0.8 s on CartPole (4 representations); 0.3 s vs 12.6 s and 5.1 s on LunarLander (8); 0.5 s vs 23.2 s and 8.7 s on PongDuel (12); 0.5 s vs 20.9 s and 7.4 s on MicroRTS-Lite (14).

• The gain grows with the number of representations. The authors attribute the widening accuracy and efficiency gaps in PongDuel and MicroRTS to the larger representation count (#Rep), which is where the O(n²) advantage over CIT methods matters most.

• Expert behavior-prediction accuracy improved with local causal models. Participants with access to local causal models answered 8/8 multiple-choice behavior-prediction questions correctly, versus 4/8 in the control condition, which had feature-level SHAP values and state navigation instead.

• Explanation and trust ratings were high. Most explanation metrics (Madumal's understandability, satisfaction, sufficiency) for global and local models received average ratings above 4 on a 5-point Likert scale, and all trust metrics (confidence, predictability, reliability) for local causal models scored above 4.

• Feedback loops were a comprehension stumbling block. One expert (E5) initially read the mutual negative loop between blue Heavy and blue Ranged units as a contradiction, and only understood it as a production trade-off after being told to compare effect magnitudes. The authors report this as a sign that users may need guidance translating graph structure into counterfactuals.

• A worked counterfactual example. In one MicroRTS state, the local model showed the blue Ranged edge into red Heavy units (−0.88) as slightly stronger than the blue Heavy edge (−0.55); the chain Blue Barrack → Blue Ranged → Red Heavy is presented as a strategy that the ensuing gameplay confirms.

• Graphs are behavior surrogates, not mechanistic accounts. The authors state explicitly that the learned models are intended to be behaviorally faithful to the policy over collected trajectories, not to recover the agent's internal neural computations.

Methodology in Plain English

The pipeline works in three steps.

1. Turn raw game data into human-readable concepts. Raw observations are too large or too dense to reason about directly (images in Atari, 16×16×27 tensors in MicroRTS, or dozens of joint angles in MuJoCo Humanoid). So a mapping function converts each experience tuple into a small vector of expert-specified interpretable representations — for MicroRTS, the count of each unit type, such as the number of allied Ranged units or enemy Barracks. These are collected by replaying episodes and stored whenever the representation vector changes, forming a representation pool.

2. Estimate whether one concept causally moves another. Every ordered pair of representations is treated as a potential (treatment, outcome) pair, with all other representations plus a time-step feature acting as covariates to control for confounders. Bayesian Additive Regression Trees (BART) fit the outcome surface, and the causal effect is read off as the difference in predicted outcome between a treated value and a control value (default 1 vs 0) — a Conditional Average Treatment Effect. A directed edge is kept only if the 2.5th percentile of the sampled effects is above zero or the 97.5th percentile is below zero; the authors describe this as a deliberately conservative rule that suppresses false positives. They mention agglomerative clustering of covariate–CATE pairs as a more flexible alternative that can surface subtler, heterogeneous effects.

3. Present two levels of explanation. The global causal graph iterates this procedure over all pairs to give an overall map of how the agent's strategic variables influence each other across episodes. A local causal model reuses the already-trained BART models, conditioned on a single state, to produce a situation-specific graph whose edge weights quantify how much and in which direction one variable would have to change to move another — supporting counterfactual "what if" reasoning. The time-step covariate is used during training but hidden from local visualizations to avoid misreading.

Evaluation. Trained agents were run for 200 episodes, and the transitions served as held-out data. For behavior prediction, each parent of the action node in the local model gets its own BART model, and their action predictions are combined with weights equal to the local edge strengths. Baselines were linear models and decision trees predicting actions directly from state features with no causal constraints. AIM-style baselines were deliberately excluded because their explanatory target (per-action influence on state transitions) differs from CausalXRL's (dependencies among strategic state representations). Runtime was measured against PC and FCI, chosen over the score-based GES because CIT methods produce more consistent graphs.

Why This Matters

Impact on research. The paper attacks two concrete bottlenecks in causal XRL — the need for a predefined causal structure and the cost of conditional independence testing — and replaces both with a treatment-effect estimation loop that scales quadratically. It also reframes the explanatory target from actions to strategic state representations, which is what makes multi-unit, multi-agent settings tractable. The reported expert study adds qualitative evidence to a literature where such evidence is sparse.

Real-world applications:

• Game AI and playtesting: surfacing an opponent AI's high-level strategy (e.g., the Blue Barrack → Blue Ranged → Red Heavy chain in MicroRTS) so designers and players can understand it without hours of observation.
• Human–autonomy teaming: the paper explicitly frames trust, predictability, and confidence as goals for settings where humans supervise autonomous agents.
• Healthcare decision support: the introduction names healthcare as a high-stakes domain where opacity blocks adoption.
• Robotics and autonomous driving: both are listed among the domains where RL has been applied and where interpretability is a barrier.
• Debugging model-free agents: local causal models give developers per-state, direction-and-magnitude feedback rather than a single saliency heatmap.

Industry relevance. Teams deploying RL in games, simulation, or safety-adjacent products need explanations that are cheap enough to run and structured enough to act on. The sub-second graph construction times reported here (0.1–0.5 s versus 0.8–23.2 s for the CIT baselines) are the kind of difference that determines whether explanation is a design-time tool or an interactive one.

Future Directions

• Reducing dependence on expert-specified representations. The authors list this first among their limitations: CausalXRL needs suitable human-interpretable abstractions to exist, which may not hold in new domains. Semi-automated concept discovery, which the paper notes still requires human grouping, pruning, and alignment, is the obvious gap to close.

• Scaling causal discovery and visualization. The discussion names scaling as a limitation (the text is truncated at this point), leaving open how the approach behaves as the representation count and graph density grow well beyond the 14 representations tested.

• Making feedback loops and counterfactuals easier to read. E5's confusion over the mutual negative loop suggests a need for interface or notation support that helps users translate graph structure into gameplay implications without experimenter clarification.

• Statistically powered user studies. The authors state the 8-participant study was exploratory and not designed for statistically powered comparisons, so quantitative human-subject evidence on explanation quality and trust remains to be established.

• Testing heterogeneous treatment effects. The clustering alternative to the conservative percentile rule is proposed but not evaluated, leaving open whether it recovers real dependencies that the current threshold discards.

Target Audience

RL and XRL researchers who want a causal-inference alternative to saliency- and attribution-based explanations; practitioners building or debugging agents in multi-agent and real-time strategy environments; HCI and human-factors researchers interested in how domain experts read causal explanations and where they misread them; and game AI engineers who need explanation tooling fast enough to use interactively. Readers without any causal-inference background will need to look up the Rubin Causal Model, CATE, and BART to follow Section 4, but the framing, results table, and expert study are accessible on their own.

Authors’ abstract

Reinforcement learning is a powerful paradigm for training autonomous agents and has achieved impressive performance in complex environments. However, this success often comes at the cost of interpretability, diminishing trust and complicating efforts to debug and improve agent behavior. To address these challenges, we introduce CausalXRL, a novel framework for explainable reinforcement learning (XRL). A key feature of CausalXRL is its use of causal graph reasoning, which provides transparent, structured, multi-level explanations of agent decision-making. We validate CausalXRL through comprehensive case studies and a two-part evaluation: (1) a quantitative analysis of explanation fidelity and causal-structure learning efficiency in benchmark RL environments, and (2) a qualitative expert study assessing explainability in the real-time strategy (RTS) benchmark MicroRTS. The quantitative results show that CausalXRL can provide faithful explanations while efficiently learning causal structures, and the qualitative expert study suggests that participants found CausalXRL useful for inspecting high-level RTS strategies.