DSLE: A Learning Environment for Dark Souls Boss Encounters

AI summary

Overview

Research area: Reinforcement learning and game-playing agent benchmarks; environment design for real-time, high-dimensional commercial games.

Technical level: Intermediate. The paper assumes familiarity with reinforcement learning terminology (PPO, DQN, CMA-ES, observation and action spaces, reward shaping), but the environment design and results are described in largely concrete terms.

Scope: This paper introduces DSLE, a containerized benchmark that exposes all 22 boss encounters of Dark Souls: Remastered through a Gymnasium-style interface, and reports baseline results on a five-boss subset.

What This Paper Is About

Most game benchmarks for AI agents expose structured, purpose-built state information, and many modern commercial games offer no such interface at all. The authors build a system that turns an unmodified commercial action-role-playing game into a repeatable benchmark, with automated resets, memory-level state reading, and headless parallel execution. They then measure how hard that benchmark actually is by running five different agent types against a representative subset of five bosses.

Key Contributions

1. DSLE, a containerized benchmark for all 22 bosses of Dark Souls: Remastered. Every encounter is verified to load, reset, and step through the same interface, with observation, action, reward, and termination configuration unified across bosses.

2. A defined starter suite, DSLE-5. The authors select five bosses — Asylum Demon (tutorial melee), Capra Demon (spatially constrained arena), Chaos Witch Quelaag (environmental hazards), Ornstein and Smough (multi-target), and Gwyn, Lord of Cinder (fast final boss) — as a lower-cost representative target for developing and comparing methods before committing to the full 22-boss set.

3. A five-method baseline evaluation on DSLE-5. A random policy, a scripted expert system, PPO and DQN trained from visual input, and the neuroevolutionary method SCOPE are compared under a fixed protocol on the same suite.

4. An explicit stochasticity characterization and scalability report. A replay test quantifies how quickly identical action sequences diverge, and the paper reports memory, disk, and wall-clock costs for running parallel instances.

Main Findings

• Only the tutorial boss is ever defeated. On Asylum Demon, the scripted expert system wins 63 of 100 episodes and SCOPE wins 1065 of 5000 episodes (21%). PPO and DQN win under 0.4% of episodes (5 of 1971 and 8 of 2469 pooled over five seeds respectively). The random policy never wins.

• The four harder DSLE-5 bosses hold every method to zero wins. PPO, DQN, SCOPE, the expert system, and the random policy all record 0 wins on Capra Demon, Chaos Witch Quelaag, Ornstein and Smough, and Gwyn across thousands of episodes each.

• SCOPE is the strongest reported neuroevolutionary policy on the Asylum Demon. At its best generation SCOPE reaches a 43% win rate, exceeding the 35% reported for the NEAT agent of O'Connor et al. (2025b) at the same best-generation granularity.

• PPO and DQN show no measurable learning within budget. Episode reward curves stay pinned at the death floor near −10 on every boss for the full 100,000-step budget, never trending upward.

• A single RL run already costs tens of wall-clock hours. One run takes 9–21 hours for PPO and 12–41 hours for DQN, with Capra Demon the most expensive because its short episodes force constant save-state reloads. Pooled over five seeds, PPO completes 1,684–6,423 episodes per boss and DQN between 2,284 and 12,549.

• Agents fail in distinguishable regimes. Reported through survival time and boss damage rather than win rate alone, the failure cases range from sub-10-second deaths in the cramped Capra and fast Gwyn fights to minute-long stalemates on Quelaag that inflict almost no damage — so long survival is not evidence of progress.

• Difficulty extends across the full 22-boss suite. A broader study running the evolutionary baseline across all 22 encounters under an advantaged all level-50 stat build yields wins on only a handful of additional early-game bosses and leaves the rest unwon.

• The environment is stochastic despite fixed save states. In a replay test on Asylum Demon executing the same actions (attack, move forward, attack, roll forward) for up to 200 steps, ten trials first diverged at step 12 and terminated at different points between 134 and 200 steps. Seeding agent-side randomness does not make game dynamics deterministic.

Methodology in Plain English

The authors wrap a running copy of Dark Souls: Remastered in a Docker container so that every environment step is a real action executed against the live game. The game runs on an Ubuntu base through Wine, with rendering provided by a virtual X11 display and DXVK translating DirectX calls to Vulkan. Each parallel game instance gets its own display number, VNC port, Wine prefix, save directory, desktop name, and XDG runtime directory so instances do not collide.

Instead of estimating game state through computer vision, DSLE reads process memory directly through the Linux /proc/<pid>/mem interface, resolving fixed base pointers for player, boss, event-flag, lock-on, and menu structures, and following boss-specific pointer chains. This gives reliable player health, boss health, death count, boss-defeated flags, and lock-on state for reward computation and termination checks.

Agents see a grayscale game frame (1×600×800, uint8) and choose from 14 discrete actions: four directional moves, a light attack, a strong attack, a heal, a backstep, four directional rolls, and two raw mouse-click actions for parrying. Each action is held for 250 ms by default, giving roughly four decisions per second of game time. Lock-on is handled automatically at a fixed interval so the action set stays focused on movement, attacks, healing, and dodging. Episodes terminate on boss defeat or player death and truncate at 7,200 steps.

Rewards combine a small step penalty, damage dealt, damage taken, a win bonus, and a death penalty: −0.001 + Δh_b − 0.25Δh_p + 100·1_win − 10·1_death. PPO and DQN receive no memory-extracted state — only the visual frames — while the memory reads remain in use for reward, termination, and logging. Both use the convolutional architecture from Mnih et al. (2015), with the grayscale frame downsampled to 84×84. SCOPE instead applies a two-dimensional discrete cosine transform to each grayscale observation and maps a sparse coefficient matrix to action preferences through a bilinear affine policy optimized with CMA-ES, producing a 1,514-parameter chromosome.

Why This Matters

Impact on research. DSLE addresses a real gap in the benchmark ecosystem: reproducible environments for real-time, third-person commercial games with close-range reactive combat, visual observations, and automated resets. Prior work like AlphaStar and OpenAI Five achieved strong play but depended on large-scale bespoke pipelines, and DSAPI supported only a single boss with vision-based state estimation. DSLE turns one such game into a standardized benchmark with 22 tasks, all verified to load and reset.

The negative results are themselves a contribution. The paper shows that gradient-based RL baselines are the weakest family tested here, beaten on an early boss by a derivative-free evolutionary policy and a hand-written expert system, even under a budget of tens of wall-clock hours per run.

Real-world applications:

• Industrial simulation and control. The pattern of an isolated, repeatable scenario drawn from a commercial application — with automated reset and direct process instrumentation — transfers to manufacturing, robotics, and teleoperation testbeds.
• Benchmark construction for high-fidelity software. The containerized, per-instance-isolated execution model is directly applicable to evaluating agents against real software rather than simplified simulators.
• Reward design research. The shaped-reward baseline (damage terms against sparse terminal bonuses) provides a documented case where shaping produced no learning, useful for studying reward design under sparse terminal signals.
• Robustness and evaluation methodology. The replay test showing trajectory divergence at step 12 is a reusable procedure for quantifying stochasticity in any nominally fixed environment.

Industry relevance. Game studios and benchmark designers can reuse the distribution model — the code ships the environment wrapper, automation, and benchmark, while the user supplies their own legally obtained copy of the game, following the same approach as the StarCraft II Learning Environment. AI labs working on agent evaluation gain a task family with a built-in difficulty gradient, where encounters vary in spatial layout, number of enemies, vulnerability windows, phase structure, hazards, and required strategy.

Future Directions

• Closing the learning gap. No published method defeats four of the five DSLE-5 bosses. Whether algorithmic advances, longer budgets, or different observation encodings can change this is left open.
• Explaining the failure modes. The paper reports that failure regimes can be inferred from survival time and boss damage, but the underlying causes — exploration difficulty, credit assignment over long horizons, or perception limits — are not resolved.
• Extending beyond DSLE-5. The encounter tags in Table 2 are explicitly offered as a way to construct additional boss suites targeting specific characteristics such as hazards, multi-target pressure, or objective completion, and the full 22-boss suite is the stated ceiling of the benchmark.
• Sample efficiency versus fidelity. The authors frame DSLE as scalable through parallelism rather than single-instance throughput, and note that it is not intended to match JAX-native environments. Reducing reset overhead and episode cost is a clear practical target.

Target Audience

This paper is most useful to reinforcement learning researchers building or evaluating game-playing agents, benchmark designers working with real-time or commercial software, and practitioners interested in neuroevolution and derivative-free optimization on high-dimensional visual input. It is also relevant to engineers who need to containerize interactive graphical applications for automated, parallel experimentation, and to anyone assessing whether current RL methods scale to sparse-reward, real-time control tasks.

Note: the provided paper content is truncated mid-sentence in Section 5.1, so the full details of Sections 5.2 and 5.3 are not available beyond what is summarized in the abstract and table.

Authors’ abstract

We introduce the Dark Souls Learning Environment (DSLE), a containerized platform that presents all 22 boss encounters of Dark Souls: Remastered as game-playing agent benchmarks through a Gymnasium-style interface. DSLE combines real-time combat, high-dimensional visual input, and sparse terminal rewards, with each environment step being a real action executed against the running game. To support controlled comparison, we define DSLE-5, a representative five-boss subset, spanning a melee fight, a spatially constrained arena, an environmental-hazard fight, a multi-target fight, and a fast final-boss fight, that we recommend as the starting suite for agents built on DSLE. On DSLE-5 we evaluate a random policy, an expert system, an evolutionary baseline, and PPO and DQN agents trained from visual input. The expert system and the evolutionary baseline each defeat the Asylum Demon, the game's tutorial boss (63% and 43% peak win rates), but none of the five methods defeats the other four DSLE-5 bosses; PPO and DQN show no measurable learning (at most 0.33% win rate on the tutorial boss, 0% elsewhere) within a budget that already costs tens of wall-clock hours per run. A broader study running the evolutionary baseline across all 22 encounters under advantaged all level-50 stats yields wins on only a handful of additional early-game bosses and leaves the rest unwon. The failure cases range from sub-10-second deaths in cramped, multi-target encounters to minute-long stalemates that inflict almost no damage, and we report them through survival time and damage dealt rather than win rate alone.