Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players

AI summary

Overview

Research area: AI research automation with LLM agents, applied to game AI — specifically a full reproduction of Hearthstone Battlegrounds and a reinforcement-learning agent that plays it on the live ladder. Technical level: Advanced (combines multi-agent LLM research orchestration, a Rust game engine, PPO training with a transformer policy, and a live-deployment loop). Scope (1 sentence): The paper asks whether LLM research agents can take a hard, real-world problem from raw data to a system that competes with humans, and answers it by having agents build a Battlegrounds simulator and a ladder-level playing agent in 20 days.

What This Paper Is About

Building a superhuman game AI has historically taken a large team of human experts who write the simulator, design the learning algorithm and debug every mismatch between the training world and the real game. This paper tests whether LLM research agents can do that work themselves on Hearthstone Battlegrounds, an eight-player auto-battler with stochastic shops and battles, hidden information and nearly a thousand interacting cards, for which no faithful simulator existed. The goal is a system that not only builds the environment and trains a strong policy, but keeps improving its own research process from feedback on the real live ladder.

Key Contributions

1. AutoTavern, a self-improving research system in which specialised LLM research agents (research lead, environment, training, opponent modelling, evaluation and audit, deployment, plus short-lived sub-agents) coordinate only through a typed, versioned research memory guarded by five verification gates. In 20 days it registered about 10,000 experiments, delivered 23 agent versions and used 418 human messages in total (0.6% of all messages).
2. TavernGym, described as the first faithful, full-content reproduction of the live game: all 978 game objects of patch 36.6.3, deterministic, running 190–280 complete eight-player games per second on a single core (3.6–5.3 ms per game), with 375 verification records and 220 confirmed rule decisions. The paper states it will be open-sourced.
3. TavernAgent, presented as the first Battlegrounds AI to play complete games on the live ladder and the strongest such agent. Its 23 successive versions raised the first-place rate against reality-calibrated opponents from 1.9% to 41.5%, and the deployed agent climbed the live ladder from a rating of 6,000 to about 15,000.
4. TavernBench, the first decision benchmark for Battlegrounds: 960 tasks in 40 scenarios across eight categories, labelled by the simulator, also slated for open source.

Main Findings

• The built system beats the builders: On TavernBench's 368-task comparison set, TavernAgent answers 91% of play tasks, against at most 63% for frontier LLMs playing directly (GPT-6 Astra 63%, Claude Sonnet 5.5 53%, Claude Opus 5.5 52%, GPT-6.1 Sol 46%, Gemini 3.8 Flash 44%, GLM-5.3 40%, Jev 37%, Intern-S2-Preview 32%, Kimi-K2.6 28%, DeepSeek-V4.1-Flash 26%, Grok 4.7 22%).
• Latency rules out live play for LLMs: TavernAgent decides in 0.03 s, while LLM medians run 1.4–26 s (DeepSeek-V4.1-Flash 0.91 s without reasoning, Jev 0.26 s). Within the one second a live turn allows, TavernAgent keeps 90%, Jev 37%, DeepSeek 26%, and every other LLM falls to 0%.
• Per-category strength: TavernAgent scores 100 on combat, 98 on positioning, 100 on ordering, 98 on targets, 64 on selection and 75 on strategy. Its policy answers 75% of turn plans scored by playing the game out, versus 54% for the best LLM.
• Reading rules is not playing: Frontier LLMs execute rules well (GPT-6 Astra 95%, Claude Opus 5.5 92%, Gemini 3.8 Flash 90%) yet fall far short on the play tasks built from those same rules. Giving an LLM TavernGym as a tool recovers much of the gap on combat (GPT-6 Astra from 62% to 97%).
• Training progress: On the main exam (one simulator build, three unseen seeds, 24,576 games per row), placement improved from 3.40 (TA-1) to 2.83 (TA-11) to 2.33 (TA-23), and first places from 1.9% to 41.5%, with top-four finishes from 83.4% to 89.0%. Baselines: random legal actions 7.97 placement and 0.0% first places; a scripted bot 7.54 and 0.0%.
• Reality-calibrated opponents were the largest single gain: From the same checkpoint and budget, calibrated opponent libraries raised first places from 20.9% to 27.2% on the main exam and from 10.3% to 13.5% on the harder one.
• Live ladder results: Over 392 games on days 5–18 the rating rose from 6,000 to 14,007, reaching about 15,000 (the server's top 100) by the end. Excluding 96 games flagged as client failures, average placement was 3.22 with 16.4% first places and 77.1% top-four finishes; the latest versions (TA-19–23) averaged 2.37 with 32.6% first and 91.3% top-four. On the public leaderboard's measure, the last 20 records average 1.60 with 80% first places, versus 2.34 and 59% for the server's ten highest-rated players.
• Catching errors changes the process: Six errors are reported. A search over copied games appeared to add 23.6 points of first-place rate but the copies carried the random-number state; reseeded per sample the gain was −0.65 ± 1.14 points. A winner's-curse effect (+1.2–2.1 points on selection seeds, −0.7 on unseen seeds) led to a re-check gate on unseen seeds. Version TA-10 nearly doubled its predecessor's exam first-place rate (24.2% vs 12.8%) yet the ladder rating fell from 8,968 to 7,658 within 32 games, 18 of which also had client failures.
• Live games drove 184 fixes: 35 to game rules and card behaviour plus 11 to card data and offer rules, 73 to input encoding, 51 to the client controller, and 14 opponent library recalibrations.
• Decision value: Removing one class of the agent's decisions costs 1.4–1.9 places for Dark Gifts, Discover choices and targets, about half a place for trinket choices and a quarter for the Hero Power.

Methodology in Plain English

The authors set up a team of long-lived LLM agents, each owning a research role, and let them work for 20 days. The agents never talk to each other directly: they read and write shared, versioned stores — a numbered ledger of findings (corrections are new entries pointing back), a registry of rule decisions and open questions, a hand-over note, a persistent list of lessons loaded at the start of every session, and an experiment registry. Nothing enters that shared state until it passes one of five gates: the code must pass the test suite, every model is tied to the simulator build it was trained on, models are compared only within one build over several seeds, a candidate is delivered only after a re-check on seeds it was not selected on, and delivered models must play the live ladder with their games replayed in the simulator.

Each iteration follows a fixed cycle: observe the memory, evaluations and recent live games; diagnose the largest gap; plan an intervention; implement it behind the test gate; train; evaluate and re-check; deploy to the ladder; then reflect by recording findings and converting caught mistakes into lessons or new gates. The system runs at three time scales — policy updates in seconds to minutes, experiments within hours, live games feeding back within days. The human enters at only two points: setting goals and ruling on mechanics the game data leave ambiguous, plus physically pressing the shortcut key the AI names in live games (a feature of the game's official plug-in).

The environment, TavernGym, is a Rust engine with a shared effect context for recruit and combat phases, one seeded random generator so a seed plus an action sequence reproduces a game bit for bit, and card data generated from the game's own files. The playing agent, TavernAgent, is an entity transformer (d=384, 8 layers, 8 heads, 18M parameters, up to 56 tokens) that represents each card by identity, static features and an embedding of its rule text, and outputs 245 masked actions plus value and placement heads. It is trained with PPO on about one million decisions per iteration, in stages of 300 iterations (about 310 million decisions, 570 thousand games), against "library seats" built from top-human statistics and recalibrated to the opponents the agent actually met on the client.

Why This Matters

Research impact: The paper provides evidence for a specific claim about LLM agents — that they build far better than they play. It also proposes a design for making agent-driven research trustworthy: typed memory, refutation-oriented verification gates, and a reality loop that re-anchors every proxy (simulator, opponent model, exams) to the real game. Its six documented failure modes, including future leakage in copied games and two errors cancelling out, are concrete hazards for anyone using agents to run experiments.

Real-world applications:

• Game studios: automated construction and rule-by-rule validation of simulators for live-service card and strategy games, and opponents calibrated to real player populations.
• Reinforcement learning pipelines that must correct simulation-to-reality gaps, via replaying live logs inside the simulator.
• Real-time decision support where a decision budget is under a second, such as in-game assistants that must respond within a turn.
• Benchmarking of LLM agents on stochastic, multiplayer, partially observable tasks with simulator-verified labels rather than text-only puzzles.

Industry relevance: The work shows an end-to-end loop where LLM agents produce a 63k-line Rust engine, roughly 10,000 registered experiments and 23 gated model releases with only 418 human messages, which is directly relevant to teams trying to automate ML engineering and environment building. The latency result is also a practical constraint for deploying LLMs in any interactive setting: models that answer correctly but in seconds cannot be used inside a live turn.

Future Directions

• Transfer beyond Battlegrounds: Whether reality-anchored self-improvement works for other games or non-game domains that also expose a slow, costly ground-truth signal is left open; the paper's evidence comes from a single title and patch (36.6.3).
• Scaling human oversight and gate design: The study used 418 human messages over 20 days, two of the six caught errors became gates and two became standing rules. How much of the remaining human role (goal setting, rulings on ambiguous mechanics, key presses) can be automated, and how new gates should be generated, remain open.
• Closing the LLM latency and compounding-error gap: TavernAgent keeps 90% accuracy within one second while no LLM exceeds 26%, and the authors argue per-decision errors compound over a turn rather than averaging out. Whether faster or tool-augmented LLMs can approach live-turn viability is an open question.
• Sustained improvement and breadth of evaluation: The authors report no sign of a plateau at about 15,000 rating; how far the loop scales, and how TavernBench should be extended beyond its 960 tasks in 40 scenarios, are natural next steps. Limitations and ethics are discussed in Appendix D.

Target Audience

Researchers and engineers working on LLM agents for scientific or ML research automation; reinforcement learning practitioners interested in simulation fidelity and sim-to-real correction; game AI researchers, especially those working on stochastic multiplayer card games; and evaluation designers who need benchmarks with simulator-verified labels and real-time constraints. Readers should be comfortable with RL training loops and multi-agent system design to get the most from the methodology sections.

Authors’ abstract

Can LLM agents carry a hard real-world problem from raw data to a system that holds its own against people? We study this in Hearthstone Battlegrounds, an eight-player auto-battler with stochastic shops and battles, hidden information and nearly a thousand interacting cards. We present AutoTavern, a self-improving research system: specialised LLM research agents share a typed research memory, every claim passes verification gates, and live games close the loop, so the agents improve the simulator, opponents and policy and, from the same feedback, their own research process; a human sets goals and rules on ambiguous mechanics. In 20 days AutoTavern built TavernGym, the first faithful, full-content reproduction of the live game, and delivered 23 versions of TavernAgent, trained with PPO against opponents calibrated to real games. TavernAgent plays complete games on the live ladder by itself: its rating rose from 6,000 to about 15,000 and, on the public leaderboard's measure, its last 20 records average a placement of 1.60 with 80% first places, better than the server's ten highest-rated players on every statistic (2.34, 59%). TavernBench, 960 tasks in 40 scenarios, shows that the system the agents built is the strongest player: TavernAgent answers 91% of the play tasks against at most 63% for frontier LLMs, and decides in 0.03 s where they need up to 26 s; within the second a live turn allows, it keeps 90% and no LLM more than 26%. LLM agents build far better than they play. We will open-source TavernBench and TavernGym at https://github.com/MCG-NJU/AutoTavern.