TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents

AI summary

Overview

Research area: AI agents, Large Language Model evaluation, reinforcement learning benchmarks, real-time strategy game environments.

Technical level: Intermediate. Readers should be comfortable with LLM agent concepts, reinforcement learning terminology (reward signals, action spaces, PPO), and standard benchmark evaluation practices.

Scope: The paper introduces TowerMind, a lightweight tower defense game environment and benchmark for evaluating LLMs on long-term planning, decision-making, and hallucination, positioned as a cheaper alternative to StarCraft II-based benchmarks.

What This Paper Is About

Evaluating whether LLMs can plan and make decisions over long horizons usually requires complex, costly game environments such as StarCraft II, which demand tens of gigabytes of disk space, dedicated GPUs, and large memory footprints. Existing lightweight alternatives lack the textual observation interfaces that LLMs need, leaving a gap between computational cost and LLM compatibility. TowerMind fills this gap by providing a tower defense game environment that is small enough to run on a CPU, offers pixel, textual, and structured observations, and measures both task performance and the frequency of invalid (hallucinated) actions.

Key Contributions

1. A lightweight, multimodal tower defense environment. TowerMind requires only 0.15 GB of disk space and RAM, runs on CPU without a GPU, and exposes pixel-based, textual (JSON), and structured one-dimensional observations through a standard OpenAI Gym interface.

2. A dual-metric benchmark for capability and reliability. The environment scores performance as in-game reward (enemies stopped) and separately tracks a valid action rate to quantify hallucination, defined as actions that contradict the game state or rules.

3. Five progressively difficult benchmark levels plus a level editor. The paper formalizes a difficulty metric combining road count, tower point count, enemy composition, and resource scarcity, and provides a graphical editor plus parameter and feature customization to reduce data contamination risk.

4. An evaluation of seven LLMs and two RL baselines. Commercial and open-source models are compared against a human expert baseline across language-only and vision-language modalities, alongside Ape-X DQN and PPO trained for 100 million environment steps.

Main Findings

• Large gap to human experts. Claude 3.7 Sonnet led language-only scoring and GPT-4.1 led vision-language scoring, yet both trailed human experts by 62% and 58% respectively. On the hardest level, Level 5, every model underperformed humans by at least 84%.

• Vision input helps most models. All evaluated models except Llama 3.2 (90B and 11B) improved their score when given pixel observations in addition to text, suggesting visual cues add information beyond the textual state. Llama 3.2 appears to degrade with dynamic visual input.

• Hallucination separates commercial from open-source models. The three commercial LLMs stayed within roughly 20% of human valid action rates, while Qwen 2.5-VL 72B was the only open-source model with acceptable results. Qwen 2.5-VL 7B and Llama 3.2 11B performed below the random baseline on several levels, meaning their actions were frequently invalid.

• Difficulty amplifies hallucination. Higher-level difficulty correlated with more invalid actions, attributed to longer prompts containing more game elements, which stresses generation stability.

• Misleading tower points consistently fooled models. Even top models built towers at locations that could never reach enemies, despite having the geometric information needed to rule those placements out, indicating weak spatial and numerical validation of plans.

• No multifinality in decision-making. Human experts routinely accomplish two goals with one action (for example, moving the hero to collect gold while attacking enemies). No LLM trajectory showed this behavior.

• Shallow understanding of the action space. Models often skipped upgrades with sufficient gold, sent reinforcements to empty areas, or fired the hero's area-of-effect skill with no enemies present.

• RL baselines also struggle. Ape-X DQN and PPO partially solved easier levels after 100 million steps but remained substantially below human expert performance, confirming TowerMind is challenging for reinforcement learning as well.

Methodology in Plain English

The authors built a tower defense game in Unity and exposed it as an AI environment through Unity ML-Agents, wrapped in the standard OpenAI Gym interface. In each level, enemies spawn in waves and walk along fixed roads toward a base; the player must place and upgrade towers, deploy knight units, and control a hero to stop them. The environment emits observations in three forms: a 512×512 color image of the screen, a JSON text description of the full game state, and a flattened numeric array of the same information. Actions are hybrid vectors: two continuous coordinates for where to act, plus a discrete integer from 0 to 11 for what to do.

Difficulty is quantified with a formula summing normalized road count, tower point count, enemy composition, and resource pressure. Five levels of increasing difficulty were created, and the researchers evaluated seven LLMs zero-shot with identical prompts, once with text only and once with text plus the pixel image. Each model ran five random seeds per level. Two metrics were recorded: the sparse reward score (negative one per enemy reaching the base, bounded between -20 and 0) and the fraction of actions that were legally executable, which serves as a hallucination proxy. Five human experts played the same levels to establish a ceiling. Separately, Ape-X DQN and PPO were trained for 100 million steps each with three seeds to establish reinforcement learning baselines.

Why This Matters

Impact on research. TowerMind lowers the barrier to entry for evaluating LLM agents on sequential decision-making. Its 0.15 GB footprint and CPU-only operation make it practical for fast iteration loops, large-scale parallel evaluation, and use as a reward model during instruction tuning, tasks where running StarCraft II would be prohibitively expensive. The separation of capability (score) from reliability (valid action rate) also offers a template for benchmarks that measure trustworthiness alongside skill.

Real-world applications.

• Agent reliability testing. The valid action rate metric transfers directly to settings where agents issue commands that must conform to system constraints, such as API calls, database operations, or robotic control.
• Rapid prototyping pipelines. Teams iterating on agent architectures or prompt strategies can run many evaluations cheaply on CPU infrastructure instead of provisioning GPUs.
• Training data generation and curriculum design. The level editor allows researchers to generate custom scenarios from trivial to extreme, useful for curriculum learning and for testing contamination resistance.
• Reinforcement learning research. The environment serves as an additional RL benchmark with sparse rewards, partial observability, and a hybrid action space, complementing existing suites.

Industry relevance. Companies deploying LLM agents in operational settings need to know not just whether a model can produce plausible plans, but whether those plans are executable and effective. TowerMind's finding that models generate rule-consistent but strategically wasteful actions maps onto a familiar industrial problem: agents that follow the letter of instructions while missing the intent.

Future Directions

• Validation mechanisms for LLM plans. The paper argues that models need outcome-driven validation, not just surface-level text checks, and calls for systems that can predict the downstream effects of an action before committing to it.

• Vision-informed prompting and preprocessing. Since visual input improved most models, further work could explore visual feature extraction pipelines or prompting strategies that exploit pixel observations more effectively.

• Scaling RL baselines and hybrid approaches. The RL results were far below human performance, leaving open questions about better algorithms for hybrid action spaces, and about whether LLM-guided RL or RL-trained components within LLM agents could close the gap.

• Broader capability coverage. Extending TowerMind to test multi-agent coordination, long-horizon memory, and generalization across procedurally generated levels would broaden what the benchmark can reveal about agent behavior.

Target Audience

This paper is most valuable to researchers building or evaluating LLM-based agents, reinforcement learning practitioners seeking lightweight benchmarks, and engineers who need to assess whether an LLM can act reliably in a constrained environment. It is also useful for benchmark designers interested in measuring hallucination through action executability rather than text-level factuality.

Authors’ abstract

Recent breakthroughs in Large Language Models (LLMs) have positioned them as a promising paradigm for agents, with long-term planning and decision-making emerging as core general-purpose capabilities for adapting to diverse scenarios and tasks. Real-time strategy (RTS) games serve as an ideal testbed for evaluating these two capabilities, as their inherent gameplay requires both macro-level strategic planning and micro-level tactical adaptation and action execution. Existing RTS game-based environments either suffer from relatively high computational demands or lack support for textual observations, which has constrained the use of RTS games for LLM evaluation. Motivated by this, we present TowerMind, a novel environment grounded in the tower defense (TD) subgenre of RTS games. TowerMind preserves the key evaluation strengths of RTS games for assessing LLMs, while featuring low computational demands and a multimodal observation space, including pixel-based, textual, and structured game-state representations. In addition, TowerMind supports the evaluation of model hallucination and provides a high degree of customizability. We design five benchmark levels to evaluate several widely used LLMs under different multimodal input settings. The results reveal a clear performance gap between LLMs and human experts across both capability and hallucination dimensions. The experiments further highlight key limitations in LLM behavior, such as inadequate planning validation, a lack of multifinality in decision-making, and inefficient action use. We also evaluate two classic reinforcement learning algorithms: Ape-X DQN and PPO. By offering a lightweight and multimodal design, TowerMind complements the existing RTS game-based environment landscape and introduces a new benchmark for the AI agent field. The source code is publicly available on GitHub(https://github.com/tb6147877/TowerMind).