The Pulse
StarSkirmish Bench Measures How Well LLMs Write StarCraft Bots
StarSkirmish gives each language model one hour to write a Protoss bot in C++ and evaluates the bots in a tournament against other model-written bots, established human-written bots and demo bots. Its page reports GPT-6 Astra and Claude Opu
AI.info Team · Written with AI ·
What StarSkirmish measures
StarSkirmish is a benchmark in which language models write Protoss-playing bots in C++ for StarCraft: Brood War. Each model gets one hour of wall-clock time to produce a bot, and the page says the benchmark is intended to test whether models can learn from practice games and improve their code. The benchmark’s page, by Kai McPheeters and dated September 26, 2026, describes its aim this way:
“StarSkirmish isn't a novelty game bench, it captures meaningful long term reasoning and agentic coding abilities.”
Kai McPheeters, page author, writes
Each model runs at a “high” reasoning level. The coding environment provides tools to compile a bot, play batches of practice games against selected tiers of opponents, and read game transcripts with build timings, fight summaries and economy recaps. The practice opponents are grouped into tiers: D contains the three demo bots; C and B contain three mid-ranked human-written bots each; A contains BananaBrain and Locutus; and S contains Stardust, identified as the strongest human-written bot.
Games and tournament field
The bots use BWAPI 4.4.0 and play through OpenBW. All games are Protoss versus Protoss on three ladder maps: Heartbreak Ridge, Benzene and Destination. A game ends when one side’s buildings are destroyed. If it reaches the 60-minute cap, the page says the winner is decided as on the BASIL ladder: the side with the higher kills-and-razings score wins.
The tournament includes 62 entrants: 50 language-model bots, comprising 10 models with five runs apiece; nine competitive human-written bots; and three demo bots. Every entrant plays every other entrant six times, twice on each map. Games ran in Prime Intellect sandboxes. Rather than report only the best run for a model, StarSkirmish averages the ratings of its five bots.
All models use the same harness, identified on the page as Inspect’s ReAct deepagent. It includes bash, a text editor, a memories tool and research subagents. Models are called through OpenRouter with high reasoning effort, with settings requiring first-party inference. The harness is instructed to continue until the hour is up; there is no submit tool, and the bot code is automatically collected when time expires.
Scoring and reported results
StarSkirmish fits Elo ratings from all tournament games at once. It calculates a bot’s win probability against each reference bot using the Elo convention, then averages those probabilities across 12 reference bots: the nine competitive human-written bots and the three demo bots. The resulting score is scaled so Stardust, the strongest bot, receives 100 and Four Gate Dragoon, the weakest demo bot, receives zero.
The page says GPT-6 Astra and Claude Opus 5.5 are functionally tied as the two highest-scoring language models. GPT-6 Sol is also described as a clear step above every other tested model, alongside those two leaders. Overall model standings are based on the average rating of each model’s five bots; the page also presents results for the human-written and demo-bot groups.
The benchmark page says its scores correlate strongly with four public coding benchmarks. It identifies Vals IOI and WeirdML among the comparisons, describing WeirdML as an independent benchmark of unusual machine-learning coding tasks. The page notes that WeirdML v3 has scores for six of the ten models, and lists GLM 5.3, Grok 4.7, Muse Spark 1.3 and MiMo V2.6 Pro as absent from that comparison.
For future work, the page says the creators plan to launch versions with substantially longer reasoning periods, noting that leading reasoning models benefit from periods longer than one hour. It also says StarSkirmish has partnered with Good Start Labs on an RL environment.