The Pulse
SWE-Game Finds Coding Agents Score Below 60 on Game-Building Tasks
SWE-Game evaluates coding agents across five game-development tasks, including building games from briefs, repairing projects and porting them between engines. Its best construction scores stayed below 60, while executable checks outperform
AI.info Team · Written with AI ·
The top-scoring model reached just 50.38 out of 100 when asked to build a game from a short brief. In a paper updated October 7, researchers introduce SWE-Game, a benchmark that evaluates coding agents on creating, finishing, repairing and porting games. Across its three construction tasks, the best overall scores stayed below 60.
The results point to a gap between producing a runnable project and implementing the requested gameplay. The benchmark covers 247 tasks built from 41 executable Godot games across 13 categories, including both 2D and 3D titles. The authors say omissions and gameplay-logic mistakes account for many of the problems they reviewed.
Five tasks, from a short brief to a Unity port
SWE-Game tests five kinds of work: building from a brief, implementing a game design document, completing a partially built project, repairing injected faults and porting a game from Godot to Unity. Each of the four task types other than repair uses one task per reference game; the repair set contains 83 cases. Together, they let researchers compare agents across distinct stages of development rather than test only code generation from scratch.
The reference projects give evaluators a target for what the game should do. Agents receive task-specific combinations of requirements, assets, videos and starter code; evaluator-owned drivers then interact with the submitted game and inspect its engine state. The scoring combines functional checks and replayed inputs with demonstrations of requested features, while visual rubrics separately assess presentation.
“Analysis of reviewed submissions identifies requirement omissions and gameplay logic errors as predominant implementation problems.”
The authors of “SWE-Game: Can Coding Agents Build the Games We Want?”—Xiaoyu Chen, Lai Wei, Jin Wang, Xiangyu Zou, Ruochen Fan, Enze Luo, Mingzhe Yao, Jiahui Zhu, Yuhua Wen, Linghe Kong and Weiran Huang—write.
Opus5 leads, but repair is a different test
The researchers evaluated six models. Opus5 ranked first overall in all five task types, scoring 50.38 on brief-to-game generation and 83.46 on bug repair. The paper reports that the runners-up varied by task: Grok4.6 placed second on brief-to-game, while GPT-5.6 Luna ranked second on the other four.
The gap between those two Opus5 scores matters: repairing a project with an injected fault is not the same challenge as translating a broad brief into a complete game. SWE-Game treats them as separate tasks, and its scores should be read as results under these particular conditions, not as a general measure of how useful coding agents are to game developers.
Runtime checks outscore video judging
The paper also tests whether its evaluation methods align with human judgments. On 1,200 human-labeled behavior assertions drawn from 100 agent-built projects, executable checks reached 92.59% balanced accuracy; a video-based vision-language-model judge reached about 78.41%. The authors report that the executable checks were especially better at detecting defects.
For visual quality, the researchers compared rubric-based model scores with human ratings of 200 gameplay clips. Those scores had a Spearman correlation of 0.829 with the human ratings. The results support using runtime evidence to judge whether a game behaves as specified and visual assessment to judge how it looks, rather than relying on either alone.
A benchmark of game-making, not just code
SWE-Game’s central distinction is between a project that runs and one that preserves the intended mechanics, content and progression. A character might appear to dodge an obstacle in a clip while the underlying collision rules are wrong; an apparent win might also follow a shortcut that bypasses required interactions. The benchmark’s executable references and evaluator-controlled inputs are designed to catch those failures.
The study does not establish how agents perform in every engine, production pipeline or collaborative workflow. It tests a defined set of Godot-derived projects and tasks, and its findings are a snapshot of six models evaluated by the authors. For game teams considering agent-written prototypes, the paper leaves a specific test: does the project merely launch, or can it demonstrate the requested behavior under evaluation?