The Pulse
RL Trained LLMs to Play Tic-Tac-Toe, but Not to Think Strategically
A study of reinforcement learning in small language models found that accuracy fell when tic-tac-toe boards were presented as ASCII instead of natural language. Its analysis suggests the models relied more on prompt-visible cues than on str
AI.info Team · Written with AI · fact-checked ·
On tic-tac-toe boards written as ASCII grids, the strongest model in a new study saw its accuracy fall from 73% with natural-language prompts to 44%. The result points to a gap between getting better at a benchmark and learning a strategy that carries across ways of presenting the same problem. In a paper submitted to arXiv on October 29, Aishik Nagar of Carnegie Mellon University and Cheston Tan of A*STAR examine what reinforcement learning changes inside small language models trained to play the game.
Higher scores, weaker evidence of strategy
The researchers trained Qwen2.5 models with 0.5 billion and 1.5 billion parameters, alongside Llama 3.2 1B Instruct, using Group Relative Policy Optimization (GRPO). They tested legal-move prediction and best-move selection across tic-tac-toe states, including splits designed to keep rotations and reflections of a board together. The game’s 5,478 reachable legal states let the team compare model behavior with a complete, exhaustively generated set of positions.
GRPO raised performance across the three models. The strongest model reached 74% best-move accuracy on the random split and 71% on the symmetry-controlled split, while legal-move prediction reached 100% in one setting. The gains were real, but they did not by themselves establish that the models had learned reusable strategic concepts.
“Quantitatively, RL yields high accuracy and robustness across prompt variations, suggesting emergence of game reasoning.”
Aishik Nagar and Cheston Tan, the paper’s authors
The models tracked the prompt more than the board’s logic
To probe internal representations, the authors used sparse autoencoders and tested the models’ activations for recognizable properties of each position. They found that GRPO made information already stated in a prompt—such as whose turn it was and how far the game had progressed—more prominent. Strategic features, including whether a move would block an opponent or create a fork, remained weakly represented.
For Qwen2.5-1.5B, a statistic measuring how well board states separated by game progression rose from 274 to 2,330 on a dominant axis in a middle layer. A corresponding statistic for turn identity fell from 5,285 to 316 there, then reappeared in later layers. The researchers also tested raw model activations without the autoencoder-based visualization pipeline: turn identity and game progression were already perfectly decodable before training, while probes for strategic situation and symmetry showed little change after GRPO.
Format changes exposed a fragile skill
The drop from natural-language descriptions to ASCII boards suggests the models did not build an input-format-independent representation of the game. Replacing the player symbols X and O with other characters caused a smaller but consistent decline. The authors also found that models chose corresponding moves for rotated or reflected boards only about 56% to 58% of the time, rather than reliably applying the same strategy in a different orientation.
The findings do not reduce the result to memorization. On held-out states, the best-move accuracy of Qwen2.5-1.5B was close to its training-state accuracy in both data splits. Instead, the authors argue that the models learned to use visible cues more effectively, without developing the higher-level abstractions their tests examined.
A narrow test, not a verdict on frontier models
The paper’s scope is deliberately limited: three instruction-tuned models between 0.5 billion and 1.5 billion parameters, one small game, and GRPO rather than other reinforcement-learning methods. The authors do not claim their results describe frontier-scale systems or open-ended reasoning. Their analysis also samples 11 of 28 model layers and checkpoints every 300 training steps, leaving room for internal changes their measurements might miss.
That boundary matters because tic-tac-toe is unusually easy to enumerate, but far removed from open-ended tasks. The authors propose extending the work to Connect Four, simple algebra and grid-world planning. Whether the same gap between benchmark gains and strategic representations appears in those domains—or in larger models—remains an open question.