The Pulse
Small Models Drop Below 4% on ARC-AGI Transfer
Three small language models reached seen-family scores as high as 54.4% but peaked at 2% to 3.5% on ARC-AGI, according to a study of abstract grid transformations. The paper also reports steep drops when models trained on small grids were t
AI.info Team · Written with AI ·
Three small language models scored as high as 54.4% on task families they had trained on, but reached peak scores of just 2% to 3.5% on the separate ARC-AGI evaluation set. The results come from a study posted to arXiv on October 6 that tests whether models learn abstract transformation rules or perform well mainly on familiar task distributions.
The gap is striking, but the comparison is not a pure test of novelty: ARC-AGI differs from the training benchmark in both task novelty and difficulty. The researchers describe those results as cross-benchmark transfer, and say exact-match scores alone cannot establish what computation a model performs internally.
Grid size exposed a second transfer gap
The paper’s more controlled test kept the transformation families while changing grid scale. When trained on small grids, the decoder-only Qwen3-1.7B scored 68.5% on small-grid evaluation and 24.3% on large grids. The mixture-of-experts Qwen1.5-MoE-A2.7B-Chat fell from 53.1% to 12.7%; the encoder–decoder T5Gemma model dropped from 64.4% to 5.9%.
Those comparisons preserve the underlying rule while changing the spatial scale of the puzzles. The declines therefore point to difficulty transferring a learned transformation across different grid sizes, although the study cannot isolate why each model struggles: its three representatives also differ in pretraining, tokenization, context length and capacity.
More training did not help every task family
The researchers ran more than 1,000 training and evaluation experiments using ARC-TGI, a benchmark that generates variations on abstract grid transformations. Their models represented three designs: decoder-only, encoder–decoder and mixture-of-experts. All underwent supervised fine-tuning, and the team tested training choices, grid sizes, prompt formats and the number of examples shown in context.
Adding training families raised scores on a shared evaluation set, but gains were uneven. As the number of training generators rose from 60 to 193, the decoder-only model’s accuracy increased from 39.3% to 51.6%, while the other two rose from 39.1% to 45.0% and 7.5% to 13.0%. Some task families improved while others remained difficult.
Prompt format also changed the outcome. In one comparison, asking models to write executable Python rules rather than directly produce a grid raised the mixture-of-experts model’s score from 33.7% to 59.0%. Four examples in the prompt helped the two decoder-based models, but the encoder–decoder model’s accuracy fell from 43.6% with one example to 23.4% with four.
The authors limit what the scores can show
The experiments use one pretrained model for each model family, so the results cannot show that an architecture alone caused the differences. The team also did not estimate variation across training seeds, and its ARC-AGI peak scores are post-hoc summaries across checkpoints. Those limits make the findings a warning against treating a strong score on familiar tasks as proof of transfer, not a verdict on every small model or every form of reasoning.
“This evidence is consistent with distribution-specific fitting, but it does not establish the absence of abstract reasoning because exact-match behavior alone cannot determine the internal computation.”
Nur A Zarin Nishat, Jens Lehmann, Andrei Aioanei and Sahar Vahdati, the paper’s authors
The paper argues for measuring task-family coverage, training trajectories and performance under controlled shifts alongside endpoint accuracy. Its cross-benchmark result remains difficult to interpret as a standalone measure of generalization; the grid-scale comparison provides a more targeted test, with the same transformation rules yielding much lower scores when the grids grow.