Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Models

AI summary

Overview

Research area: Artificial intelligence / large language model evaluation, specifically the 2D spatial-reasoning abilities of models trained only on text and code.

Technical level: Intermediate overall; the representation-probing and causal-intervention parts of the paper (Experiment 3) are Advanced.

Scope: The paper introduces a benchmark (AM-Bench) that separates a text-only model's ability to express a stated spatial layout in code from its ability to compose a layout on its own, evaluates eight open-weight models under two output media, and probes whether a spatial plan exists in the model before it starts generating.

What This Paper Is About

Text-only language models can sometimes write code that draws recognizable images, but a finished image alone cannot tell you whether the model had an internal picture of the 2D layout or merely translated a spatial description into code. This paper builds a benchmark that pulls those two abilities apart, scores them separately, and then looks inside the model's activations to see whether a layout plan is present before any drawing code is written. The goal is to determine whether 2D spatial performance in text-only LLMs is a genuine reasoning capability, a code-writing skill, or a property of the output format.

Key Contributions

1. AM-Bench, a four-stage benchmark pipeline (prompt, internal plan, program code, rasterized geometry) built around autoregressive mosaics: short LLM-written programs rendered to a 24 × 24 raster through a custom Python API with six primitives (fill, set_pixel, rect, circle, line, poly), a 30-name recommended color palette, Bresenham lines, midpoint-circle fill, and polygon fill.

2. A translation/layout task split. The translation task gives the model the complete geometry of an image in words (every shape with position, size, and color, stated in row/column terms) and scores it symbolically with a per-part IoU metric (PIoU) in normalized [0, 1]² continuous coordinates, so the score is resolution-independent. The layout task gives only an underspecified prompt and scores the model with two vision-language model judges on five dimensions (0–5, pass ≥ 3.0).

3. An output-medium ablation comparing the custom canvas-code API against raw SVG generation, with all SVGs re-rasterized to 24 × 24 from their vector sources before scoring so only expressiveness, not rendering fidelity, is measured.

4. A pre-generation probing study plus a causal intervention using ridge regression on residual-stream activations to decode a coarse 6 × 6 occupancy grid, and a textual coordinate-displacement test on GLM-4 32B and Gemma-2 27B to check whether models actually use a pre-formed plan during generation.

Main Findings

• Translation is not the bottleneck. Translation produced 13,920 attempts on the 145-reference set (145 references × 8 models × 12 samples). The median PIoU is 1.000 for all eight models, and 98.1% of all 13,920 attempts pass the threshold τ_trans = 0.6, far above the 0.072 random baseline. Mean PIoU ranges from 0.929 (Llama 3.1 8B) to 0.995 (Qwen2.5-Coder 14B).

• Layout performance varies widely while code generation does not. Under Judge 1 (Qwen2.5-VL-7B), GLM-4 32B leads at every tier with an overall score of 3.57, down to CodeLlama 34B last at every tier with 1.75. Pooled overall scores are 2.79 (Judge 1) and 3.69 (Judge 2).

• The judges are lenient differently but rank identically. Judge 2 (InternVL3-8B) scores about a point (0.90 overall) higher than Judge 1, yet the two agree exactly on model-level rank ordering (Spearman ρ = 1.0), despite only moderate per-sample agreement (Pearson r = 0.43–0.54, Spearman ρ = 0.42–0.57 across the eight models). Two human annotators judging 350 pairs reached Krippendorff's α = 0.60 and agreed on 70.6% of pairs; on pairs where they agreed and the judge separated the pair, Judge 1 matched the human verdict on 82.4% (61/74) and Judge 2 on 76.7% (46/60).

• The expected difficulty gradient inverts. T3 Compositional outscores T2 Iconic across all models (Judge 1: 2.69 vs. 2.45; Judge 2: 3.52 vs. 3.43). The inversion (Δ = T2 − T3 < 0) holds for five models under Judge 1 and six under Judge 2. Gemma 2 27B inverts most strongly (Judge 1: −0.90, Judge 2: −0.48); Gemma 2 9B and CodeLlama 34B keep the expected order; Llama 3.1 8B splits (Judge 1: +0.12, Judge 2: −0.09). The authors attribute this to the 24 × 24 resolution being too coarse for recognizable iconic objects.

• Models diverge most on compositional prompts. Cross-model spread peaks at T3 under both judges (Judge 1 range 2.17 at T3 vs. 1.66 at T2 and 1.62 at T1; Judge 2: 1.78 vs. 1.58 and 1.01). Subcategory-wise, 1E Color Partitions is easiest (Judge 1: 3.93), while 2C Everyday Objects and 3D Scene Compositions are consistently hardest (Judge 1: 1.97 and 2.18; Judge 2: 2.85 and 2.63).

• Most outputs run but some degenerate. Of 13,200 layout attempts, 12,932 (98.0%) are valid and 268 (2.0%) fail to execute or render. Within the valid set, 11,976 (92.6%) are non-trivial and 956 (7.4%) are trivial (over 98% of the 24 × 24 pixels share one color). Worst-case validity is 88.2% (Gemma 2 9B, T3); CodeLlama 34B produces trivial outputs on 12–27% of attempts, and Gemma 2 9B spikes to 20.5% at T3. Both judges scored every valid attempt (12,932 for Judge 1; 12,926 for Judge 2, with six images Judge 2 could not process across five models).

• The output medium matters. Pooled across all eight models, switching from canvas code to raw SVG raises layout scores by +0.37 (95% CI [0.26, 0.47]); six of eight models improve significantly on their own. CodeLlama 34B shows the largest improvement, which the authors read as evidence of SVG pretraining exposure rather than a general code-model effect. The relative model ranking is preserved, but the paper states that absolute layout scores are a property of the canvas-code interface, not of spatial reasoning in general.

• A coarse layout plan is decodable before generation, but only the generic part. Against each model's own consensus occupancy target, decodability beats the text baseline for all eight models after Benjamini-Hochberg correction (mean ΔR² = +0.164). A model's own continuous consensus predicts a single reference generation at 0.688 on average versus a 0.625 text baseline, and in four of eight models that difference is under 0.05 (which is why the consensus target is used).

• The encoded plan is shared across models, not model-specific. For the two strongest large models, the leave-one-model-out shared component decodes far above text (ΔR² = +0.26 for GLM-4 32B and +0.28 for Gemma 2 27B), while the specific residual component does not (R² ≤ 0.005). Split-half reliability ceilings are 0.883 and 0.941, so the target is measurable but not saturated. Prompt length does not predict the target (R² = −0.03), and a TF-IDF baseline (0.135) agrees with a mean token embedding baseline (0.132).

• Generation tracks the evolving geometric state rather than executing a fixed plan. Displacing a completed shape into an unused region of the code prefix raises the rate at which the model keeps generating: GLM-4 32B from 43.8% to 84.6% (Δ = +0.42) and Gemma-2 27B from 2.5% to 41.2% (Δ = +0.39), both at BH-FDR p = 0.0007. A smaller displacement into a region the model intended to fill later (median 4 px vs. 9 px) gives a weaker response (+0.08 for GLM-4 32B, +0.16 for Gemma-2 27B). The test covers 731 eligible generations, 2,924 (item, arm) pairs, and 26,316 continuations.

• Color accuracy is the strongest rubric dimension. It is the highest-scoring of the five dimensions for every model under both judges; the other four dimensions track each other closely, with no single dimension consistently weakest.

• The benchmark is not saturated at 10 samples. The three strongest models reach 0.92–0.93 overall and 0.96 at T3, while Gemma 2 9B and CodeLlama 34B plateau at 0.66 and 0.69.

Methodology in Plain English

The authors avoid showing the model any image. Instead they have it write a short Python program that paints on a small 24 × 24 grid using only six drawing commands, and a deterministic executor renders the result. To avoid confusing "can it code?" with "can it arrange shapes?", they run two tasks. In the translation task the prompt spells out the entire layout in words — every shape, its row, its column, its size, and its color — with subject names deliberately withheld so the model cannot recall a familiar drawing of, say, a house. Because the correct geometry is known, the output is scored symbolically with exact polygon geometry and a per-part intersection-over-union average (PIoU), plus an over-paint penalty so that a model cannot win by drawing extra shapes on top. In the layout task the prompt names an object but underspecifies its appearance, so the model must decide the arrangement itself; because no reference geometry exists, the resulting mosaic is scored by two independent vision-language models across five rubric dimensions, with human annotators used to validate the judges.

The second experiment repeats the layout suite twice per model under an otherwise identical protocol, once with canvas code and once with raw SVG, re-rasterizing all SVGs down to 24 × 24 so that resolution and anti-aliasing do not contaminate the comparison. The third experiment reads the model's internal activations from a single forward pass over the layout prompts and trains a ridge regression to predict a coarse 6 × 6 occupancy grid over the canvas, using as the target each model's own average occupancy across its repeated attempts, and reporting the improvement over a TF-IDF text baseline. A follow-up intervention truncates a real generation after its first drawing statement and resumes it under four conditions — nothing appended, the model's own next statement, that statement moved to an unused area, or moved to an area the model planned to fill later — to see whether the model treats a displaced shape as an error to correct.

Why This Matters

Impact on research. The paper gives the field a way to stop conflating spatial composition with code-generation skill, showing that models which are near-perfect at translating stated geometry still differ substantially when they must invent a layout. It also produces a cautionary result for anyone evaluating spatial reasoning through SVG: prior exposure to the SVG format inflates scores, which is why the authors use an uncontaminated custom API. Finally, it adds a representation-level finding to the emergent-structure literature: what is decodable before generation is only the generic layout the prompt implies, while the particular layout is decided as generation proceeds.

Real-world applications (from the paper's framing and results):

• Generating small pixel-art or icon assets from text descriptions.
• Producing SVG/vector graphics from natural-language prompts in design and illustration tools.
• Screening models for layout-sensitive tasks such as diagrams, posters, or game tiles, where the paper shows code correctness is a poor proxy for spatial correctness.
• Extending the same pipeline to 3D through structured-language media, as suggested in the conclusion.

Industry relevance. The result that the output medium changes measured performance implies that model choice and interface choice are separate decisions: a model that looks weak under one programmatic interface may score higher under another. The translation gate also offers a practical quality-control step — only models whose median translation score clears τ_trans = 0.6 are admitted to cross-model comparison.

Future Directions

• Extending the evaluation beyond the eight open-weight models tested (8B–34B); the authors state that generalization to larger or closed-weight models remains untested.
• Intervening directly on internal activations (for example, activation patching) rather than on the textual geometric state, which would test whether the decoded representation actually drives generation.
• Repeating the shared-versus-specific decomposition on all eight models; it currently covers only the two strongest.
• Reducing judge variance, and expanding the pairwise human validation beyond the current 350-pair, two-annotator setup.
• Extending the mosaic framework to 3D via structured-language media.

Target Audience

Researchers working on LLM spatial reasoning, emergent representations, and programmatic visual generation; benchmark designers who need to separate capability from expression medium;

Authors’ abstract

Large language models (LLMs) trained only on text and code can sometimes generate programs that draw recognizable images. However, it is unclear whether this reflects an internal representation of 2D spatial layout or simply the ability to translate spatial descriptions into code. We introduce Autoregressive Mosaics (AM-Bench), a benchmark that separates these factors: First, a translation task gives a model a fully specified geometry of a picture in words as a prompt and asks for the code that produces it. Second, a layout task requires the model to compose an image from an underspecified prompt. Across eight open-weight text-and-code-only models, all models reliably translate specified geometry into code, but their open-ended layout performance differs substantially, indicating that these differences are not explained by code-generation ability alone. An output-medium ablation further shows that the interface or medium of expression that the model uses matters: replacing procedural code with raw SVG improves layout scores across all models. Finally, probing model activations shows that a coarse layout plan is present before generation, but reflects only the layout implied by the prompt. During generation, models track the evolving geometric state instead of executing an initially fixed plan. Overall, these results show that 2D spatial performance in text-only LLMs depends on both the model and the output medium, and is not explained by code-generation ability alone.