LTD-Bench: Evaluating Large Language Models by Letting Them Draw

AI summary

Overview

Research area: Large language model evaluation and spatial reasoning (arXiv:2511.02347v1, cs.CL, published 04 Nov 2025), from the Youtu-Agent Team at Tencent Youtu Lab and Xiamen University.

Technical level: Intermediate. The benchmark's outputs are visual and explained without heavy math, but reading the paper assumes familiarity with benchmark evaluation, code generation, and LLM judging protocols.

Scope: The paper introduces LTD-Bench, a 183-item benchmark that evaluates LLMs on spatial perception and spatial imagination by making them produce drawings as dot matrices or executable Python code.

What This Paper Is About

Current LLM evaluation leans on aggregate scores and opaque numerical metrics, which hide whether a model actually understands space and the physical world. The authors argue this creates a disconnect between reported benchmark performance and practical ability, and that a single number like "85% on a benchmark" gives no intuitive picture of a model's spatial strengths or weaknesses. LTD-Bench addresses this by requiring models to draw: every generation output is rendered as an image that a human or an automated judge can look at directly.

Key Contributions

1. A visually interpretable benchmark. LTD-Bench is presented as the first benchmark that turns LLM evaluation from opaque metrics into visual artifacts, letting models draw dot matrices or Python code that renders to images, so spatial reasoning limits are visible even to non-experts.
2. A dual-path, three-level methodology. The benchmark pairs generation tasks (spatial imagination) with recognition tasks (spatial perception) across Easy, Normal, and Hard levels, covering both directions of the language-to-space mapping.
3. Quantified evidence of a capability gap. Experiments across seven advanced models show that even strong reasoners struggle to build reliable bidirectional mappings between linguistic symbols and spatial entities.
4. Visual comparison as a diagnostic for model similarity. Style similarity among images generated by different models is used as a preliminary proxy for measuring how alike models are, something traditional metrics do not capture.

Main Findings

• Overall performance is weak. Of the seven evaluated models, only Deepseek-r1 exceeds a 70% average score (71.54), followed by GPT-4.1-mini (64.38). Qwen2.5-72B-Instruct (30.75) and Llama3.3-70B-Instruct (27.07) reach only around 30%. Human experts solve the Easy and Normal tasks with near-perfect accuracy even in text-only settings.
• Level-by-level degradation. Averages drop from Easy to Normal and then to Hard for most models. For example, Deepseek-r1 scores 82.00 on Easy generation (80.00 by human evaluation, 84.00 by GPT-4.1) but 65.28 on Normal generation and 63.20 on Hard generation.
• Deep reasoning helps recognition, not generation. Deepseek-r1 beats GPT-4.1-mini by over 25% in recognition accuracy but trails it on generation tasks. Llama3.3-70B distilled with Deepseek-r1 data gains 18.05% on recognition yet loses 2.91% on generation (33.71 vs 36.62), with the average rising 7.57.
• Multimodal models show no clear advantage. GPT-4.1-mini and GPT-4o do not consistently outperform text-only models such as Deepseek-r1 and Deepseek-v3 on LTD-Bench, contrary to the intuition that visual experience should help spatial reasoning.
• Recognition is generally harder than generation for weaker models. GPT-4.1-mini scores 77.46 on generation but 47.22 on recognition; Llama3.3-70B-Instruct scores 36.62 on generation and 15.28 on recognition.
• Failures are concrete and visible. Case studies show models rendering "¿" as "¡" and "J" as "L" in dot matrices, drawing "W" upside down in Python (including QwQ-32B on "Draw a blue letter W"), and failing on a clock whose pointer must point to 9:30, an airplane with two wings and two tails, and a jagged leaf with veins.
• Style tracks model lineage. In the Hard-level style comparison across Qwen2.5-72B-Instruct, Qwen2.5-32B-Instruct, and GPT-4.1-mini, 12 of 22 valid samples judged the two Qwen2.5 models most similar (over 50%), while only 1 sample judged Qwen2.5-72B-Instruct and GPT-4.1-mini most similar, 2 judged Qwen2.5-32B-Instruct and GPT-4.1-mini most similar, and 7 found all three different. The total of 22 excludes images that failed to generate.

Methodology in Plain English

The benchmark has 183 items split across three levels. The Easy level (50 generation, 36 recognition) uses finite dot matrices: a model is asked to draw a character as a grid of 1s and 0s, and in the reverse direction it is given a matrix and asked to name the character. The Normal level (36 generation, 36 recognition) moves to continuous, unbounded 2D coordinates: models write Python code that draws a letter or digit using curves only, with direct text-rendering functions such as Text and TextPath explicitly banned, and recognition means reading code and naming the character it would draw. The Hard level (25 generation, no recognition) asks for open-ended real-world objects with specified attributes, such as a cat with pointed ears, long whiskers, and round eyes.

Scoring differs by task. Recognition items have fixed ground-truth answers, so accuracy is computed directly. Generation items have no fixed answer, so Easy and Normal outputs were graded by both human annotators and GPT-4.1, while Hard outputs were graded only by GPT-4.1 on a 0.0–1.0 scale, because open-ended human scoring was judged too subjective; failed code execution scores 0. Human evaluation used 10 annotators per level with mixed technical backgrounds, averaged per level. All models were run at temperature 0, and because GPT-4.1 outputs still varied at temperature 0, each GPT-4.1 evaluation was repeated 5 times. The paper reports that GPT-4.1 tends to give slightly higher scores than humans but preserves the same relative model ranking.

Why This Matters

Research impact. The work argues that numeric benchmark scores conceal spatial reasoning failures, and that rendering model outputs as images creates a shared, inspectable artifact for researchers, developers, and non-experts. It also proposes a new diagnostic use of generated imagery: comparing visual style to probe model similarity, which aggregate metrics do not capture. The authors frame this capability gap as a priority direction for the next generation of AI systems and as a limitation on LLMs acting as genuine world models.

Real-world applications (the paper names these deployment domains as those where spatial reasoning is essential):

• Robotics, where models must translate instructions into physical arrangements.
• Autonomous systems that must reason about position and layout.
• Design tools that generate or interpret visual and spatial specifications.
• Any application that must interact with and reason about the physical world.

Industry relevance. Because LTD-Bench is fully renderable and the evaluation pipeline can be automated with GPT-4.1 as judge, the authors position it as a low-cost, end-to-end screening tool for model selection, while noting human evaluation remains the more precise but resource-intensive reference.

Future Directions

• Grow the benchmark. The authors state that the current dataset is relatively small and covers only spatial perception and imagination; future work plans to expand the dataset and add a wider array of tasks to improve comprehensiveness and generalizability.
• Make model-similarity analysis rigorous. The style comparison is described as preliminary and proxy-based; the authors call for more systematic and quantitative metrics and analytical methods for comparing models.
• Investigate the overthinking effect. Results suggest deep reasoning may hurt generation tasks where inherent spatial ability is insufficient, raising the question of when reasoning helps and when it degrades output.
• Reconcile visual and textual features. The finding that multimodal models do not clearly outperform text-only models on these text-based spatial tasks motivates further study on aligning visual and textual representations in multimodal learning.

Target Audience

Researchers and engineers working on LLM evaluation, spatial and physical-world reasoning, or multimodal model development; practitioners choosing models for robotics, autonomy, or design applications; and benchmark designers interested in evaluation methods whose outputs are directly inspectable rather than purely numerical. The rendered outputs also make the paper's core evidence accessible to non-specialists, though the evaluation design and results tables assume a working knowledge of LLM benchmarking.

Authors’ abstract

Current evaluation paradigms for large language models (LLMs) represent a critical blind spot in AI research--relying on opaque numerical metrics that conceal fundamental limitations in spatial reasoning while providing no intuitive understanding of model capabilities. This deficiency creates a dangerous disconnect between reported performance and practical abilities, particularly for applications requiring physical world understanding. We introduce LTD-Bench, a breakthrough benchmark that transforms LLM evaluation from abstract scores to directly observable visual outputs by requiring models to generate drawings through dot matrices or executable code. This approach makes spatial reasoning limitations immediately apparent even to non-experts, bridging the fundamental gap between statistical performance and intuitive assessment. LTD-Bench implements a comprehensive methodology with complementary generation tasks (testing spatial imagination) and recognition tasks (assessing spatial perception) across three progressively challenging difficulty levels, methodically evaluating both directions of the critical language-spatial mapping. Our extensive experiments with state-of-the-art models expose an alarming capability gap: even LLMs achieving impressive results on traditional benchmarks demonstrate profound deficiencies in establishing bidirectional mappings between language and spatial concept--a fundamental limitation that undermines their potential as genuine world models. Furthermore, LTD-Bench's visual outputs enable powerful diagnostic analysis, offering a potential approach to investigate model similarity.