ASCIIBench: Evaluating Language-Model-Based Understanding of Visually-Oriented Text

AI summary

Overview

• Research area: Multimodal machine learning and benchmark construction — specifically, how language and vision-language models understand and generate ASCII art, a medium that sits at the intersection of text and vision.
• Technical level: Intermediate. Familiarity with multimodal models, CLIP-style embeddings, and classification metrics helps, but the paper is readable without deep technical background.
• Scope: The paper introduces ASCIIBench, a curated benchmark of ASCII art images, and uses it to test how well current models classify and generate ASCII art, and whether CLIP embeddings can serve as a reliable evaluator for that task.

What This Paper Is About

Large language models are good at long-form text and reasoning, but they remain weak at tasks requiring precise spatial and positional understanding. ASCII art — where characters are used as visual building blocks rather than as words — is a hard probe of exactly this weakness. The paper builds a benchmark for both classifying and generating ASCII art, and asks whether existing models and existing evaluation tools (like CLIP) can handle a format that is both text and image at once.

Key Contributions

1. ASCIIBench dataset: A filtered, class-labeled dataset of 5,315 unique ASCII art pieces across 752 classes (e.g., aircraft, birds), sourced from ascii.co.uk. The authors describe it as the first publicly available benchmark of its kind.
2. Released fine-tuned CLIP weights: The authors release weights for a CLIP model adapted to capture ASCII structure, enabling evaluation of LLM-generated ASCII art.
3. Systematic classification evaluation across modalities: Seven models — Llama 3-8B, Llama 3-8B-Instruct, GPT-3.5, GPT-4o, GPT-4o-mini, GPT-5-mini, and Claude 3.5 Sonnet — tested under text-only (T), vision-only (V), and text+vision (T+V) prompts, measured by micro accuracy, macro accuracy, and pass rate.
4. Representation analysis: Evidence that the evaluation bottleneck lies in the embedding model's representational capacity for ASCII structure rather than only in the quality of generated ASCII, supported by alignment/uniformity measurements and a non-monospaced font ablation.

Main Findings

• Vision-only beats text-only and text+vision: Across both raw and filtered datasets, vision-only models consistently outperformed text-only and combined text+vision settings. Under filtering, GPT-5-mini vision-only reached 77.25 micro accuracy and 84.13 macro accuracy, and GPT-4o vision-only reached 77.49 micro and 82.16 macro. The paper's interpretation text states that GPT-4o achieved the highest macro accuracy at 82.2%, which does not match the table's GPT-5-mini vision-only macro value of 84.13.
• Adding text to vision does not help: Combined text+vision prompting did not improve performance over vision alone and in some cases degraded it, which the authors attribute to current multimodal fusion strategies not capturing ASCII structure effectively.
• Text-only performance collapses for weaker models: LLaMA3.1-8B text-only scored 29.39 micro / 25.40 macro accuracy and GPT-3.5-turbo text-only scored 39.98 micro / 33.77 macro accuracy on the filtered dataset, while all accuracy values remained above 25%, indicating models were not choosing arbitrarily.
• Filtering barely changes classification trends: Responses were filtered for possible string parsing errors, removing less than 2% on average, and the filtering had little effect on overall trends, indicating robustness of the observed modality gaps.
• CLIP fails to separate ASCII classes on unfiltered generations: On unfiltered generations, CLIP showed weak class separation with ROC-AUC of approximately 0.55 and a silhouette score of -0.46, and t-SNE revealed no clear clusters.
• Filtering generations dramatically improves CLIP's discrimination: After filtering inconsistent generations (standard deviation greater than 0.15, mean similarity less than 0.3), ROC-AUC rose to 0.83, showing CLIP can discriminate effectively when generations are semantically consistent.
• Low-variance classes still fail: Even the lowest-variance subsets approached near chance performance at AUC = 0.641, so noise in generations alone does not explain the failure.
• High-similarity classes are discriminable: Restricting analysis to classes with high mean similarity raised AUC to 0.83, indicating CLIP can represent ASCII structure only for a subset of well-formed categories.
• No representation collapse after fine-tuning: Out-of-the-box CLIP showed alignment of 5.85 (squared 34.20); fine-tuning increased alignment to 8.90 (squared 79.16) and improved uniformity to -7.61 (t=1), -8.09 (t=5), and -8.21 (t=10).
• Models appear to rely on OCR-like recognition: Replacing the monospaced font with a proportional one barely changed results (GPT-5 vision & text 0.7057, GPT-5 vision only 0.7118), suggesting models depend on character recognition rather than reasoning about spatial alignment.
• The curated dataset has a long-tail class distribution: The largest categories are aircraft (13.3%), land transportation (11.1%), and birds (10.4%).
• Order emerges in character usage: The space character dominates with more than 1.6 million occurrences, followed by structural characters such as -, |, and _, with alphanumeric characters used sparingly as accents.

Methodology in Plain English

The authors first built the dataset. They scraped ASCII art from ascii.co.uk and put it through an 11-step automated cleaning pipeline to strip out artist signatures, tags, dates, email addresses, and invisible Unicode characters, followed by a multi-stage manual review by three annotators using a strict rubric. That process removed over 13,000 low-quality images and 1,800 ambiguous classes, leaving the final 5,315 images across 752 classes.

For classification, they rendered each ASCII image as an image (black DejaVu Sans Mono font on a white background, no blur, following Jia et al. 2024) and prompted each model to pick one of four multiple-choice answers, with a maximum of 50 output tokens. They ran each model in three modes: text only, image only, and text plus image together. Performance was measured with macro and micro accuracy plus a pass rate, and they compared raw versus filtered responses.

For generation, they asked GPT-3.5, GPT-4, and GPT-4o to produce 5 ASCII images per class. To score those generations, they embedded both generated and reference images with CLIP and computed cosine similarity, checking whether same-class pairs scored higher than different-class pairs using ROC-AUC. They also measured the geometry of the embedding space through alignment (how tightly same-class items cluster) and uniformity (how spread out the whole space is), visualized embeddings with t-SNE, and examined silhouette scores. Finally, they fine-tuned CLIP with triplet loss and reran the analysis.

Why This Matters

The paper argues that ASCII art is a stress test for multimodal representations: it is present in language model pretraining data and natively aligned with tokenization schemes, yet its characters carry spatial rather than semantic meaning. The results suggest a dual bottleneck — unstable ASCII generation and a general-purpose embedding model (CLIP) that cannot represent ASCII structure well. Because the dataset and fine-tuned CLIP weights are released publicly, whereas the related ASCIIEval work by Jia et al. (2024) did not release accompanying resources at publication time, this work enables reproducibility and community extensions.

The paper does not enumerate specific real-world applications, but its framing points toward several plausible ones:

• Evaluation tooling for symbolic visual modalities: Structure- and variance-aware metrics could replace or supplement generic image similarity scores for any medium where characters encode form.
• OCR-like and document understanding systems: Since models appear to rely on character recognition rather than positional reasoning, benchmarks like this can expose weaknesses in systems that read structured or layout-sensitive text.
• Safety and jailbreak robustness: The related work cites ArtPrompt-style attacks, where ASCII obfuscation circumvents model safeguards, making ASCII comprehension a security-relevant capability.
• Programmatic and diagram-style generation: The paper situates ASCII art alongside the generation and editing of TikZ drawings, suggesting relevance to models that must produce structured symbolic layouts.

Industry relevance: Any product that evaluates generated structured content — diagrams, terminal-rendered art, layout-constrained text — needs metrics that work on symbolic modalities. The finding that a widely used evaluator like CLIP performs near chance on unfiltered ASCII generations is a caution for teams using embedding similarity as a default quality signal.

Future Directions

• Specialized, smaller models: The authors suggest future work explore specialized, smaller models that may capture ASCII-specific patterns more effectively than CLIP.
• Structure- and variance-aware metrics: The conclusion advocates for metrics that better capture symbolic layout rather than relying on general-purpose embedding similarity.
• Standardized rendering and preprocessing protocols: The paper calls for common protocols to enable fair cross-model comparisons.
• Better prompting and training for ASCII generation: Improving how models are prompted and trained to produce consistent ASCII output is identified as necessary before evaluation quality can be trusted.
• Open question on filtering: The authors note that filtering demonstrates an upper bound of performance but is not a sustainable evaluation strategy, since it amounts to testing on inputs already close to the training distribution.

Target Audience

Researchers working on multimodal representations, benchmark design, and evaluation metrics — particularly those interested in symbolic or layout-sensitive visual modalities. It is also useful for practitioners building or auditing evaluation pipelines who want to understand the failure modes of CLIP-based similarity scoring, and for teams studying model robustness to obfuscated or non-natural-text inputs. The paper is accessible at an intermediate level: readers should understand basic classification metrics and the idea of embedding similarity, but no specialized mathematical background is required.

Authors’ abstract

Large language models (LLMs) have demonstrated several emergent behaviors with scale, including reasoning and fluency in long-form text generation. However, they continue to struggle with tasks requiring precise spatial and positional reasoning. ASCII art, a symbolic medium where characters encode structure and form, provides a unique probe of this limitation. We introduce ASCIIBench, a novel benchmark for evaluating both the generation and classification of ASCII-text images. ASCIIBench consists of a filtered dataset of 5,315 class-labeled ASCII images and is, to our knowledge, the first publicly available benchmark of its kind. Alongside the dataset, we release weights for a fine-tuned CLIP model adapted to capture ASCII structure, enabling the evaluation of LLM-generated ASCII art. Our analysis shows that cosine similarity over CLIP embeddings fails to separate most ASCII categories, yielding chance-level performance even for low-variance classes. In contrast, classes with high internal mean similarity exhibit clear discriminability, revealing that the bottleneck lies in representation rather than generational variance. These findings position ASCII art as a stress test for multimodal representations and motivate the development of new embedding methods or evaluation metrics tailored to symbolic visual modalities. All resources are available at https://github.com/ASCIIBench/ASCIIBench.