Sorry Robot, Happy Human: Vision-Language Models Read Only One of Two Legible Typographic Layers

AI summary

Overview

Research area: Multimodal vision-language models (VLMs), optical character recognition (OCR), and typographic-robustness / typographic-attack research in natural language processing and computer vision.

Technical level: Intermediate. The evaluation protocol is straightforward (transcription accuracy at two resolutions under two prompts), but the framing draws on concepts such as spatial frequency, hybrid images, and contour-versus-shading visual processing.

One-sentence scope: This paper introduces DecoyBench, a 300-image benchmark built with the Decoy Font method, and uses it to test whether six recent closed-source VLMs from three model families can read two superimposed text layers that human readers can distinguish.

What This Paper Is About

VLMs are good at reading text in images but are known to be fragile and manipulable through "typographic attacks." This paper asks a different question: when two complete texts are overlaid in the same visual space — one drawn as thin, sharp contour lines and one formed by soft shading — can a model read both, the way humans can? The authors build a dataset and evaluation protocol, verify human legibility, and then measure how six state-of-the-art closed-source models behave under two prompting strategies and two image resolutions.

Key Contributions

1. The DecoyBench dataset. 300 images generated with the Decoy Font method, each pairing a high-spatial-frequency contour text with a low-spatial-frequency shading text. Text pairs were matched one-to-one on word count and character length to keep the two layers visually integrated. The dataset was released in a structured benchmark format with file path, contour text, and shading text labels, and is distributed at two fixed resolutions (512x512 and 64x64). Summary statistics: 300 total text pairs, mean character length 16.9, mean word count 3.8, mean Jaccard similarity 0.06, and 576 total unique vocabulary items.

2. A human legibility verification study. Ten independent participants were split into 5 subgroups of 2, each evaluating one fifth of the 300 images, with no time limit or viewing-distance restriction, while viewing images at both resolutions.

3. A systematic VLM evaluation grid. Six closed-source models from three families (GPT-5.6 Luna, GPT-5.6 Terra; Gemini 3.6 Flash, Gemini 3.5 Flash-Lite; Claude Sonnet 5, Claude Haiku 4.5) were tested across two prompting conditions (naive and guided) and two resolutions (512x512 and 64x64), scored with Exact Match (EM) and normalized Levenshtein Similarity (LS).

4. A resolution-based diagnostic of the failure mode. By reducing resolution to 64x64 — which eliminates the sharp contour lines — the authors isolate spatial frequency as the likely driver of the models' behavior.

Main Findings

• Humans read both layers at high resolution. At 512x512, participants read 99.7% of contour text (EM) and 96.3% of shading text, and read both layers completely in 96.0% of images.

• At low resolution, the roles reverse for humans too. At 64x64, humans read 0.0% of contour text but 98.7% of shading text, and 0.0% of images had both layers read.

• Models read the contour layer at near-human accuracy at 512x512. Under the naive prompt, the lowest performance was GPT-5.6 Luna at 80.3% EM. Under the guided prompt, all models scored 94% and above, with GPT-5.6 Luna the only model reaching human-level accuracy at 99.7%.

• Models almost never read the shading layer at high resolution. No model reached 1% EM on the shading layer even under guided prompting, while humans achieved 96.3%. The highest shading LS values under guided prompting were 0.320 (Gemini 3.6 Flash), 0.269 (GPT-5.6 Luna), and 0.256 (Gemini 3.5 Flash-Lite).

• The failure is consistent across all three model families. The pattern holds across model variants within the GPT, Gemini, and Claude families, though GPT variants showed a larger gap between each other under the naive condition.

• At 64x64, models read the shading text with high accuracy. Contour EM fell to 0.0 for every model, while shading EM ranged from 93.7% to 100.0%. For example, Gemini 3.5 Flash-Lite and Gemini 3.6 Flash reached 100.0% shading EM at low resolution under the naive prompt.

• Asking for two texts does not mean two texts are found. Guided prompting generally increased the average number of extracted texts (for example, Gemini 3.5 Flash-Lite rose from 1.09 to 1.99 at 512x512, and Gemini 3.6 Flash from 1.02 to 1.87), but Table 3 shows this did not correspond to true detection of the shading layer. The authors interpret this as the models producing a second output because the instruction requires it rather than because they extracted a signal from the image.

• Partial signal exists but is processed incompletely. In some cases the models correctly extracted a few letters of the shading text and then completed the rest with plausible letters; in most cases the second output bore no relation to the real shading text.

Methodology in Plain English

The authors used the Decoy Font method to produce images where each letter position contains two overlapping letterforms: one drawn with thin, sharp contour lines, and one made of soft, diffuse shading. They generated 300 such images using text pairs of identical word count and length, so the two layers fit together seamlessly.

To establish a human baseline, they built an interface that split the 300 images into 5 parts, gave each part to 2 participants, and asked people to write only what they actually read — no guessing — at both resolutions. In total 10 participants contributed, combined as the average of the 5 subgroups.

The models were queried in two ways. A naive prompt simply asked them to transcribe all text in the image. A guided prompt explained that two texts were superimposed, described the thin contour lines versus the soft shading, and walked the model through reading the contour text first, then ignoring the contours entirely and reading the shading as if the image were blurred. Both conditions used the same system prompt, which forced output into a strict JSON schema ({"texts": [...]}) and forbade guessing.

Model outputs were normalized (case, spacing) and compared against the verified contour and shading labels using Exact Match and normalized Levenshtein Similarity, a 0-to-1 score measuring how close an output is to the target text.

Why This Matters

Impact on research. Prior typographic-attack work shows that text added to an image can suppress or distort what a model perceives, and that font size, color, and opacity influence attack success. This paper reframes the issue: even without an adversarial goal, when two legitimate, human-readable text layers coexist, the models collapse onto one of them. The failure is consistent across all three evaluated model families, which suggests a structural property of how these systems treat high- and low-spatial-frequency cues rather than a quirk of one vendor's model. It also connects to the older observation that models tend to classify by texture while humans classify by shape, and to work on hybrid images whose human perception shifts with viewing distance.

Real-world applications (the paper explicitly names watermarks, stamps, and archival documents as examples of superimposed-text documents):

• Automated processing of watermarked or stamped documents, where the mark and the underlying content occupy the same visual space.
• Digitization and OCR pipelines for archival documents that carry overlapping annotations, seals, or bleed-through text.
• Quality control for document-scanning products, where a system might silently return one layer and omit the other without any error signal.
• Accessibility tooling that reads text aloud from images and would need to report every readable layer to match human capability.

Industry relevance. Any product that relies on VLMs for OCR, document understanding, or screenshot parsing inherits this blind spot. A pipeline that returns high-confidence output on the contour layer while missing the shading layer entirely can look correct on aggregate accuracy metrics yet fail on exactly the documents where the second layer carries meaning.

Future Directions

• Test other document styles. The authors suggest investigating how similar methods perform on watermarks, stamps, and archival documents.
• Vary the typography. The limitations section notes that different fonts, contrast ratios, and shading densities were outside the scope of this study.
• Test the layers in isolation and at more resolutions. The two layers were never presented separately, and only two resolutions (512x512 and 64x64) were used, leaving the intermediate behavior unmeasured.
• Extend beyond English. The evaluation was limited to English text pairs; whether the same results hold in other languages or writing systems is unknown.
• Larger human validation. The 10-participant sample limits generalizability to a broader population.

Target Audience

Researchers working on multimodal robustness, typographic attacks, and OCR evaluation; engineers building document-processing or scanning pipelines on top of closed-source VLMs; and accessibility practitioners who depend on models reading text out of images reliably. The paper is also useful for readers interested in the gap between human and machine visual perception, since the human-versus-model comparison at two resolutions is its most distinctive element.

Authors’ abstract

Vision-language models (VLMs), despite their success in optical character recognition (OCR) tasks, are vulnerable to typographic attacks and have a fragile structure for images with multiple text layers. In this study, the DecoyBench dataset was created using the Decoy Font method. The dataset consists of 300 images, each containing text with sharp contour lines superimposed on another text with soft shading. Six recent closed-source models from three different model families were evaluated using this dataset under two different prompting conditions (naive and guided) and at two different resolutions ($512\times512$ and $64\times64$). A validation study showed that human participants could read both text layers with high accuracy. In contrast, the models, with most variants and both prompting methods, read the contour text with near-human accuracy at high resolution, but almost never fully extracted the shading text. At low resolution, the contour text could not be read by either the models or humans, while the shading text could be extracted with high accuracy. The findings indicate that the evaluated VLMs exhibit a consistent behavioral limitation when processing typographic structures containing multiple spatial frequency layers.