The Pulse
New VVRBench Finds Image Models Fail Complex Prompts
The strongest image generator tested on VVRBench-Challenge, GPT-Image-2.5-Sunburst, solved 21.39% of its tasks and 7.92% of the hardest. Training on synthetic scenes also improved results on natural prompts, University of Washington researc
AI.info Team · Written with AI ·
Even the top score is below 22%
The strongest image generator tested on VVRBench-Challenge, OpenAI’s GPT-Image-2.5-Sunburst, solved 21.39% of its tasks, and its accuracy fell to 7.92% on the most complex ones. Apart from OpenAI’s GPT-Image-2, every other model tested on the challenge set solved at most 8% of it. The results come from a paper that four University of Washington researchers posted on arXiv on Monday.
The paper, “Verifiable Visual Rewards Transfer from Synthetic Scenes to Natural Prompts,” submitted on 28 September 2026, is by Shuyue Stella Li, Xiaochuang Han, Yulia Tsvetkov and Luke Zettlemoyer. It introduces VVRBench, a 10,000-task benchmark, and VVRBench-Challenge, a separate set of 720 harder tasks. GPT-Image-2 scored 10.28% on the challenge, half the leader’s score. On the main benchmark the strongest open-weight model, FLUX.2-dev, reached 19.2%.
The authors built the challenge set because VVRBench-Fast, an 820-task subset of the main benchmark with simpler tasks, barely separated the two strongest models: GPT-Image-2.5-Sunburst solved 84.51% of those tasks and GPT-Image-2 82.20%. On the challenge set the leader’s accuracy dropped from 31.67% in the lowest complexity band to 24.58% in the middle band and 7.92% in the highest.
Verifiers check the picture, not a model’s opinion
Each VVR task is a scene of colored geometric shapes with requirements that can be checked against the generated image, such as how many objects appear, what color and shape they are and where they sit relative to one another. The researchers wrote deterministic verifiers, Python functions that read the image’s pixels, instead of relying on an object detector or a vision-language model to judge whether a prompt was followed. Their task generator only composes instructions whose constraints can all be satisfied together.
That design gives a clear pass-or-fail result for a whole image and shows which individual requirements a model missed. On the challenge set, averaged across the API models tested, 95% of “grounding” constraints were satisfied against 56% of “topology” constraints. The hardest single types involved counts or relations across groups of objects: “same count” passed in 31% of checks, “times as many” in 33%, and “each contains”, which requires each object of one group to contain a different object of another, in 34%. The authors, whose first author is Shuyue Stella Li, stated the pattern as one of the paper’s findings. They wrote:
“Generators satisfy requirements on individual objects and pairs but fail requirements that constrain whole sets of objects.”
Shuyue Stella Li, Xiaochuang Han, Yulia Tsvetkov and Luke Zettlemoyer, University of Washington, in the VVRBench paper
Synthetic training improves results beyond shapes
The authors also used VVR scores as rewards to train Stable Diffusion 3.5 Medium with reinforcement learning. Training on an easier set of VVR tasks raised accuracy on the full benchmark from 2.81% to 28.27%; training on tasks matched to the benchmark’s difficulty distribution raised it to 46.60%. The easier training set also improved results on tasks more complex than any the model saw in training, by 17.16, 8.31 and 1.35 points across the three hardest bands, so the gains shrank as complexity rose.
To test whether practice with colored shapes carried over to ordinary prompts, the researchers had annotators compare images for 160 natural prompts outside VVR. The model trained on the easier VVR tasks beat the pretrained model with a win rate of 71.6%, counting ties as half a win. Across all 400 image pairs in the study, which used three annotators, two annotators who both picked an image agreed 83.8% of the time on average. The same model improved on eight of ten non-VVR metrics, including GenEval by 0.113 and OCR by 0.111.
A precise test, deliberately narrow
VVRBench covers requirements with objective answers: counts, colors, shapes, positions and containment. The central benchmark uses simplified geometric scenes rather than the full range of objects, styles and ambiguous requests people bring to image generators, and the natural-prompt human study covers 160 prompts. The transfer results are evidence that the skills carry over, not proof that every kind of instruction-following failure will improve.
The benchmark, the verifier and the training code are public on GitHub, and the task sets are on Hugging Face. The authors write that the benchmark can be updated with higher complexity as frontier models evolve.