VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning

AI summary

Overview

Research area: Computer Vision — specifically native visual reasoning, where a model solves a problem by generating and updating visual states (images and videos) rather than by producing text. Technical level: Advanced. The paper assumes familiarity with diffusion/autoregressive image and video generators, mixture-of-experts and mixture-of-transformers architectures, LoRA fine-tuning, and reinforcement learning from reward signals. Scope in one sentence: The paper introduces VBVR-Pro, a closed-loop suite of 300 procedurally generated visual reasoning tasks with deterministic, verifiable reward scorers, and uses it to train and compare more than 30 image, video, and interleaved text-image generators under one shared task distribution.

What This Paper Is About

Most machine reasoning is studied through language, but many physical-world problems involve spatial transformations, temporal continuity, object persistence, and dynamic interaction. Native visual reasoning proposes that models should solve such problems by constructing and updating visual states, treating images and videos as the substrate of reasoning rather than as inputs to a language reasoner or as final rendered outputs.

Progress in this paradigm is blocked by three gaps: there is little scalable, trainable task data; open-ended outputs are hard to score reliably; and image, video, and interleaved generators have rarely been compared under a shared task distribution and verification protocol. VBVR-Pro is the authors' attempt to close all three gaps in a single testbed.

Key Contributions

1. A scalable, multi-modal task suite (VBVR-Pro-Dataset). 300 procedurally generated tasks built by more than 50 researchers and engineers worldwide. 150 generators are reimplemented and revised from VBVR, and 150 are newly designed. Each instance is rendered into aligned modalities — video, image, and interleaved textual annotations — from the same underlying problem, so different generator types can be compared fairly. The dataset contains 3,471,558 images and 1,300,500 videos, with 1,250,000 SFT instances, 50,000 RL instances, and 500 test instances.

2. Verifiable reward scorers. A dedicated scorer for each of the 100 tasks in VBVR-Pro-Bench (50 in-domain, 50 out-of-domain). Rather than comparing pixels, the scorers extract task-relevant semantic attributes (color, shape, position, count) using classical computer vision — HSV color segmentation, contour detection, OCR for rendered digits and labels, and trajectory tracking across frames — then apply task-specific checks with calibrated weights. Soft criteria are combined as a weighted sum; hard constraints are combined multiplicatively so a single violation sharply reduces the score.

3. A controlled modality study. Identical task suites and scorers are used to train and evaluate more than 30 proprietary and open-source image, video, and interleaved generators, varying scale, output format, and modality configuration.

4. Diagnostic evidence for vision-native trajectories. Counterfactual, ablation, and middle-state intervention experiments argue that intermediate visual states are a more critical substrate for visual reasoning than explicit linguistic chains of thought. All data, models, scorers, and code are released.

Main Findings

• Transfer beyond the suite: Models trained on VBVR-Pro-Dataset gain consistently across seven external visual reasoning benchmarks — RISE-Video, V-ReasonBench, RULER-Bench, MME-CoF-Pro, VideoThinkBench, BabyVision, and IntelligentVBench — often by more than 20 percentage points.

• Gains are not instruction-pattern fitting: Nearest-neighbor analyses in visual and textual embedding spaces show held-out benchmark cases are not explained by close matches to training samples. Same-prompt visual counterfactuals show models respond correctly when the visual state changes but the instruction stays fixed. Models also display visual reasoning behaviors such as multi-path exploration.

• Task coverage matters more than instance count: The authors hypothesize that the bottleneck in prior synthetic data is task coverage rather than instance count. Among the 300 tasks, the median new task contains 80 distinct connected color regions versus 12 for the reworked tasks (averaged over 50 sampled question frames per task). In a blinded pairwise comparison, the new task was judged to require deeper reasoning in 113 of 150 pairs. Only 7% of the reworked VBVR tasks require multi-step reasoning, compared with 47% of the new tasks.

• VLM-as-a-judge fails in three ways: numerical imprecision, neglect of fine-grained evidence, and misunderstanding of task rules. In an additive color-mixing task, all three VLM judges gave at least 0.80 to an output with an incorrect hue, versus 0.40 from the scorer. In a concentric-ring task, judges gave 0.78 or 0.93 despite decisive errors. In a pattern-completion task, a correct answer received 0.94 from the scorer but only 0.24–0.80 from VLM judges.

• VLM judges are not reproducible: Re-running evaluators on the same videos, the verifiable scorer changed scores for 0.0% of cases, versus 54.6% (GLM-4.6V-Flash), 69.5% (Gemma4-31B), 74.5% (Qwen3.6-27B), 80.8% (Qwen3.7-plus), 82.9% (InternVL3.5-38B), and 92.8% (Gemini-3.1-Pro). Maximum score change was 0.000 for the scorer and up to 0.221 for Gemini-3.1-Pro. The abstract summarizes this as 55–93% of samples changing score.

• Better human alignment at lower cost: In an arena-style study using 100 tasks × 5 instances, eight video generation models, and 4,000 videos, with ten independent human annotations per model pair, the scorer's per-vote agreement with human preferences exceeds 0.60, beating GPT-5.5 (0.54) and Gemini-3.1-Pro (0.52) and reaching about 78% of the human agreement ceiling of 0.77. At the model-ranking level, the VBVR-Pro scorer reaches Spearman ρ = 0.95 in-domain and ρ = 1.00 out-of-domain, with Pearson r ≥ 0.95 on both splits; out-of-domain ρ improved from 0.93 and r from 0.92 relative to the earlier VBVR scorer.

• Rewards work for reinforcement learning: The scorers provide unambiguous, task-grounded reward signals that yield steady improvements even on strong baselines, supporting large-scale multi-task RL for visual reasoning.

• Training gains split by generalization type: Task-specific training on VBVR-Pro-Dataset improved all nine open-source foundation models, with an average overall gain of 0.290. In-domain improvement averaged +0.401 and out-of-domain improvement averaged +0.179.

• Video remains strongest where state tracking matters: The best interleaved model (VBVR-Pro-SenseNova-U1) performs on par with the best video model (VBVR-Pro-Wan2.2-I2V-A14B) on many in-domain tasks while costing substantially less to generate, but video retains an advantage on in-domain transformation tasks and across out-of-domain categories. Single-image generation remains substantially weaker than both trajectory-based representations, particularly on transformation tasks.

• Strongest benchmarked models: On VBVR-Pro-Bench, VBVR-Pro-Wan2.2-I2V-A14B scores 0.670 overall (0.808 in-domain, 0.532 out-of-domain) and VBVR-Pro-SenseNova-U1 scores 0.638 (0.811 in-domain, 0.464 out-of-domain). Among proprietary models, Nano Banana Pro reaches 0.564 overall, Seedream-5.0-Pro 0.557, GPT-Image-2 0.507, and Seedance 2.0 0.499. Even the strongest trained model remains well below human-level performance.

• Language chains are not the critical substrate: Degrading language reasoning in interleaved models has limited effect compared with removing visual states. Middle-state intervention shows intermediate visual states are causally used: corrupting or replacing them predictably degrades or redirects the final answer.

Methodology in Plain English

The authors treat visual reasoning as a task-space design problem rather than a data-collection problem. Instead of scraping examples, they write 300 parameterized programs — one per task — that sample configurations (grid size, object count, layout, appearance, difficulty) and instantiate problems. A task-specific solver computes the ground truth automatically, so supervision requires no manual annotation. Tasks are organized under five cognitive faculties adapted from the earlier VBVR work: perception, spatiality, transformation, abstraction, and knowledge, and a single task can carry several labels.

Each solved instance is rendered into several aligned forms of the same problem. Videos show the full trajectory. Images use one of three regimes depending on what the answer requires: Last-Frame when the final state suffices, Key-Frame when a few essential transitions matter (with solution trajectories drawn on the frames for path tasks such as maze navigation), and Multi-Frame when process integrity or temporal continuity is being judged. Interleaved versions add textual descriptions of intermediate steps, generated with Gemini-3.1-Pro-Preview using a per-task prompt applied uniformly. Every instance also gets a metadata file with its seed, problem specification, solution, and key element attributes, which later feeds both de-duplication and the scorers.

For evaluation, the authors reject the common practice of asking a vision-language model to grade outputs. They write a scorer per task that extracts semantic attributes with classical vision methods and grades those attributes against the recorded metadata. They then stress-test the scorers two ways: by measuring how often each evaluator gives different scores on re-run over the same videos, and by running a human arena study where annotators rate videos under a rubric, which is converted into pairwise preferences and compared against each automatic evaluator mapped into the same format.

Finally, they train nine open-source foundation models — image, interleaved text-image, and video generators ranging from 5B to 32B parameters — on the dataset for one epoch at 512×512, using Rank-32 LoRA for video models and for image/interleaved models above 10B parameters, with smaller models fully fine-tuned (video models at 16 fps). They then benchmark these alongside more than 30 proprietary and open-source models, and run counterfactual, ablation, and middle-state intervention experiments to test whether the model is really reasoning over visual states.

Why This Matters

Impact on research. The paper argues that the field's bottleneck is not data volume but the combination of task coverage, verifiable feedback, and controlled comparison. It also challenges a default assumption — that language chains of thought are the natural place for reasoning — by showing that intermediate visual states are causally used and that removing them hurts more than degrading language reasoning. The reproducibility finding (VLM judges changing scores on 54.6–92.8% of re-run samples even at temperature 0) is a general caution for anyone using model judges as RL rewards, not just for this suite.

Potential real-world applications:

• Training data for embodied and robotics systems, where spatial relations, object persistence, and dynamic interaction cannot be specified well in language.
• Reward models for reinforcement learning on generated media, where rollouts are too numerous and too costly for human or API-based grading.
• Evaluation infrastructure for generative media tools, giving teams a deterministic, cheap way to check whether a model's output actually satisfies a stated task rule.
• Assistive and educational content generation, for tasks such as step-by-step visual instruction, where the validity of each intermediate frame matters, not just the final image.

Industry relevance. The scorers replace API-based judging with lightweight deterministic computation, which directly addresses the cost problem of scoring many RL rollouts under a fixed compute budget. The accuracy–efficiency trade-off between interleaved and video generation (interleaved models matching video performance on many in-domain tasks at substantially lower cost) is a practical signal for teams choosing which generator family to fine-tune.

Future Directions

• Closing the human gap. The authors state that even the strongest trained model remains well below human-level performance, and that most academic open-source models lag substantially behind top proprietary systems. How to close that gap is left open.
• Extending the suite beyond the current 300 tasks. The paper treats broader capability coverage as the main driver of transfer, which implies that adding further vision-native task families is the natural scaling axis — but which capabilities to add is not addressed.
• Understanding the comparative advantage of video versus interleaved generation. The paper shows video wins where state transitions must be tracked and interleaved wins on cost, but does not specify the boundary conditions that determine which substrate is preferable for a given new task.
• Making verifiable scorers general rather than task-specific. Each of the 100 scorers is hand-built for one task. Whether that effort can be amortized or automated for new tasks is not reported.

Note: the provided paper content is truncated mid-sentence and does not include an explicit future-work section, so the items above are open questions the reported results raise rather than recommendations stated by the authors.

Target Audience

Researchers and engineers working on multimodal generative models, visual reasoning benchmarks, and reinforcement learning with learned reward signals. It is most useful to readers who already understand diffusion and autoregressive generation, since the paper's contributions are about task design, evaluation protocol, and controlled comparison rather than about a new architecture. Benchmark designers and evaluation teams will find the VLM-as-a-judge failure analysis and reproducibility table directly actionable. Readers looking for an introduction to visual reasoning will find the taxonomy and task figures accessible, but the training configurations and benchmark tables assume a working knowledge of the model families involved.

Authors’ abstract

Native visual reasoning treats visual generation as the medium of reasoning itself: visual states (i.e. images and videos) are not merely inputs to be understood or outputs to be rendered, but first-class substrates for problem solving beyond language. Yet progress remains bottlenecked by the lack of scalable training tasks, reliable feedback, and controlled comparisons across generative substrates. In this work, we introduce VBVR-Pro, a closed-loop testbed that makes native visual reasoning through generation trainable, verifiable, optimizable, and experimentally controllable. 1) Task scaling. VBVR-Pro turns visual reasoning into a controlled task space of 300 procedurally generated tasks. Models trained on VBVR-Pro show strong transfer beyond the proposed suite across seven external visual reasoning benchmarks such as RISE-Video, MME-CoF-Pro, and BabyVision. 2) Verifiable rewards. VBVR-Pro provides verifiable reward scorers for task-grounded evaluation. Through a systematic study of leading MLLMs as judges, we identify recurring failure modes of the prevalent VLM-as-a-judge paradigm. In contrast, the proposed scorers are grounded in deterministic, task-specific rules, achieve fine-grained alignment with human judgments. Importantly, they serve as reliable reward signals for large-scale multi-task reinforcement learning and demonstrate stronger post-RL performance across visual reasoning tasks. 3) Mechanism study. VBVR-Pro enables controlled modality studies across more than 30 image, video, and interleaved generators. Our analysis shows that video generation remains strongest for tasks requiring persistent spatiotemporal state tracking, while interleaved generation provides a compute-efficient alternative. Critically, ablations and probing suggest the presence of vision-native trajectories that are crucial to visual reasoning. We release all data, models, scorers, and code.