Research
OmniVerifier-C1: Decomposed Graded Verifier for Rule-Governed Generative World Modeling
Most visual verifiers — models that check whether a generated image matches a text prompt — output only a True/False verdict. That coarse signal cannot distinguish between an almost-correct image and…

- arXiv
- 2610.22355
- Published
- 2026-10-15
- Authors
- Bowei Liu, Xinchen Zhang, Xuhuan Li, Tianlin Zhang, Kaian Jiang, Jingwei Liu, Congyang Zhao, Yifei Qian, Junhong Liu, Zhiheng Li, Yujiu Yang, Yang Cai, Ling Yang
AI summary
Overview
Research area: Multimodal generative modeling — specifically visual verifiers used as generative reward models for reinforcement learning, applied to text-to-image generation and to rule-governed "world modeling" tasks.
Technical level: Advanced. The paper combines reinforcement learning theory (ranking accuracy, GRPO-style advantage normalization), a two-stage training pipeline, and image-to-image generation experiments across game-like structured tasks.
Scope: The paper proposes OmniVerifier-C1, a verifier that decomposes prompts or environment rules into fine-grained sub-questions scored individually and aggregated into a continuous reward, and shows it outperforms binary verifiers as a reward model for generative models, with a further refinement (pairwise verification against a ground-truth image) for rule-governed world modeling.
What This Paper Is About
Most visual verifiers — models that check whether a generated image matches a text prompt — output only a True/False verdict. That coarse signal cannot distinguish between an almost-correct image and a badly wrong one, which weakens the learning signal when the verifier is used as a reward model for reinforcement learning, and can encourage shortcuts or reward hacking. The paper's goal is to build a verifier that decomposes a complex prompt or set of environment rules into fine-grained sub-questions, judges each with a 0/1 decision, and turns the proportion satisfied into a continuous score that carries more information and generalizes better to unseen critique tasks. It then studies how this verifier behaves as a reward model in two settings: ordinary text-to-image generation, and rule-governed world modeling where a generated image must represent a correct next state of a game-like environment.
Key Contributions
-
Decomposed Graded Verifier (OmniVerifier-C1): a verifier that automatically splits a complex prompt or environmental rule set into fine-grained sub-questions, reasons over each one, and outputs a continuous score equal to the fraction of satisfied sub-questions — preserving partial correctness rather than collapsing everything into True/False.
-
Decoupled RL training strategy: a training scheme that converts each prompt–image pair into K+1 reinforcement learning samples — one for global alignment scoring and K for sub-question-level binary judgments — and optimizes them with separate rewards, so precise 0/1 local feedback builds the fine-grained capability that supports reliable global scoring.
-
Pairwise verification paradigm for world modeling: for rule-governed tasks where a ground-truth image exists, the verifier is given that ground-truth image as an extra input and compares generated state against it directly, removing the need to internally infer the target state and thereby removing the associated reasoning error.
-
Empirical demonstration as a generative reward model: the verifier is used to drive reinforcement learning for both text-to-image backbones and image-editing backbones on six constructed game-based tasks, with reported gains in semantic alignment scores, structured task accuracy, and generalization to harder, out-of-distribution instances.
Main Findings
-
Graded verifiers generalize better than binary verifiers on ViVerBench. Starting from Qwen3-VL-8B (overall 0.654), SFT on the graded decomposition followed by decoupled RL reaches an overall score of 0.684. Binary variants reach 0.671 with RL only, 0.656 with SFT only, and 0.672 with SFT+RL — consistently below the graded result. SFT on the graded format alone gives 0.659.
-
The supervision signal, not the training pipeline, is the deciding factor. The authors compare the binary verifier in both RL-only and SFT+RL settings because it does not need SFT to learn a decomposition format; it still underperforms, which they attribute to the sparsity of binary labels encouraging reliance on shallow patterns.
-
A theoretical ranking argument favors graded rewards. With the prompt losslessly decomposed into K sub-questions and z(x) the number satisfied, the graded reward r_c(x) = z(x)/K has ranking accuracy Acc_rank(r_c) = 1, while the binary reward r_d(x) = 1[z(x)=K] has Acc_rank(r_d) = 2q_K(1−q_K) / (1 − Σ_{k=0}^{K} q_k²) ≤ 1, which is strictly below 1 whenever two imperfect samples with different true quality levels exist.
-
Graded rewards improve text-to-image generation. On GenEval overall, Stable-Diffusion-3.5-medium moves from 0.66 to 0.77 with the Decomposed Graded Verifier (versus 0.68 with the Discrete Binary Verifier and 0.73 with UnifiedReward); CLIPScore moves from 32.8 to 34.4. FLUX.1-dev moves from 0.68 to 0.79 on GenEval overall and from 33.2 to 34.5 on CLIPScore, against 0.70/33.3 for the binary verifier and 0.74/34.0 for UnifiedReward.
-
Gains are spread across compositional sub-skills. For FLUX.1-dev with graded rewards, GenEval sub-scores are Single Obj. 1.00, Two Obj. 0.93, Counting 0.77, Colors 0.90, Position 0.43, Attri. Bind. 0.71, compared with baseline 0.99, 0.85, 0.61, 0.84, 0.22, 0.58.
-
Training with graded rewards is steadier. Intermediate checkpoints evaluated every 10 steps on a held-out set of 500 textual prompts, scored under the same decomposition-based protocol by Gemini3.1-Pro as an evaluator not involved in data processing, show steadier and more pronounced improvement than the binary verifier.
-
Pairwise verification beats pointwise in world modeling. On the six constructed game tasks (Normal / Hard), pairwise-trained models reach Maze 85.31 / 51.92, Sudoku 89.97 / 62.99, Auxiliary Line Construction 76.19 / 46.12, Sequence Prediction 73.48 / 63.64, Word Search 62.18 / 34.27, and 3D Projection 69.28 / 47.51; pointwise-trained models reach 67.81 / 34.78, 77.74 / 50.71, 66.98 / 37.49, 61.27 / 34.18, 50.92 / 26.94, and 52.19 / 33.16 respectively. The unmodified Qwen-Image-Edit-2509 baseline starts at 17.31 / 8.73 for Maze and 3.86 / 1.71 for 3D Projection.
-
Pairwise verification also wins on existing edit benchmarks. On RISEBench the overall score goes from 6.7 at baseline to 15.0 with SFT+RL(Pairwise), versus 11.7 for pointwise; on UniREdit-Game the average goes from 41.41 to 65.15 (pointwise 58.66), and on UniREdit-Real from 67.48 to 68.92 (pointwise 68.53).
-
SFT supplies execution priors, RL supplies reasoning. The authors attribute large SFT gains (for example Maze Normal 17.31 to 56.77) to domain adaptation: the editing backbone learns the visual style and transformation patterns of the tasks and begins performing meaningful operations rather than leaving images unchanged or distorting them, while RL adds the reasoning ability.
-
Reasoning is not only in the text encoder. The text encoder is frozen during both SFT and RL, and only the DiT parameters are updated, yet structured reasoning performance improves — implying the DiT itself can learn and express rule-governed reasoning capabilities.
-
Pointwise error decomposes into two terms. Formally, the pointwise score's deviation from truth is ε_point = ε_reasoning + ε_judge, whereas the pairwise score's deviation is ε_pair = ε_judge, because pairwise comparison removes target-state inference.
Methodology in Plain English
The researchers first build training data by taking prompts from ShareGPT-4o-Image and controlling prompt length to vary complexity, then generating images from several models (FLUX.1-dev, Qwen-Image, Z-Image) for diversity. They teach a Qwen3-VL-8B model the decomposition habit through supervised fine-tuning, distilling the sub-question decomposition paradigm from Seed1.8. Using that SFT model, they discard samples whose predictions exactly match the ground truth and apply a GPT-5 Bo10 filtering strategy that removes samples with accuracy below 60% while keeping the proportion of fully correct samples at 30%. The surviving samples are used for decoupled reinforcement learning with DAPO on 16 NVIDIA A800-80G GPUs for 80 steps, evaluated on ViVerBench.
The decoupled RL step is the core trick: each (prompt, image) pair becomes K+1 training samples. One asks the model for a global alignment score between 0 and 1, rewarded by 1 − |Ŝ − S*|, the closeness to the ground-truth score. The other K each ask about a single sub-question, rewarded by an exact-match indicator 1[Â_k = A_k*]. Mixing these in one optimization lets the precise local 0/1 feedback sharpen the model's fine-grained judgment, which in turn makes the global score trustworthy.
For guiding a generative model, the verifier's score S = (1/K) Σ J(q_k, y) becomes the reward signal for policy optimization. In the world-modeling setting, the input is an initial visual state plus a rule-based instruction, and the model must produce the resulting state image; the pointwise verifier must first infer the target state from the initial state and the rules before judging, while the pairwise verifier is simply handed the ground-truth image and compares against it. Generative backbones are trained for 200 steps with the DiffusionNFT algorithm on 16 NVIDIA A800-80G GPUs, with Stable-Diffusion-3.5-medium and FLUX.1-dev for text-to-image and Qwen-Image-Edit-2509 for the game tasks. The six game tasks cover path planning, logical reasoning, geometric construction, pattern induction, character recognition, and spatial reasoning, each with a Normal and a harder, larger-scale variant excluded from training to test generalization.
Why This Matters
Impact on research. The paper reframes the verifier not as an evaluation tool but as the bottleneck for turning image generators into structured reasoning agents. It supplies a formal argument (ranking accuracy under group-relative RL) for why graded rewards beat binary ones, and a clean error decomposition arguing that pairwise verification dominates pointwise when a reference image exists. It also reports that reasoning improves with a frozen text encoder, which challenges the assumption that reasoning in generation systems lives in the language side.
Real-world applications.
- Post-training of image generation and editing models, where a denser reward signal can target compositional details like counting, spatial position, and attribute binding rather than just overall plausibility.
- Simulation and planning interfaces where a model must render the next state of an environment under explicit rules — grid puzzles, route planning, geometric construction, and orthographic projection of 3D structures.
- Automated quality inspection pipelines for generated visual content, where a graded, per-sub-question critique is more actionable than a pass/fail flag.
- Reinforcement learning data curation, since graded scores preserve ordering among imperfect samples and provide within-group signal that binary labels wash out.
Industry relevance. The work directly addresses reward hacking and sparse-reward instability, two practical obstacles to scaling RL for visual generation. The two-stage recipe — SFT for domain execution priors, then RL for reasoning — plus the pairwise shortcut of supplying reference images, is a deployable pattern for teams training editing models on structured tasks. The involvement of Alibaba's Amap group and Tsinghua and Peking University suggests industrial interest in map-, navigation-, and layout-style structured generation.
Future Directions
- Where pairwise verification is impossible. The paper's central claim is conditional on ground-truth images being available; how to recover comparable reliability in open-ended text-to-image generation, where multiple outputs are valid, remains open.
- Robustness of the decomposition itself. The graded score depends on a lossless decomposition into K sub-questions; the paper does not report how sensitive results are to the number or granularity of sub-questions or to decomposition errors.
- Scaling and generalization beyond the six tasks. The game benchmark is constructed by the authors, and hard variants are excluded from training; broader evidence on unseen rule systems and larger instances would test the transferability claim.
- Separating SFT priors from RL reasoning more precisely. The reported pattern — SFT teaches execution, RL teaches reasoning — is inferred from benchmark scores; isolating the two components and probing what the DiT actually internalizes is a natural next experiment.
Target Audience
Researchers and engineers working on multimodal reward models, RLHF/RLAIF-style post-training for image generation, and generative world models will get the most from this paper. It also suits readers interested in verifier design for compositional instruction following and in structured reasoning benchmarks for generative models. A background in reinforcement learning and vision-language models is needed to engage with the theoretical section; the experimental sections are readable with only a general grasp of image generation evaluation. Some artifacts — the appendix proof beyond Lemma A.1 and parts of the implementation detail — are not visible in the provided content and are not summarized here.
Authors’ abstract
Generative visual verifiers play a crucial role in advancing multimodal intelligence. To enhance the robustness of visual verifiers in complex multimodal scenarios involving compositional requirements, we move beyond conventional binary verifiers that produce only True/False discrete judgments and introduce a verifier that automatically decomposes complex prompts or environmental rules into fine-grained components and assigns pointwise scores to provide a continuous and more…
The full abstract on arXiv