Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards

AI summary

Overview

• Research area: Post-training (reinforcement learning) of text-to-image generative models, specifically reward design combining learned human-preference models with rubric-based evaluators.
• Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching models, Bradley–Terry preference modeling, GRPO-style group-relative RL, LoRA adapters, and weight-space model merging.
• Scope (1 sentence): The paper proposes and evaluates a three-stage reward-composition recipe for post-training two frontier text-to-image base models (FLUX.2-dev and Ideogram-4), measured by human pairwise votes on the Arena text-to-image leaderboard and by a held-out 1k-prompt validation protocol.

What This Paper Is About

Frontier text-to-image models already produce high-quality images, but improving them further with reinforcement learning is limited because no single reward signal captures everything users care about — a preference model rewards general aesthetics but can ignore whether the requested object was actually drawn, while targeted evaluators can be gamed by the policy. This paper's goal is to build a reward system and combination strategy that keeps the broad coverage of human preference learning while adding explicit, prompt-conditioned rubric checks (prompt faithfulness, constraint compliance, and detection of reward hacking) that survive optimization. The authors then apply that recipe to two production-scale base models and measure whether the resulting models are actually preferred by real users.

Key Contributions

1. A composed reward system for open-domain text-to-image RL: a Bradley–Terry preference reward trained on roughly 5 million pairwise human votes collected from the Arena leaderboard, combined with rubric-based rewards — a presence-oriented faithfulness reward (182k checklist questions over 10k prompts), an absence-oriented constraint reward (26k constraint questions over 10k prompts), and rubric anti-reward-hack detectors.
2. A reward-composition strategy consisting of reward-decoupled per-axis normalization (GDPO-style), prompt-conditioned gating of the constraint and anti-hack rewards, a one-sided veto that suppresses positive preference reward when a hack is detected rather than adding an extra reward term, and weight-space ensembling of policies trained under different reward configurations.
3. Demonstrated leaderboard gains on two frontier base models: an Elo 69 points above the FLUX.2-dev base model, and a post-trained Ideogram-4 reaching an Elo of 1223.5 (table value 1224 ± 13), which the authors report as surpassing every open-source model on the Arena text-to-image leaderboard as of the September 4, 2026 snapshot.
4. A released training subset, Arena-T2I-Training, described as a 1K subset of the training data that recovers some of the gains of full-scale training, plus a distributed open-source evaluation of the recipe.

Main Findings

• Composed rewards beat single-axis rewards. On FLUX.2-dev, preference-only training reaches win rates of 0.507 (Arena RM, Gemini-3.5-flash judge) and 0.461 (PickScore), while the Stage 1 composition reaches 0.642 and 0.581.
• Preference-only and faithfulness-only training each fail differently. Figure 1 reports that preference-only training drifts toward generic embellishment and ignores requested content, while faithfulness-only training washes out detail and drops requested style and scenery.
• The intent gate on the constraint reward matters. Gating raises the Gemini win rate from 0.6046 (ungated) to 0.6416 with Arena RM and from 0.5200 to 0.5810 with PickScore; without gating, constraint pressure flattens benign background detail, as illustrated by a background collapsing into a flat white field.
• Applicability gating on anti-hack detectors matters, and its effect is reward-dependent. With Arena RM, applicability gating raises the Gemini and GPT-5.4 win rates by 0.1086 and 0.0940 respectively.
• A one-sided veto beats an additive anti-hack reward. Under Gemini, treating detector compliance as an additive reward axis gives 0.5260 (Arena RM) and 0.4194 (PickScore); the one-sided veto used in Stage 2 gives 0.6346 and 0.5710.
• Cross-recipe weight souping beats same-run souping. Averaging two checkpoints from the same Stage 2 run yields 0.6435 (Arena RM) and 0.5610 (PickScore) under Gemini, while averaging a Stage 1 and a Stage 2 run yields 0.6598 and 0.6256.
• The two parent runs are complementary. Stage 1 has 204 clear losses against the base model and the gated-veto run has 234, with only 92 shared. Stage 1 wins concentrate in composition (65%) and style (16%), while Stage 2 wins more through faithfulness (16% vs 9%) and clean text rendering (9% vs 3%).
• The soup retains both parents' strengths. Stage 3 wins on 0.560 of Stage 1's exclusive loss set and 0.397 of Stage 2's, and still reaches a 0.304 win rate on prompts where both parents lose.
• Downstream RL performance scales with preference-reward training data. Using intermediate reward-model checkpoints trained on roughly 10K, 100K, and 500K pairwise votes gives MMRBv2 win rates of 0.537, 0.607, and 0.611, while the full approximately 5.6M-pair reward model reaches 0.642.
• The released 1K subset recovers part of the gain. With PickScore, Stage 1 and Stage 3 on the open-sourced 1K dataset reach Gemini win rates of 0.566 and 0.615, approaching the full 10K-dataset performance of 0.581 and 0.626.
• Transfer to a second base model. Applying Stage 1 to Ideogram-4 (with an additional OCR reward added because post-training degraded text rendering) yields 0.553 and 0.537 under Gemini with Arena RM and PickScore, and 0.623 and 0.525 under GPT-5.4.
• Arena leaderboard placement. On the table of public reference models, nano-banana-2 is listed at 1261 ± 5, seedream-5.0-pro at 1258 ± 4, and nano-banana-pro at 1232 ± 5, above the reported post-trained Ideogram-4 value.

Methodology in Plain English

The authors start with an already-strong open text-to-image model and fine-tune it with DiffusionNFT, an RL method that optimizes the model's forward (noising) process rather than the reverse sampling loop. They keep the base weights frozen and train LoRA adapters.

To score generated images, they build several signals. The first is a learned human-preference scorer trained with a Bradley–Terry objective on around 5 million real pairwise votes from the Arena leaderboard, using a Qwen3.6-27B backbone with a single linear head. The second and third are rubric rewards: an LLM converts each training prompt offline into a checklist of yes/no questions about objects, attributes, relations, and styles (faithfulness), and a separate checklist about undesirable or unrequested content (constraints). A VLM judge answers those questions during training, and the reward is simply the fraction answered positively. The fourth is a set of "hack detectors" that watch for known pathological behaviors, either by inspecting a single image (for example, garbled unrequested text) or by comparing the current policy's output against a generation from the frozen base model for the same prompt (for example, drifting toward cinematic photorealism when a cartoon style was requested), judged in both presentation orders to reduce position bias.

Combining these signals is the paper's central engineering problem. Each reward axis is normalized separately within a batch so no axis dominates by scale. The constraint reward is switched on only for prompts that an offline LLM classifies as "strict" intent, and each hack detector is switched on only when its failure mode is relevant to the prompt. Hack detectors do not add reward; instead, when a detector fires, any positive preference reward for that sample is removed, so the policy cannot earn points merely by satisfying the detector. The remaining signals are summed with equal weight (all weights set to 1), converted into a group-relative advantage across the rollouts for each prompt, and clipped. Finally, because runs trained with different reward configurations excel at different things, the authors average the LoRA deltas of two independently trained runs to produce a single adapter.

Training uses 10k filtered Arena user prompts and a 1k validation set, with 48 prompt groups and 24 rollouts per prompt at 512² resolution, with evaluation done by comparing checkpoints against the frozen base model using gemini-3.5-flash (and re-judging with GPT-5.4), following the MMRBv2 evaluation protocol.

Why This Matters

• For research: It argues that preference learning and rubric-based evaluation are complements rather than competitors, and it shows concrete failure modes of naive reward combination (additive hack rewards, ungated constraint pressure) that are likely to recur in other generative domains. The reported scaling relationship between preference-reward training data and downstream RL quality is also a useful design signal for anyone building reward models.
• For practice: The paper's specific design choices — per-axis normalization, prompt-conditioned gating, one-sided vetoes, and cross-recipe weight souping — are simple and reported as requiring no per

Authors’ abstract

Recent text-to-image generation models have achieved remarkable visual quality, but improving them through post-training remains challenging because no single reward signal captures the full range of human preference. In this work, we develop a simple and effective post-training recipe for open-domain text-to-image generation based on the composition of complementary reward signals. Our reward system consists of two main components: a preference reward, trained on large-scale human preference data using a Bradley-Terry objective to capture overall human aesthetic and perceptual preferences, and rubric-based rewards, which explicitly evaluate prompt faithfulness and other desirable properties while providing safeguards against reward hacking. A key challenge is how to combine these heterogeneous reward signals. We show that a naive weighted average leads to suboptimal optimization behavior, and propose a simple reward composition strategy that more effectively balances preference optimization with rubric satisfaction. In the Arena text-to-image leaderboard (https://arena.ai/), our RL-trained Flux2dev achieves an Elo rating 69 points above the base model, and our post-trained Ideogram-4 surpasses every open-source model on the leaderboard, reaching an Elo of 1223.5. (Claims of state-of-the-art performance are based on the Arena leaderboard snapshot as of September 4, 2026.) Our results suggest that effective rewards for frontier generative-model training require broad coverage of user intent and robustness to exploitation under optimization. To support reproducible research, we release Arena-T2I-Training, a 1K subset of training data that recovers some gains of full-scale training, providing a resource that we hope will facilitate future work on post-training for text-to-image models.