MIRO: MultI-Reward cOnditioned pretraining improves T2I quality and efficiency

AI summary

Overview

Research area: Text-to-image (T2I) generative modeling, specifically reward alignment, pretraining objectives, and controllable generation with diffusion/flow-matching models.

Technical level: Advanced. The paper assumes familiarity with flow matching, classifier-free guidance, reward models, and RLHF-style alignment pipelines, and it includes formal theorems on reward-tilted sampling and diversity preservation.

Scope: The paper proposes MIRO (MultI-Reward cOnditioning), a pretraining framework that conditions a T2I generator on a vector of seven reward scores instead of applying filtering, finetuning, or reinforcement learning from human feedback (RLHF) as separate post-hoc stages.

What This Paper Is About

Standard T2I systems are built in three stages: pretraining on noisy web data, supervised finetuning on a curated subset, and RLHF against a single reward such as user preference. Each stage contracts the data distribution — filtering discards "low-quality" images that teach image structure, and RLHF collapses the model onto one scalar reward, hurting diversity and semantic fidelity while locking the quality trade-off at training time. MIRO instead annotates every pretraining image with seven reward scores and conditions the generator directly on that reward vector, so alignment happens during pretraining and reward trade-offs become user-controlled inputs at inference.

Key Contributions

1. MIRO, a multi-reward conditioned pretraining framework that integrates alignment directly into training, eliminating data filtering and post-hoc RLHF.
2. State-of-the-art results on GenEval and user-preference benchmarks (PickScore, ImageReward, HPSv2), with a 0.36B-parameter model surpassing FLUX-dev (12B) at 370× less training compute.
3. Up to 19× faster convergence than baseline pretraining and 3× less inference compute than FLUX under sample-based scaling.
4. Theoretical results showing that MIRO's multi-reward classifier-free guidance samples from a reward-tilted distribution, and that the supervised objective preserves the full data distribution and its entropy.

Main Findings

• Multi-reward conditioning beats single-reward conditioning across all metrics. Evaluated on the CC12M+LA6 dataset across AestheticScore, PickScore, ImageReward, HPSv2, and JINA CLIP score (with OpenAI CLIP score as an out-of-distribution metric not used in training), MIRO consistently outperformed all baselines. Single-reward AestheticScore models achieved high aesthetic scores but severely degraded other metrics.
• Training converges dramatically faster. The paper reports 19× speedup for AestheticScore, 6.2× for HPSv2, 3.5× for PickScore, and 3.3× for ImageReward over baseline. Figure 3 annotates the corresponding curves with 19.1×, 6.3×, 3.7×, and 3.5× labels.
• Compositional alignment improves. On GenEval with real captions, MIRO reached an overall score of 57 versus the baseline's 52 (a 9.6% improvement). Color Attribution rose from 29 to 38 (+31%), Two Objects from 55 to 68 (+24%), and Counting from 49 to 55 (+12%).
• Combining MIRO with synthetic captions gives the strongest results. Using 50% real and 50% synthetic captions, MIRO reached a GenEval score of 68 versus the synthetic-caption baseline of 57 (+19%). Position rose from 30 to 46 (+53%), Color Attribution from 43 to 52 (+21%), Single Object from 93 to 97 (+4%), Two Objects from 58 to 73 (+26%), and Counting from 44 to 61 (+39%).
• A custom aesthetic target further improves GenEval. The variant labeled MIRO†, which sets all positive reward targets to 1 except the aesthetic reward at 0.625, reached an overall GenEval score of 75, with Position 58 and Color Attribution 58.
• A 0.36B model outperforms a 12B model at a fraction of the compute. MIRO scored 68 on GenEval versus FLUX-dev's 67 while using 4.16 TFLOPs versus 1540 TFLOPs, described as a 370× efficiency improvement. MIRO's Position score of 46 improved on the previous state of the art of 34 (SD3-medium) by 31%, and its Color Attribution of 52 advanced on FLUX-dev's 47 (+11%).
• User-preference scores improve under inference scaling. On PartiPrompts with 128-sample inference scaling, MIRO reached 6.81 on Aesthetic Score versus FLUX-dev's 6.56, and 1.61 on ImageReward versus Sana-1.6B's 1.23 — while using 532 TFLOPs versus 1540 TFLOPs, a 3× efficiency advantage.
• Test-time scaling is far more sample-efficient. For ImageReward, MIRO with 8 samples matched the baseline with 128 samples (16× efficiency). For PickScore, MIRO matched the baseline's 128 samples with only 4 (32× efficiency). For Aesthetic Score and HPSv2, MIRO achieved with a single sample what the baseline could not reach even with 128 samples.
• Single-reward models trade off differently. Aesthetic-only models scored 33.0 on GenEval, CLIP, VQA, and JINA CLIP models scored 57 (matching MIRO), and the SciScore model scored the highest single-reward GenEval of 58, but with reduced aesthetic quality.
• Inference-time reward weighting controls trade-offs. Varying the aesthetic reward weight at inference produced the highest GenEval score at a weight of 0.625; the resulting MIRO† matched the GenEval performance of ImageReward-based selection with 128-sample test-time scaling using a single weighted selection, and reached ImageReward 1.18, matching FLUX-dev without test-time scaling.
• Qualitative training progression supports the quantitative gains. For the "tiger in a tuxedo" prompt, MIRO produced a visually appealing, properly composed image within 50k training steps, a quality level requiring 200k steps for the baseline; for "mad scientist panda," the baseline failed to generate a recognizable panda until 400k steps.
• Post-training with MIRO approaches but underperforms full training from scratch. This is reported as an ablation in the Supplementary Material.

Methodology in Plain English

MIRO works in three steps.

Dataset augmentation. Every image-caption pair in the pretraining dataset is scored by a set of N reward models covering different notions of quality: aesthetics, user preference, text-image alignment, visual reasoning, and scientific correctness. The paper's model uses seven rewards. Because raw reward scores have different scales and distributions, scores are normalized using a uniform binning strategy into B bins, which keeps quality levels balanced.

Multi-reward conditioned flow matching. The generator is trained with a flow-matching objective, which has the model predict the difference between a noisy sample and the clean image. MIRO modifies this objective so the model additionally receives the binned reward vector as a conditioning input, learning p(x | c, s) over images, captions, and quality. Every training image is kept at its own quality level rather than being filtered out, so low-, medium-, and high-scoring regions of the data are all modeled.

Reward-guided inference. At generation time, users specify the reward vector they want. High-quality generation simply conditions on the maximum bin for every reward. The method also extends classifier-free guidance to multiple rewards: the model is run with a positive reward target and a negative reward target, and the difference is amplified by a guidance scale ω. By default the positive target is all-maximum and the negative is all-zero, but users can set custom targets, isolate a single reward, or interpolate between two rewards while holding the others fixed. The paper proves that this guidance procedure corresponds to sampling from a reward-tilted distribution, where increasing ω concentrates samples on regions where high rewards are most likely relative to low rewards, and argues this steers toward the Pareto frontier rather than collapsing to extremes of any single reward. A second theorem states that the supervised objective covers the full reward spectrum, preserves the data distribution and its entropy, and lets users choose any point on the diversity-quality spectrum at inference.

For synthetic captions, MIRO handles the fact that some reward models cannot process captions longer than 77 tokens by generating both long captions (roughly 200 tokens) for training and shorter versions for reward model evaluation.

Why This Matters

Impact on research. The paper challenges the dominant three-stage recipe of pretraining, finetuning, and RLHF. It argues that alignment can be folded into pretraining itself, keeping all data rather than filtering it and preserving diversity rather than collapsing onto one scalar reward. It also provides theory connecting multi-reward guidance to reward-tilted sampling, and a formal comparison with RL-based alignment (DDPO) arguing that DDPO suffers gradient conflicts and mode collapse while MIRO preserves diversity.

Real-world applications:

• Content creation and design tools where a user wants to steer output toward aesthetics, prompt adherence, or preference depending on the job.
• Advertising and marketing imagery, where brand-appropriate aesthetics may matter more than literal prompt fidelity, or vice versa.
• Scientific and technical illustration, addressed by the paper through its scientific-correctness reward (SciScore).
• Deployment on constrained hardware: a 0.36B model using 4.16 TFLOPs and 532 TFLOPs at 128-sample scaling is far cheaper to run than a 12B model at 1540 TFLOPs.

Industry relevance. The compute-efficiency claims are the headline for practitioners: training efficiency (370× less training compute than FLUX-dev) and inference efficiency (3× less inference compute under sample-based scaling, 16× and 32× sample-efficiency gains on ImageReward and PickScore) directly translate to lower serving costs. The ability to expose reward trade-offs as inference-time knobs, rather than baking them in at training time, also lets one model serve many user preferences. Code and weights are stated to be available.

Future Directions

• Extending reward-aware pretraining beyond this setup. The authors state they hope the work "opens new research on reward-aware pretraining," leaving open how the framework scales to larger models and datasets.
• Selecting and weighting the reward set. The paper includes a leave-one-out reward analysis in the Supplementary Material, but the question of which rewards to use and how they should be normalized or binned remains open, along with the binning-strategy comparisons reported as an ablation.
• Improving post-training efficiency. The paper reports that fine-tuning an existing baseline with MIRO approaches but slightly underperforms full training from scratch, indicating an unresolved question of how to retrofit MIRO onto already-trained models.
• Better balancing aesthetics against semantic alignment. Test-time scaling by Aesthetic Score was observed to reduce GenEval performance, and inference weighting (with an optimum reported at 0.625) is presented as the mitigation; this tension is not fully resolved.

Target Audience

Researchers and practitioners in generative modeling who work on text-to-image alignment, controllable generation, or training efficiency. It is most useful to readers already comfortable with diffusion and flow-matching objectives, classifier-free guidance, and reward-model-based alignment, since the method's benefits and its theoretical claims are framed relative to those baselines. Readers interested primarily in the high-level question of whether alignment belongs in pretraining rather than in finetuning or RLHF will also find the paper relevant, though the full argument relies on the formal results in the appendices.

Authors’ abstract

The default paradigm of post-training text-to-image generators includes post-hoc selection of generated images, and subsequent training with one reward model to align the generator to the reward, typically user preference. This discards informative data as well as optimizes only for a single reward, hence harming diversity, semantic fidelity and efficiency. Instead, we propose MIRO, a method that conditions the model on multiple rewards during training, thus letting the model learn user preferences directly. MIRO pre-training both improves the visual quality of the generated images and speeds up the training, achieving state of the art on the GenEval compositional benchmark and user-preference scores (PickAScore, ImageReward, HPSv2).