DP$^2$O-SR: Direct Perceptual Preference Optimization for Real-World Image Super-Resolution

AI summary

Overview

• Research area: Computer vision — real-world image super-resolution (Real-ISR) using large pre-trained generative models (diffusion and flow matching), combined with preference-alignment techniques adapted from language model training.
• Technical level: Advanced. The paper assumes familiarity with diffusion/flow models, DPO-style preference optimization, ControlNet conditioning, and no-reference/full-reference image quality assessment (IQA) metrics.
• One-sentence scope: The paper proposes a training framework that turns the stochasticity of generative super-resolution models into a supervision signal by ranking multiple outputs per image with a hybrid IQA-based reward and optimizing the model toward the top-ranked samples using a hierarchically weighted preference loss.

What This Paper Is About

Generative Real-ISR models built on text-to-image diffusion or flow backbones produce realistic details but also output different results for the same low-resolution input depending on the random noise seed — so quality varies from run to run. Most prior work treats this randomness as a defect to be suppressed or stabilized. This paper instead treats it as a free source of training signal: it samples many outputs per input, scores them with image quality metrics that approximate human preference, and trains the model to favor the better ones. The goal is higher perceptual quality and greater output consistency without any human annotation or handcrafted loss.

Key Contributions

1. A hybrid perceptual reward for Real-ISR. The reward combines full-reference metrics (LPIPS, TOPIQ-FR, AFINE-FR) and no-reference metrics (MANIQA, MUSIQ, CLIPIQA+, TOPIQ-NR, AFINE-NR, Q-Align), each group normalized across the sampled candidates and then averaged with equal 0.5 weight per group regardless of group size. Full-reference terms promote structural fidelity; no-reference terms promote naturalness. PSNR and SSIM are deliberately excluded as not perceptual.

2. A richer preference-pair curation scheme. Rather than Diff-DPO's single best-versus-worst pair drawn from different models, DP²O-SR samples M outputs from a single model using different noise seeds and pairs the top-N against the bottom-N, yielding N² possible preference pairs per input. The paper systematically studies how the sample count M and selection ratio N/M affect learning.

3. Hierarchical Preference Optimization (HPO). An extension of the Diff-DPO loss that adaptively weights each training pair at two levels: intra-group (emphasizing pairs with larger reward gaps) and inter-group (prioritizing candidate groups with greater reward dispersion). Pair weight is the product of the two.

4. Evaluation across contrasting backbones. The method is applied to ControlNet-SD2 (a 0.8B UNet diffusion model) and ControlNet-FLUX (a 12B MMDiT-based flow model), showing that the optimal data-curation settings depend on model capacity, and that gains transfer to the out-of-domain RealSR benchmark within 500 training steps.

Main Findings

• Perceptual gains on in-domain and out-of-domain data. On the Syn-Test set, DP²O-SR (SD2) improves LPIPS from 0.4332 to 0.4268 and TOPIQ-FR from 0.4336 to 0.4396; on trained no-reference metrics, MANIQA rises from 0.6684 to 0.7165, CLIP-IQA+ from 0.7595 to 0.8124, and QALIGN from 4.2481 to 4.5526.

• Generalization to metrics never used in training. On the untrained metric VQ-R1, both baselines improve: 4.38 to 4.57 for SD2 and 4.40 to 4.65 for FLUX (Syn-Test). Untrained fidelity metrics PSNR and SSIM decline, which the authors attribute to the well-established perception-distortion tradeoff.

• Strong standing against prior Real-ISR methods on RealSR. Compared against StableSR, DiffBIRv2, SeeSR, CCSR, AddSR, and OSEDiff, DP²O-SR (SD2) attains the highest MANIQA (0.7031) and CLIP-IQA+ (0.7852) among all listed methods; DP²O-SR (FLUX) reaches VQ-R1 4.32 and QALIGN 4.1492.

• Rapid early improvement. Figure 1 reports that within the first 500 training iterations C-FLUX moves from approximately 0.51 to 0.65 on the perceptual reward while C-SD2 reaches 0.62, surpassing SeeSR and OSEDiff, with larger models benefiting more from preference supervision.

• Optimal selection ratio depends on model capacity. Larger M improves stability and reward but with diminishing returns. C-SD2 performs best at N/M = 1/4, while C-FLUX performs best at lower ratios such as 1/16 and degrades at 1/2. C-SD2 exhibits reward collapse under low N or high N/M; C-FLUX stays stable and monotonic across most configurations. Final settings: N = 8, M = 32 for C-SD2 and N = 4, M = 64 for C-FLUX.

• Robustness, not just average quality. Across M samples, baseline models show rising Best@M and falling Worst@M as M grows, while Mean@M stays relatively stable. DP²O-SR improves all three statistics, most notably Worst@M, raising the quality floor. Under the CCSR protocol of sampling 10 outputs per input and reporting mean ± standard deviation on RealSR, DP²O-SR (SD2) improves MANIQA from 0.664±0.019 to 0.705±0.012 and MUSIQ from 70.34±1.79 to 73.24±0.81, and DP²O-SR (FLUX) improves QALIGN from 3.654±0.231 to 4.143±0.113.

• Global reward produces localized refinement. Although the reward uses only global IQA metrics, comparisons across three random seeds show DP²O-SR (FLUX) sharpening specific local structures such as wing venation while leaving unrelated regions (for example head reflections) nearly unchanged.

• HPO components are complementary. An appendix ablation reports perceptual reward of 0.645 for the baseline, 0.648 with intra-group weighting only, 0.649 with inter-group weighting only, and 0.651 with both.

• Reward type matters qualitatively. Figure 1(a) reports that full-reference-only training suppresses detail, no-reference-only training encourages hallucinations, and the hybrid reward preserves structure while improving realism.

Methodology in Plain English

1. Generate a diverse pool. For each low-resolution training image, run a frozen pre-trained super-resolution model many times with different random seeds, producing M candidate high-resolution outputs (M = 64 used during training data generation).

2. Score each candidate. Compute a battery of image quality metrics on each output. Full-reference metrics compare against the ground-truth high-resolution image; no-reference metrics judge the image alone. Scores are direction-aligned so higher is better, then min-max normalized within each input's candidate pool. Each candidate's reward is the average of the normalized FR scores and the average of the normalized NR scores, combined with equal 0.5 weight.

3. Build preference pairs. Select the top-N and bottom-N candidates per input, forming N² positive/negative pairs. Choosing N and M controls how much contrast versus diversity the training signal contains.

4. Train with weighted preference loss. Start from the same pre-trained model as the frozen reference and optimize a DPO-style objective on the pairs. HPO reweights each pair: pairs with a larger reward gap get more emphasis within a group, and groups with a wider spread of rewards get more emphasis across the dataset, with both weight terms normalized so the expected weight stays near 1.

5. Evaluate broadly. Use 14 IQA metrics in four categories — trained FR, trained NR, untrained NR perceptual, and untrained FR fidelity — across a synthetic test set whose degradations match training and the real-world RealSR benchmark whose degradations do not.

Training setup details reported: C-FLUX was fine-tuned for Real-ISR on roughly 1 million high-quality images with batch size 32, learning rate 1×10⁻⁴, and 45,000 steps; C-SD2 used batch size 256, learning rate 2×10⁻⁴, and 35,000 steps. Post-training used 30,000 of 30,100 curated internet images spanning six major scene types and 266 sub-categories, with the remaining 100 forming Syn-Test; degradation followed the first-order pipeline from ResShift. DP²O-SR itself used batch size 1024, learning rate 2×10⁻⁵, β = 5,000, and 1,000 iterations on 8×A800 GPUs. Preference pairs used 25 inference steps and CFG 2.5 for C-FLUX, and 50 steps and CFG 3.5 for C-SD2. Offline generation of 1.92 million images took approximately 168 hours for C-SD2 and 432 hours for C-FLUX, with an additional 72 hours for IQA labeling.

Why This Matters

Impact on research. The work shows that the stochasticity of generative super-resolution models — usually framed as a liability — can be reframed as a supervision source, and that preference optimization does not require human labels if a good proxy reward exists. It also provides empirical evidence that the right preference-pair curation strategy scales with model capacity, which is a transferable finding for anyone applying DPO-style methods to image generation tasks beyond super-resolution.

Real-world applications:

• Smartphone photography pipelines, where low-light, zoomed, or otherwise degraded images need enhancement with natural texture rather than smoothing.
• Old photo and archival restoration, where hallucinated content is unacceptable and structural fidelity must be preserved.
• Medical, satellite, or industrial imaging, where upscaled imagery must remain diagnostically or analytically faithful while still looking plausible.
• Media and content platforms that upscale user-uploaded images or video frames at scale, where consistency across repeated processing runs matters as much as peak quality.

Industry relevance. The author affiliations include OPPO Research Institute alongside The Hong Kong Polytechnic University, indicating direct relevance to consumer device imaging. The method is designed as a post-training step that lifts existing backbones rather than requiring a new architecture, and Figure 1 reports meaningful gains within only 500 steps, which lowers the barrier to adoption. Offline candidate generation is expensive in GPU hours, which is a practical cost consideration for deployment teams.

Future Directions

• Better reward models. The authors state as a limitation that the IQA-based reward lacks interpretability and does not fully capture subjective perceptual quality; designing more accurate and explainable reward models is left open.
• Online or iterative optimization. The current pipeline is fully offline. The authors suggest that iterative or online preference optimization could improve performance and adaptability — which would also reduce the large upfront cost of generating and labeling 1.92 million candidate images.
• Human validation. The main text defers a user study and additional visual comparisons to the appendix; whether metric-level gains correspond to human judgments at scale is a natural extension to verify.
• Generalizing the curation rules. The finding that optimal N/M depends on model capacity was established on two backbones (0.8B and 12B). Whether a predictable rule links capacity to the ideal selection ratio across a wider range of architectures is unresolved.

Target Audience

Researchers and engineers working on image restoration and super-resolution, generative model alignment, or diffusion/flow-based image synthesis. It is most useful to readers who already understand DPO-style preference optimization and want to see it adapted to a continuous-output vision task, and to practitioners who need to post-train an existing super-resolution backbone for better perceptual quality under limited sampling budgets. Readers seeking an introductory treatment of diffusion models or preference learning will find the paper assumes substantial background.

Authors’ abstract

Benefiting from pre-trained text-to-image (T2I) diffusion models, real-world image super-resolution (Real-ISR) methods can synthesize rich and realistic details. However, due to the inherent stochasticity of T2I models, different noise inputs often lead to outputs with varying perceptual quality. Although this randomness is sometimes seen as a limitation, it also introduces a wider perceptual quality range, which can be exploited to improve Real-ISR performance. To this end, we introduce Direct Perceptual Preference Optimization for Real-ISR (DP$^2$O-SR), a framework that aligns generative models with perceptual preferences without requiring costly human annotations. We construct a hybrid reward signal by combining full-reference and no-reference image quality assessment (IQA) models trained on large-scale human preference datasets. This reward encourages both structural fidelity and natural appearance. To better utilize perceptual diversity, we move beyond the standard best-vs-worst selection and construct multiple preference pairs from outputs of the same model. Our analysis reveals that the optimal selection ratio depends on model capacity: smaller models benefit from broader coverage, while larger models respond better to stronger contrast in supervision. Furthermore, we propose hierarchical preference optimization, which adaptively weights training pairs based on intra-group reward gaps and inter-group diversity, enabling more efficient and stable learning. Extensive experiments across both diffusion- and flow-based T2I backbones demonstrate that DP$^2$O-SR significantly improves perceptual quality and generalizes well to real-world benchmarks.