Image Diffusion Preview with Consistency Solver

AI summary

Overview

• Research area: Efficient inference for image diffusion models (text-to-image generation and instruction-based image editing), sitting at the intersection of numerical ODE solvers, reinforcement learning, and interactive generative interfaces.
• Technical level: Advanced. The paper assumes familiarity with probability-flow ODEs, diffusion sampling schedules, linear multistep methods, and policy-gradient reinforcement learning.
• Scope: The paper introduces a "preview-and-refine" workflow for diffusion generation and the trainable, RL-optimized solver (ConsistencySolver) that makes it viable, evaluated on Stable Diffusion, FLUX.1-Kontext, SDXL, and related models.

What This Paper Is About

Diffusion models generate high-quality images but are slow, because producing one image requires numerically integrating a differential equation over dozens of steps. This makes interactive use painful: users wait a long time only to discover the image does not match their intent.

The paper proposes Diffusion Preview, a two-stage workflow where a fast, low-step sampler produces a rough "preview" the user can judge, and only after the user approves it does the expensive full-step sampler produce the final image. The core technical challenge is making that cheap preview both high quality and consistent with what the final image will look like, so the user's approval decisions are meaningful.

Key Contributions

1. A trainable high-order ODE solver framework (ConsistencySolver). Derived from general linear multistep methods but simplified in three principled ways (explicit-only, anchored to the current state, timestep-conditioned coefficients), it predicts the integration weights with a small MLP rather than using fixed theoretical coefficients.

2. An RL-based optimization strategy for sampling dynamics. Instead of distilling a teacher trajectory or modifying model weights, the solver's coefficients are treated as a policy and optimized with PPO to maximize a perceptual similarity reward between low-step previews and full-step reference outputs.

3. The Diffusion Preview paradigm itself. A formalization of the preview-and-refine workflow with three stated desiderata — fidelity, efficiency, and consistency — and an argument for why existing acceleration methods fail at least one of them.

4. Comprehensive empirical validation across two diffusion families and several unseen models, plus human and LLM-proxy user studies measuring end-to-end interaction time rather than just distribution-level image metrics.

Main Findings

• Fewer steps, comparable quality: ConsistencySolver reaches FID scores on par with the Multistep DPM-Solver using 47% fewer sampling steps. At 8 steps it achieves FID 18.82 vs. 19.53 for Multistep DPM at the same budget, and at 12 steps it reaches 18.53 while Multistep DPM reaches 18.95.
• Consistency is where the margin is largest: Across six consistency metrics (CLIP, DINO, Inception, segmentation Dice, PSNR on pixels, and depth PSNR), the solver's previews match the final refined outputs substantially more closely than training-free solvers at matched step counts — for example, 78.5 segmentation Dice at 8 steps vs. 76.3 for Multistep DPM.
• It beats distillation baselines: Compared against LCM, PCM, Rectified Diffusion, DMD2, AMED, and the paper's own distillation-trained variant (Ours-Distill), the RL-trained solver delivers better fidelity and better preview-to-final consistency.
• Distillation harms consistency in ways FID does not capture: In a user study with Claude Sonnet 4 as proxy evaluator, single-step DMD2 satisfied only 57.0% (with GAN) and 47.1% (without) of GenEval prompts within 10 attempts, versus 94.2% for ConsistencySolver and 100% for the 40-step base model. This is a key argument that distribution-level FID is insufficient for interactive workflows.
• Real interaction time drops roughly in half: Measuring end-to-end time (denoising plus VAE decoding) across GenEval, COCO 2017, and LAION prompt sets, the preview mode cut average inference time by up to 55% on LAION, with only a small rise in the average number of attempts (6.00 to 6.35). Speedups ranged from 1.85x to 1.88x across evaluators and prompt sets.
• Cross-model transfer without retraining: A solver trained on SD1.5 transfers directly to SD1.4, DreamShaper, and SDXL. On SDXL at 10 steps, it reaches FID 23.32 vs. 26.32 for Multistep DPM; on SD1.4 at 5 steps, 20.22 vs. 25.22. The authors interpret this as evidence that different diffusion models share similar optimal sampling dynamics.
• Optimal configuration is Order 4 with a depth reward: Order 4 gave the best accuracy/complexity trade-off; order 5 yielded only marginal gains. Among six candidate reward models (Depth, Inception, CLIP, image PSNR, DINO, segmentation), depth maps gave the most balanced structural consistency.
• Instruction editing also improves: On FLUX.1-Kontext, ConsistencySolver achieves higher Edit Reward (0.86 vs. 0.83 for Multistep DPM at 5 steps) and better depth consistency (25.18 vs. 24.59).

Methodology in Plain English

Think of the diffusion sampling process as tracing a smooth path from random noise to a finished image. Solvers are the numerical recipes for walking that path in as few steps as possible. Classical solvers like DPM-Solver use fixed mathematical formulas to decide how far to step and how to combine information from previous steps.

This paper keeps the general structure of a classical linear multistep solver but replaces the fixed coefficients with values predicted on the fly by a tiny neural network that sees the current timestep and the next one. That network is trained with reinforcement learning: a preview is generated in a handful of steps, a high-quality reference image is generated with the full 40-step solver from the same starting noise and prompt, and a perceptual similarity score (by default, comparing depth maps) is used as the reward. The network's weights are updated with PPO so that its coefficient choices increasingly produce previews that resemble their full-step counterparts.

The design deliberately avoids two alternatives. Training-free solvers rely on theoretical assumptions that may not hold for real models. Distillation methods modify the model itself or train an auxiliary network to imitate trajectories, which is expensive and breaks the deterministic noise-to-image mapping that makes previews trustworthy. By only changing the solver and leaving the diffusion model untouched, the method preserves the property that a given prompt and random seed always lead to the same final image — which is exactly what makes a preview predictive.

Why This Matters

Impact on research. The paper makes a case that FID and other distribution-level metrics can mask real quality failures in accelerated models. Its user-satisfaction comparison against DMD2 — where two methods with similar FID diverge dramatically on whether users actually get an image they want — is a methodological argument that consistency, not just sample quality, deserves to be a first-class evaluation criterion. The RL-as-solver-optimization framing also offers a middle path between training-free numerics and weight-modifying distillation.

Real-world applications.

• Design prototyping and content creation tools: Designers can iterate through variants in seconds and commit compute only to the direction they like, cutting both wait time and cost.
• Consumer and mobile image generation: The solver is a tiny MLP that leaves the base model intact, so it fits naturally into resource-constrained deployments where on-device distillation would be impractical.
• Instruction-based photo editing: The FLUX.1-Kontext experiments target workflows like character reference, text editing, style transfer, and local edits, where users need to see whether an edit instruction will work before paying for it.
• Creative prompt iteration at scale: Marketing and asset-generation pipelines that explore many prompt/seed combinations before selecting candidates benefit disproportionately, since most attempts are discarded.

Industry relevance. Any product built on diffusion inference has a direct cost per generation. A solver that reaches target quality in roughly half the steps, and that can be dropped into an existing pipeline without retraining the base model, is attractive to both API providers paying for GPU time and product teams optimizing for latency.

Future Directions

• Beyond text-to-image and instruction editing: Whether the same solver framework transfers to video diffusion, 3D generation, or audio, where the sampling trajectory structure and reward design may differ substantially.
• Richer reward functions: The ablation found depth maps best among six simple perceptual metrics. Learned or task-specific rewards — or human-preference models — might push consistency further, and the RL formulation supports non-differentiable rewards naturally.
• Closing the remaining consistency gap: ConsistencySolver reaches 94.2% GenEval user satisfaction versus 100% for the 40-step base model. Understanding and shrinking that residual gap is the obvious next target.
• Adaptive or per-prompt solver behavior: The current solver conditions only on timesteps. Conditioning on the prompt or on intermediate state statistics could allow the solver to allocate steps where each prompt actually needs them.
• Theoretical grounding: The authors adapt the solvers' form from classical linear multistep theory but learn the coefficients empirically. A convergence or error analysis of the learned solver would strengthen the contribution.

Target Audience

Researchers and engineers working on diffusion model inference efficiency, particularly those building interactive or latency-sensitive generative products. It will be most valuable to readers who already understand diffusion sampling and want a method that improves few-step quality without retraining the base model. Practitioners evaluating acceleration options for a production pipeline — against training-free solvers on one side and distillation on the other — are the primary intended audience. The paper is not an entry point for someone new to diffusion models, though the user-study framing and the simplicity of the "preview then refine" idea make the motivation accessible to product and design audiences.

Authors’ abstract

The slow inference process of image diffusion models significantly degrades interactive user experiences. To address this, we introduce Diffusion Preview, a novel paradigm employing rapid, low-step sampling to generate preliminary outputs for user evaluation, deferring full-step refinement until the preview is deemed satisfactory. Existing acceleration methods, including training-free solvers and post-training distillation, struggle to deliver high-quality previews or ensure consistency between previews and final outputs. We propose ConsistencySolver derived from general linear multistep methods, a lightweight, trainable high-order solver optimized via Reinforcement Learning, that enhances preview quality and consistency. Experimental results demonstrate that ConsistencySolver significantly improves generation quality and consistency in low-step scenarios, making it ideal for efficient preview-and-refine workflows. Notably, it achieves FID scores on-par with Multistep DPM-Solver using 47% fewer steps, while outperforming distillation baselines. Furthermore, user studies indicate our approach reduces overall user interaction time by nearly 50% while maintaining generation quality. Code is available at https://github.com/G-U-N/consolver.