Stable Velocity: A Variance Perspective on Flow Matching

AI summary

Overview

Research area: Generative modeling — specifically flow matching and stochastic interpolants for image and video generation.

Technical level: Advanced. The paper combines stochastic-process theory (variance decompositions, posterior derivations, unbiasedness proofs) with large-scale empirical validation on state-of-the-art text-to-image and text-to-video systems.

Scope in one sentence: The paper reinterprets flow-matching training and sampling through the lens of conditional-velocity variance, splitting the diffusion trajectory into a high-variance regime near the prior and a low-variance regime near the data, and proposes three components — StableVM, VA-REPA, and StableVS — that exploit this split.

What This Paper Is About

Conditional Flow Matching (CFM) trains a network by regressing onto a single-sample "conditional velocity," which is only a Monte Carlo estimate of the true marginal velocity field it is trying to learn. That single-sample target has high variance at timesteps where the marginal distribution is still close to the noise prior, making optimization noisy and slow. The paper's goal is to characterize this variance precisely, then design training and sampling procedures that reduce it (during training) or exploit its absence (during inference) without changing the correct solution the model converges to.

Key Contributions

1. A variance characterization of flow matching. The authors quantify CFM target variance as the average trace of the conditional velocity covariance, evaluated on Gaussian mixture models, CIFAR-10, and ImageNet 256×256 latents from the Stable Diffusion VAE. They identify a low-variance regime (0 ≤ t < ξ) and a high-variance regime (ξ ≤ t ≤ 1), and show the split point ξ shifts toward 1 as data dimensionality grows.

2. Stable Velocity Matching (StableVM). An unbiased variance-reduction objective that replaces the single-sample conditional velocity target with a self-normalized importance-weighted average over n i.i.d. reference samples, drawn from a composite GMM conditional path.

3. Variance-Aware Representation Alignment (VA-REPA). A modification of representation alignment (REPA) that applies auxiliary semantic supervision adaptively, weighted by a function w(t) that emphasizes the low-variance regime.

4. Stable Velocity Sampling (StableVS). A finetuning-free acceleration method derived from closed-form simplifications of the dynamics in the low-variance regime, including DDIM-style posterior expressions for the reverse SDE and an exact ODE solution.

Main Findings

• Two regimes exist and are empirically consistent. CFM variance remains close to zero at small t and increases rapidly as t grows, in GMMs of varying dimensionality, CIFAR-10, and 256×256 ImageNet latents. In the low-variance regime, the posterior p_t(x_0 | x_t) concentrates on a single reference sample and the conditional velocity nearly coincides with the true velocity; in the high-variance regime it spreads over multiple samples.

• StableVM is unbiased and strictly less variable. Theorem 3.1 shows the StableVM target's expectation equals the marginal velocity v_t(x_t) and that its global minimizer is the true velocity field. Theorem 3.2 shows V_StableVM(t) < V_CFM(t) strictly; Theorem 3.3 gives a stronger bound showing variance decays at rate O(1/n).

• Training gains on ImageNet 256×256 with CFG (Table 1). With the SiT-XL backbone at 80 epochs, the method reaches FID 1.80, sFID 4.52, IS 272.4, precision 0.81, recall 0.60, versus REPA at 80 epochs (FID 1.98, IS 263.0), iREPA at 80 epochs (1.93), and REG at 80 epochs (1.86). At 400 epochs the method reaches FID 1.47 and at 480 epochs FID 1.44. Under the class-balanced sampling protocol (marked *), it reaches FID 1.71 at 80 epochs, 1.34 at 400, and 1.33 at 480, compared with REPA-E's 1.67 (80), 1.12 (800). The paper notes REPA-E attains lower FID but requires end-to-end fine-tuning of both autoencoder and diffusion transformer, while this pipeline keeps the autoencoder fixed.

• Scaling across model sizes without CFG (Table 2). At 100k iterations, FID improves from 52.06 to 49.69 on SiT-B/2 (130M), from 22.75 to 21.03 on SiT-L/2 (458M), and from 18.59 to 17.12 on SiT-XL/2 (675M). At 200k iterations on SiT-XL/2 the method reaches 10.56 versus REPA's 11.04; at 400k, 7.58 versus 8.13.

• Drop-in compatibility with REPA variants (Table 3). At 100k iterations: vanilla REPA improves from FID 18.59 to 17.12, REG from 8.90 to 8.11, and iREPA from 16.62 to 16.02.

• Split point ablation (Table 4). A smaller split point (ξ = 0.6) gives the best FID at early training (17.38 at 100k), but ξ = 0.7 becomes best as training progresses (10.56 at 200k, 7.58 at 400k). ξ = 0.8 degrades performance (17.85 at 100k, 7.62 at 400k), which the authors attribute to noisy supervision from the high-variance regime.

• Weighting and memory bank ablations (Fig. 4). Soft weighting schemes outperform the hard threshold, with the sigmoid weighting w_sigmoid(t) performing best. Memory bank capacity K = 256 already gives stable variance reduction; increasing to K = 1024 yields only marginal gains.

• Faster sampling on text-to-image (Table 6, GenEval, 1024×1024, ξ = 0.85). SD3.5-Large with Euler: 30-step baseline overall score 0.723 (Euler(19)); 20-step Euler(13) drops to 0.710 with PSNR 16.93, SSIM 0.753, LPIPS 0.333; StableVS(9) at 20 total steps matches the baseline at 0.723 with PSNR 36.92, SSIM 0.980, LPIPS 0.021. Flux-dev: 30 steps 0.660, 20-step Euler(13) 0.659 (PSNR 19.74, SSIM 0.820, LPIPS 0.244), StableVS(9) at 20 steps 0.666 (PSNR 35.45, SSIM 0.968, LPIPS 0.025). Qwen-Image-2512: 30 steps 0.733, Euler(12) at 17 steps 0.721 (PSNR 17.01, SSIM 0.767, LPIPS 0.277), StableVS(9) at 17 steps 0.731 (PSNR 32.27, SSIM 0.962, LPIPS 0.031).

• Faster sampling on text-to-video (Table 5, T2V-CompBench, 640×480, Wan2.2, ξ = 0.85). UniPC at 30 steps scores Consistency 0.842, Dynamic 0.120, Spatial 0.607, Motion 0.299, Action 0.749, Interact 0.708, Numeracy 0.476. UniPC(13) at 20 steps drops in several categories (Consistency 0.821, PSNR 15.61, SSIM 0.593, LPIPS 0.377), while StableVS(9) at 20 total steps recovers or exceeds the baseline (Consistency 0.843, Dynamic 0.123, Motion 0.289, Action 0.753, Numeracy 0.476, PSNR 31.10, SSIM 0.942, LPIPS 0.036).

• Ablation on β in Eq. (12). Only f_β ∈ [0,1] is introduced as a controllable parameter; the reported default is f_β = 0.

Methodology in Plain English

The authors start by measuring, rather than assuming, how noisy the CFM training signal is. For a given timestep, they compute the average squared distance between the single-sample conditional velocity and the true marginal velocity, averaged over the data distribution. Plotting this against time on Gaussian mixtures, CIFAR-10, and ImageNet latents reveals a consistent shape: near-zero variance at early timesteps, rising sharply later.

Two fixes follow from that shape. For training, instead of teaching the network to match one conditional velocity computed from one data sample, StableVM draws n reference samples, forms a mixture of their conditional paths, and asks the network to match the importance-weighted average of their velocities — normalized so the weights sum to one. Because the average is over samples drawn from the true data distribution, the target still points, in expectation, at the correct marginal velocity. A class-conditional memory bank of capacity K, updated FIFO-style, supplies enough same-class references for conditional generation even when a minibatch contains few examples of a given class.

For representation alignment, the authors observe empirically that REPA's auxiliary loss stays low and learnable in the low-variance regime but saturates at high values in the high-variance regime. They therefore weight the alignment loss by a function of time — a hard indicator, a sigmoid, or an SNR-based ratio — normalized across the minibatch so the term does not vanish.

For sampling, the observation that the posterior collapses to a single dominant data point early in the trajectory means the velocity field is effectively deterministic there. This lets the authors derive exact closed-form transition rules — a DDIM-style Gaussian posterior for the reverse SDE and an integral-form solution for the probability-flow ODE — and in the linear-interpolant case with β_t = 0, both reduce to a plain Euler step x_τ = x_t + (τ − t) v_t(x_t). Since the trajectory is a straight line with constant velocity in that regime, an arbitrarily large step is exact, so StableVS replaces the base solver only on [0, ξ] while leaving [ξ, 1] to the original solver.

Evaluation follows prior work's protocols: FID, IS, sFID, and precision/recall over 50K generated samples using an SDE Euler-Maruyama sampler with 250 steps, CFG scale w = 1.8 with interval-based scheduling; GenEval (553 prompts, 6 categories, 4 samples per prompt with seeds {0, 1000, 2000, 3000}) for text-to-image; T2V-CompBench (1,400 prompts across seven compositionality aspects) for text-to-video, with PSNR, SSIM, and LPIPS measured against a 30-step baseline. All ImageNet models use latents z ∈ ℝ^{32×32×4} from the pretrained Stable Diffusion VAE, with ξ = 0.7 and K = 256 for training components and 9 low-variance steps with ξ = 0.85 and f_β = 0 for StableVS.

Why This Matters

Impact on research. The paper replaces a loose empirical intuition — that diffusion training is noisy at high noise levels — with a measurable, dimension-dependent quantity and then derives objectives and samplers directly from it. Because StableVM's unbiasedness and O(1/n) variance decay are proved, and because the samplers are finetuning-free, the framework is a general-purpose add-on rather than a new model family. The demonstration that it works across vanilla REPA, REG, and iREPA suggests the variance perspective is orthogonal to existing acceleration techniques.

Real-world applications:

• Faster text-to-image generation in production systems, where 30-step baselines can be reduced to fewer total steps with PSNR 36.92 and SSIM 0.980 against the baseline on SD3.5-Large.
• Text-to-video pipelines, where StableVS with 20 total steps (9 low-variance) matches a 30-step UniPC baseline on T2V-CompBench consistency (0.843 versus 0.842).
• Lower training cost for latent diffusion transformers: matching or exceeding prior FID at 80 epochs versus REPA at 800 epochs, and improving FID at every checkpoint from 100k to 400k iterations.
• Drop-in adoption by teams already using REPA-style training, since the method requires no autoencoder fine-tuning and a memory bank of K = 256 suffices.

Industry relevance. The evaluated models — SD3.5, Flux, Qwen-Image, Qwen-Image-2512, and Wan2.2 — are all widely deployed or publicly released systems, and the sampling acceleration is described as finetuning-free, which lowers the barrier to integration into existing inference stacks. The author affiliations include the Kling Team at Kuaishou Technology alongside academic institutions, indicating direct industrial engagement with video generation.

Future Directions

• Choosing ξ without ground-truth data. The paper states that the exact split point ξ depends on the unknown data distribution and only offers a dimensionality-dependent trend as guidance. A principled, data-driven estimator for ξ remains open. The ablation shows the optimal ξ changes with training stage (0.6 best at 100k, 0.7 best at 400k), suggesting a schedule rather than a fixed constant could help.

• Reconciling the two ξ values. StableVS uses ξ = 0.85 while VA-REPA uses ξ = 0.7, and the authors report that the smaller value yields samples closer to the 30-step baseline while the larger allows more aggressive step reduction. Whether a single adaptive criterion could serve both training and sampling is not resolved.

• Extending the memory bank analysis. K = 256 is reported as sufficient with only marginal gains at K = 1024. The scaling behavior for very large n and for many-class or long-prompt conditional settings is not characterized beyond the class-conditional memory bank design.

• Broadening modality and solver coverage. The evaluation covers ImageNet 256×256, GenEval at 1024×1024, and T2V-CompBench at 640×480 across Euler, DPM-Solver++, and UniPC. Generalization to other resolutions, other interpolant choices beyond the linear one, and higher-dimensional or non-image modalities is not reported.

Target Audience

Researchers and engineers working on diffusion and flow-matching generative models who are comfortable with stochastic interpolants, posterior derivations, and unbiased estimator arguments. The training-efficiency results will be most useful to practitioners already using REPA-style representation alignment who want a drop-in improvement, while the sampling results will appeal to teams deploying SD3.5, Flux, Qwen-Image, or Wan2.2 who need to cut inference steps without retraining or losing perceptual quality. Readers looking for an introductory treatment of flow matching should start elsewhere, as the paper assumes familiarity with CFM, score-based diffusion, DDIM, and variance-reduction theory.

Authors’ abstract

While flow matching is elegant, its reliance on single-sample conditional velocities leads to high-variance training targets that destabilize optimization and slow convergence. By explicitly characterizing this variance, we identify 1) a high-variance regime near the prior, where optimization is challenging, and 2) a low-variance regime near the data distribution, where conditional and marginal velocities nearly coincide. Leveraging this insight, we propose Stable Velocity, a unified framework that improves both training and sampling. For training, we introduce Stable Velocity Matching (StableVM), an unbiased variance-reduction objective, along with Variance-Aware Representation Alignment (VA-REPA), which adaptively strengthen auxiliary supervision in the low-variance regime. For inference, we show that dynamics in the low-variance regime admit closed-form simplifications, enabling Stable Velocity Sampling (StableVS), a finetuning-free acceleration. Extensive experiments on ImageNet $256\times256$ and large pretrained text-to-image and text-to-video models, including SD3.5, Flux, Qwen-Image, and Wan2.2, demonstrate consistent improvements in training efficiency and more than $2\times$ faster sampling within the low-variance regime without degrading sample quality. Our code is available at https://github.com/linYDTHU/StableVelocity.