Glance Forcing: Making Bidirectional Diffusion Autoregressive at a Glance

AI summary

Overview

• Research area: Computer vision, specifically autoregressive (causal) video diffusion and few-step distillation of bidirectional diffusion models.
• Technical level: Advanced. The paper builds on Probability Flow ODE (PF-ODE) trajectories, consistency distillation, and Distribution Matching Distillation (DMD), so comfort with diffusion/flow-matching terminology is assumed.
• Scope: The paper proposes Glance Forcing, a dual-LoRA, phase-aware distillation framework that converts a bidirectional video diffusion teacher (Wan2.1-1.3B) into a 4-step autoregressive student using only a single training trajectory.

What This Paper Is About

Autoregressive video diffusion models can generate video in real time, but today's best way to build them distills a bidirectional diffusion teacher into a causal few-step student using large collections of offline PF-ODE trajectories, which cost tens of thousands of GPU-hours and terabytes of storage. The authors ask how far that supervision can be cut while keeping quality, push it to the extreme of one single trajectory, and find that a single monolithic student collapses under that setting. Their goal is a training scheme that makes one-trajectory distillation work as well as, or better than, the fully trained state-of-the-art baseline.

Key Contributions

1. A diagnosis of why one-sample distillation fails. The authors show that a student trained on a single trajectory suffers severe quality collapse, and trace this to the teacher's trajectory geometry: over 92.5% of cumulative trajectory curvature sits in the early, high-noise regime, while the low-noise regime behaves differently, so one network fitting both creates representation interference.
2. The Glance Forcing framework. A lightweight dual-LoRA design that splits denoising into a Slow-LoRA expert for high-noise semantic layout and motion, and a Fast-LoRA expert for low-noise detail, with the pretrained teacher kept entirely frozen so only LoRA parameters are optimized.
3. Two phase-specific objectives. A temporal frequency spectrum loss (1D-DFT magnitude matching along the temporal axis) for the Slow expert to suppress temporal flicker while preserving motion, and a spatial high-pass loss (fixed 2D Laplacian kernel) for the Fast expert to sharpen edges and textures, added without any extra student network.
4. A general mechanism claim about few-step distillation. The paper argues PF-ODE distillation mainly straightens the few-step trajectory geometry, while the later distribution-matching stage (DMD) independently governs distributional quality, and shows a single trajectory already suffices for the geometric part.

Main Findings

• One trajectory is enough for trajectory straightening. Using Mean Trajectory Straightness and Step-wise Cosine Similarity, the authors report that ODE initialization greatly improves both trajectory straightness and step-wise directional alignment at 4-step inference compared with the pretrained AR teacher's long-step trajectory, and that scaling ODE supervision from a single trajectory to 3K produces only marginal geometric change. DMD then improves generation quality while leaving overall trajectory geometry largely unchanged.
• Curvature is concentrated in high noise. Across 100 denoising trajectories from Wan2.1-T2V-1.3B (50-step denoising, noise schedule shift of 8, classifier-free guidance scale of 6, one deterministic seed per prompt), 91.82% ± 0.15% of total curvature mass accumulates at t ≥ 0.9, with a 95% bootstrap confidence interval of [91.53, 92.11]%; the high-noise-to-mid-noise curvature ratio is 32.4 ± 0.6 (95% CI [31.2, 33.6]). A temporal-difference version of the metric still places 87.9% of curvature mass at t ≥ 0.9 with a high/mid ratio of 18.6.
• Two experts beat one on both regimes. Splitting the ODE solution trajectory at a threshold timestep t_κ (N = 48, κ = 34 by default), the Slow expert outperforms the vanilla VCM baseline on structure and the Fast expert outperforms it on fine detail.
• Headline efficiency and quality result. Using only one PF-ODE trajectory, Glance Forcing raises the VBench Total score from 84.04 to 84.22 while cutting trajectory-collection cost from thousands of GPU-hours to about one A100-GPU-hour. The Figure 1 caption quantifies the training-cost reduction as 11,600x, with one extra sample needed for curation.
• Scores scale with budget. In the initialized-model setting, Glance Forcing reaches a Total of 82.69 at 3K data (Causal Forcing: 82.01) versus 82.09 for the one-sample variant. In the distilled setting at 25K data, Glance Forcing reaches 84.33 (Total), 85.01 (Quality), 81.64 (Semantic), 78 (Dynamic Degree), 6.921 (VisionReward), 69 (Instruction Following), compared with Causal Forcing's 84.04 / 84.59 / 81.84 / 68 / 6.326 / 56. The one-sample distilled variant ("Glance Forcing*") is listed at 84.22 / 85.50 / 79.11 / 75 / 6.870 / 64. The text also states that after autoregressive distillation Glance Forcing reaches 84.58, a figure that does not match the 84.33 in Table 1.
• Gains are largest on motion, reward, and instruction following. The abstract reports markedly larger gains in dynamic degree, VisionReward, and instruction-following than in the aggregate score.
• Human raters prefer it. In a pairwise study with three annotators on 50 prompts (150 votes per comparison), Glance Forcing wins 137 of 150 decided votes against Self Forcing (91.3% win rate) and 95 of 150 against Causal Forcing (63.3% win rate).
• Ablations confirm each component. VBench Total moves from 83.73 for the base variant to 84.05 with Slow–Fast decoupling, 84.20 with Slow–Fast plus the temporal frequency spectrum loss, 84.21 with Slow–Fast plus the high-pass loss, and 84.33 with all three. Text accompanying the table describes the Slow–Fast decoupling as improving the overall score from 83.98 to 84.33.
• Even step allocation is best. Among 4-step chunk allocations, S2–F2 is optimal (84.33 Total, 78 Dynamic, 6.921 VisionReward, 69 Instruction Following), while S1–F3 (83.92), S3–F1 (84.01), Slow-only S4 (83.42), and Fast-only F4 (83.15) are worse. Slow-only yields higher visual quality, Fast-only better preserves motion dynamics and semantic consistency.
• One-sample choice matters. A single sample supports both ODE initialization and later DMD refinement, but sensitivity varies: Low Motion and Open Domain start higher under ODE initialization, while High Motion (75.37) and Specific Domain (71.94) start lower; DMD recovers and improves all settings, with the largest gains for High Motion (to 82.72) and Simple Geometry (73.79 to 83.52). Semantic scores stay relatively stable across settings, so one-sample training mainly affects quality rather than semantic alignment.
• The method also accelerates bidirectional diffusion. Distilling 50-step Wan2.1 into an 8-step bidirectional model using one training sample yields VBench Total 82.16 at 8 NFEs (17.0 FPS, 1.38 s latency) and 81.14 at 4 NFEs (34.0 FPS, 0.69 s), versus 83.37 for Wan2.1 at 50 NFEs (0.78 FPS, 103 s) and 79.83 for LTX-1.9B at 20 NFEs.
• Small speed trade-off versus Causal Forcing. Glance Forcing runs at 16.4 FPS with 0.75 s latency versus Causal Forcing's 17.0 FPS and 0.69 s, both measured on a single H100 GPU excluding VAE time.

Methodology in Plain English

The authors start from the three-stage pipeline used by Causal Forcing: train a multi-step autoregressive teacher with teacher forcing, initialize a causal ODE student from that teacher, then apply asymmetric DMD with student self-rollout. Their first move is to test how little trajectory data the middle stage really needs, and they find that even one trajectory produces essentially the same trajectory geometry as 3K trajectories, because what matters is covering the diffusion time domain, not covering many videos.

Their second move addresses why one trajectory nonetheless breaks a normal student. When they measure how far the teacher's path deviates from a straight line to the endpoints, they find that nearly all of the bending happens at high noise, where global layout and motion are decided, while the low-noise part handles texture. One set of weights trying to fit both becomes an inconsistent regression target.

Their solution keeps the pretrained teacher frozen and attaches two LoRA adapters to it. Slow-LoRA is trained only on the high-noise sub-trajectory and handles layout and motion; Fast-LoRA is trained only on the low-noise sub-trajectory and handles detail. Each gets an extra objective matched to its job: the Slow expert matches the magnitude spectrum of a 1D discrete Fourier transform along time to keep motion steady, and the Fast expert matches Laplacian-filtered latents to keep edges crisp. At inference the two adapters are swapped in and out by timestep within a 4-step chunk (Slow at timesteps 1 and 0.937, Fast at 0.833 and 0.625).

Implementation specifics: the base model is Wan2.1-1.3B producing 480×832 videos with 81 frames via chunk-wise autoregressive generation; training uses 8× A100 GPUs, batch size 64, Adam at learning rate 2×10⁻⁶ with β₁ = 0 and β₂ = 0.999; LoRA rank 32 and alpha 128 inserted into all linear layers of the diffusion transformer; the one-sample runs use a global batch size of 1, bf16 mixed precision, gradient clipping of 1.0, 60 steps (60 effective epochs on the single sample), 8-bit Adam, and quantized weight loading. ODE distillation runs for 1K steps by teacher forcing on a 3K-sample slow-fast dataset built alongside a roughly 3K-sample bidirectional dataset generated with VidProM prompts; the DMD stage uses Wan2.1-14B as s_real and Wan2.1-1.3B as s_fake, matching Self Forcing and Causal Forcing settings.

Why This Matters

The work reframes what the expensive part of autoregressive video distillation actually is: not data volume, but coverage of the diffusion time domain, combined with the observation that correction and distribution refinement are separable jobs. If one A100-hour of trajectory collection can match a baseline trained with full trajectories, the practical barrier to building causal, low-latency video generators drops sharply, and a general mechanism (straighten geometry first, then match distributions) is proposed for other few-step distillation settings.

Real-world applications:

• Real-time interactive video generation, such as streaming avatars or live content tools, where per-frame latency must stay low.
• Video world models for agents and simulation, where rollouts must be produced causally and cheaply.
• Low-cost adaptation of large video diffusion models to new styles or domains, since only LoRA parameters are trained and the teacher stays frozen.
• Accelerating existing bidirectional diffusion pipelines, as demonstrated by turning 50-step Wan2.1 into an 8-step generator.

Industry relevance: the training-cost reduction (11,600x in the Figure 1 caption, and thousands of GPU-hours down to roughly one A100-hour) and storage savings directly address the preprocessing bottleneck that makes autoregressive video distillation expensive in production. The reported throughput (16.4 FPS) and latency (0.75 s on H100) position the output model for latency-sensitive deployment, though it is slightly slower than Causal Forcing at the same data scale.

Future Directions

• Sample selection. Because results vary substantially with the single chosen sample (High Motion and Specific Domain degrade most under ODE initialization), a principled criterion for picking or curating that one sample is an open question.
• Beyond two experts and a fixed split. The split timestep t_κ (κ = 34 of N = 48) and the S2–F2 allocation were chosen empirically; whether more experts, learned boundaries, or adaptive allocation help is not addressed.
• Generalization to other backbones and tasks. Only Wan2.1-1.3B is used as the base, with one extension to 8-step and 4-step bidirectional acceleration; transfer to other video diffusion families remains untested.
• Resolving reported ambiguity. The text's figure of 84.58 after autoregressive distillation conflicts with the 84.33 in Table 1, and the paper does not report a per-sample storage or wall-clock breakdown beyond the aggregate cost claim.

Target Audience

Researchers and engineers working on diffusion distillation, autoregressive and causal video generation, and real-time video generation systems. It is most useful to readers already familiar with flow matching, consistency distillation, and DMD, since the paper's distinctive insight depends on distinguishing trajectory geometry from distributional quality. Practitioners constrained by GPU budget who want to reproduce a causal video generator from a bidirectional teacher will also find the implementation details and ablation tables directly actionable.

Authors’ abstract

Autoregressive generation underlies real-time, causal video synthesis and is the backbone of video world models. Today's dominant approach distills a bidirectional diffusion model into a causal few-step student, but existing pipelines make this conversion prohibitively expensive, requiring offline Probability Flow ODE} (PF-ODE) trajectories that cost tens of thousands of GPU-hours and terabytes of storage. We show it can be made dramatically more efficient. Pushing this efficiency to its extreme, we reduce trajectory supervision to a single sample, the strictest setting possible, and find that it causes severe quality collapse. We trace this to the teacher's trajectory geometry: the high-noise regime, governing semantic layout and motion, and the low-noise regime, governing fine texture exhibit fundamentally different curvature and optimization behavior, so a single monolithic student cannot jointly fit both, causing severe representation interference. Motivated by this, we propose Glance Forcing, a lightweight distillation framework that decouples the denoising trajectory into two complementary slow and fast experts, each guided by a tailored, stage-specific objective, while keeping the pretrained teacher entirely frozen. This decoupling further reveals a general mechanism underlying few-step distillation: PF-ODE distillation mainly straightens the few-step trajectory geometry, while the subsequent distribution-matching stage independently governs distributional quality. Using only one PF-ODE trajectory, Glance Forcing already surpasses the fully-trained state-of-the-art baseline, raising the VBench Total score from 84.04 to 84.22 while cutting trajectory-collection cost from thousands of GPU-hours to about one A100-GPU-hour, with markedly larger gains in dynamic degree, VisionReward, and instruction-following, and consistent preference from human raters over strong baselines.