InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation

AI summary

Overview

Research area: Computer vision, specifically autoregressive visual generation (text-to-image, text-to-video, image-to-video, and long interactive video synthesis).

Technical level: Advanced. The paper assumes familiarity with visual tokenizers, vector quantization, VAR-style next-scale prediction, causal attention masks, and video diffusion baselines.

Scope: ByteDance researchers present InfinityStar, an 8B-parameter purely discrete autoregressive framework that unifies image and video generation by modeling a video as one image pyramid followed by a sequence of clip pyramids, reaching 83.74 on VBench and generating a 5-second 720p video in 58 seconds on a single GPU.

What This Paper Is About

Diffusion models produce high-quality video but are slow because they run tens to hundreds of sequential denoising steps and cannot easily extend videos beyond a fixed length, while existing autoregressive video models are inherently streamable but lag in visual fidelity and require tens of thousands of inference steps. InfinityStar asks whether a purely discrete, next-scale autoregressive model can close the quality gap with diffusion while being far faster and naturally supporting image-to-video and video extrapolation. The answer the paper gives is yes: a single architecture handles text-to-image, text-to-video, zero-shot image-to-video, and zero-shot video extrapolation, and can be further extended to multi-round interactive video generation.

Key Contributions

1. Spacetime pyramid modeling. A unified framework that decomposes a video into a first clip of a single frame (T=1, encoding static appearance) and subsequent multi-frame clips, where each clip is a 3D volume pyramid that grows only along spatial dimensions, not time. This decouples appearance from motion and lets a video model inherit text-to-image knowledge.

2. A discrete video tokenizer built by knowledge inheritance. Rather than training from scratch, the authors insert a parameter-free binary spherical quantization quantizer between the encoder and decoder of a pretrained continuous video VAE (Wan 2.1 VAE), then fine-tune. They also introduce Stochastic Quantizer Depth (SQD), which discards each of the last N scales with probability p during training (yielding 2^N possible scale schedules) to counter the imbalanced information distribution across scales.

3. Transformer modifications for the temporal dimension. Semantic Scale Repetition (SSR) repeats the first K_s scale tuples N times to refine early "semantic" scales; Spacetime Sparse Attention (SSA) attends only to the last scale of the preceding clip plus the current clip's preceding scales, instead of full history; and Spacetime RoPE is described in the appendix.

4. First discrete autoregressive model to reach industrial-level 720p video, plus an extension (InfinityStar-Interact) using Semantic-Detail conditions and sliding-window chunking for long interactive generation. All code and models are released.

Main Findings

• VBench state of the art among autoregressive models: InfinityStar scores 83.74 overall on VBench, versus 80.12 for Nova, 80.96 for Emu3, and 83.24 for diffusion-based HunyuanVideo. Sub-scores: Human Action 96.43, Scene 52.08, Multiple Objects 78.66, Appearance Style 21.81, Quality Score 84.73, Semantic Score 79.78. The VBench comparison in the paper notes HunyuanVideo at 13B parameters while InfinityStar is 8B.

• Strong text-to-image numbers: On GenEval, InfinityStar achieves an overall score of 0.79 with a prompt rewriter (Two Obj. 0.90, Position 0.62, Color Attri. 0.67), exceeding Infinity by 6% on overall score. On DPG it reaches 86.55 overall (Global 91.68, Relation 91.87), surpassing Infinity by 3.09%.

• Large speed advantage: Using tokenizers of the same compression rate, the abstract reports roughly a 10x inference-latency reduction relative to leading diffusion models for a 5s 720p video. In the latency table, InfinityStar generates a 5s, 81-frame 720p video in 58 seconds versus 1864 seconds for Wan 2.1 (14B) — a 32x speedup — and 354 seconds for Nova (0.6B) at 480p, a 6x speedup despite InfinityStar being 13x larger than Nova.

• Continuous-VAE knowledge inheritance improves the tokenizer: On an internal high-motion 480p/81-frame benchmark, continuous video VAE weights give PSNR 33.37, SSIM 0.94, LPIPS 0.065, versus image VAE weights at 29.10 / 0.90 / 0.123 and no pretrained weights at 30.04 / 0.90 / 0.124. The paper also shows faster convergence from video-VAE initialization.

• SQD helps both reconstruction and generation: SQD makes early-scale reconstructions markedly clearer, and improves VBench overall from 81.07 to 81.28 (a gain of 0.21), with faster convergence during video-generation training.

• Spacetime pyramid beats the pseudo-spacetime pyramid: The pseudo-spacetime variant (time held constant) raises VBench overall from 80.30 to 81.28 when replaced by the spacetime pyramid, and the paper reports that the spacetime version produces richer detail and higher motion.

• SSR is the largest single ablation gain: Removing Semantic Scale Repetition drops VBench overall from 81.28 to 75.72 (quality 76.73, semantic 71.68).

• Sparse attention beats full attention while costing less: SSA scores 81.28 versus 80.77 for full attention. SSA achieves a 1.5x speedup generating 192p 161 frames; for 480p 161 frames, full attention fails with OOM while SSA completes in 44.7s using 63GB VRAM. Alternative variants attending to the 3rd- or 6th-largest scale of the preceding clip score 80.86 and 80.75 respectively, with only 1.1x and 1.5x speedups reported.

• Human preference: In side-by-side human evaluation against HunyuanVideo-13B, InfinityStar-8B outperformed on all T2V metrics and was judged superior on I2V, particularly in prompt following and overall quality.

• Interactive generation: InfinityStar-Interact uses sliding-window 10-second chunks with 5-second stride, 7M curated videos longer than 7 seconds, and 16K synthetic multi-round interactive prompts (four rounds each). Semantic-Detail conditions compress condition token length from 33.6K to 5.8K for 480p generation, and maintain face-ID consistency where conditioning on only the last frames of the preceding clip does not.

Methodology in Plain English

The starting point is Infinity, an image generator that predicts tokens scale by scale — first a tiny block of tokens, then progressively larger blocks — rather than token by token, which is much faster. InfinityStar extends that idea to video by treating a video as an image pyramid for the first frame plus a series of clip pyramids for the rest. Each clip pyramid has K scales of shape (T, h_k, w_k), so the pyramid grows spatially but keeps its temporal length fixed. Clips are then generated one after another, conditioned on earlier clips and the text prompt, which in principle allows arbitrarily long videos.

To build the discrete tokenizer, the team took a trained continuous video VAE, dropped a parameter-free binary spherical quantizer between its encoder and decoder, and fine-tuned, so the discrete tokenizer inherits the VAE's knowledge instead of learning reconstruction from scratch. To stop the tokenizer from dumping all information into the last few scales, they randomly drop later scales during training (SQD), forcing useful content into the early scales.

For the Transformer, they made generation cheaper and better: the earliest scales carry global semantics, so they repeat those tuple positions several times so the residual predictions get refined more than once; and because attending over the entire video history would explode the sequence length, each clip attends only to the previous clip's largest scale plus its own earlier scales. A sliding-window scheme and split semantic/detail conditions then extend the model to interactively produce long videos with successive user prompts.

Why This Matters

Impact on research: The paper pushes the discrete autoregressive paradigm past the quality of some diffusion-based video generators while being an order of magnitude or more faster, which challenges the assumption that diffusion is necessary for high-fidelity video. It also shows a discrete tokenizer can be derived from a continuous video VAE by weight inheritance, an idea that could transfer to other tokenizer pairs, and demonstrates that a next-scale image model can be extended to video with minimal conceptual change.

Real-world applications:

• Rapid text-to-video content creation at 720p, where a 58-second single-GPU generation time makes iteration far more practical than multi-minute denoising runs.
• Interactive storytelling and streaming generation, where InfinityStar-Interact takes new user instructions round by round and continues a video instead of restarting.
• Video extrapolation and image-to-video for footage extension, storyboarding, or animating a still frame, both available zero-shot without task-specific fine-tuning.
• High-throughput image generation pipelines, since the same model handles text-to-image at GenEval 0.79.

Industry relevance: The work comes from ByteDance and is fully open-sourced with code and models, and its efficiency claims (32x over Wan 2.1 at 720p on a single GPU) target the serving-cost bottleneck that dominates commercial video generation.

Future Directions

• Closing the quality/motion trade-off. The authors state that fine-grained visual details can be compromised in high-motion scenes, so a better balance between image quality and motion fidelity remains open.
• Scaling up. The paper notes that limited compute prevented scaling training and parameter count to match leading diffusion models, which it says constrains the performance ceiling.
• Reducing cumulative drift in interactive generation. InfinityStar suffers from cumulative errors that visibly degrade quality as the number of interaction rounds grows, which the authors flag as a problem that must be addressed.
• Optimizing the inference pipeline. The paper says the pipeline has not yet been fully optimized, leaving headroom on the speed side, and the authors hope the release will catalyze work on fast, long video generation.

Target Audience

Researchers and engineers working on visual generation who already understand autoregressive transformers and diffusion baselines — particularly those interested in efficient video synthesis, discrete tokenizers, or unifying image and video in one model. It is most useful to practitioners who care about inference latency and 720p deployment, and to teams evaluating whether autoregressive generation is a viable alternative to diffusion. Beginners would find the tokenizer and attention details dense, but the high-level framing of appearance-versus-motion decoupling is accessible.

Authors’ abstract

We introduce InfinityStar, a unified spacetime autoregressive framework for high-resolution image and dynamic video synthesis. Building on the recent success of autoregressive modeling in both vision and language, our purely discrete approach jointly captures spatial and temporal dependencies within a single architecture. This unified design naturally supports a variety of generation tasks such as text-to-image, text-to-video, image-to-video, and long interactive video synthesis via straightforward temporal autoregression. Extensive experiments demonstrate that InfinityStar scores 83.74 on VBench, outperforming all autoregressive models by large margins, even surpassing some diffusion competitors like HunyuanVideo. Without extra optimizations, our model generates a 5s, 720p video approximately 10x faster than leading diffusion-based methods. To our knowledge, InfinityStar is the first discrete autoregressive video generator capable of producing industrial level 720p videos. We release all code and models to foster further research in efficient, high-quality video generation.