DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models

AI summary

Overview

Research area: Text-conditioned visual autoregressive (VAR) image and video generation, specifically inference-time diversity enhancement.

Technical level: Intermediate. The paper assumes familiarity with autoregressive generation, classifier-free guidance (CFG), diffusion-based diversity techniques such as CADS, and multi-scale latent encoders (VQ-VAE).

Scope: This paper introduces DiverseVAR, a training-free test-time framework that combines noise injection into text embeddings with a "scale-travel" latent refinement step to improve per-prompt diversity in VAR models while limiting the associated quality loss.

What This Paper Is About

Text-conditioned visual autoregressive models such as Infinity, Switti, and InfinityStar generate high-quality images and videos, but they produce nearly identical outputs when given the same prompt, even with different random seeds. Prior work has focused almost entirely on image quality, so this diversity limitation has gone largely unaddressed. The authors set out to fix it using only inference-time modifications, with no retraining, fine-tuning, or substantial additional compute.

Key Contributions

1. Identifying and reporting a diversity gap in text-conditioned VAR models. The authors state they are the first to systematically study and increase diversity in VAR models, showing that the limited per-prompt variation in models like Infinity and InfinityStar is a real and previously overlooked failure mode.

2. A systematic comparison of diversity-enhancement techniques for VAR. They adapt two families of diffusion-inspired techniques to the scale-wise autoregressive setting: CFG weight scheduling (piecewise constant and interpolation variants) and condition-annealing (noise injection at either the text embedding or the <SOS> token), evaluated across multiple schedules.

3. Scale-travel, a novel latent refinement method for VAR. Exploiting VAR's multi-scale autoencoder, scale-travel encodes a partially generated latent back into token scales, keeps only the first few scales, and resumes generation from that coarser point. This "rewinds" the trajectory along the scale axis to repair artifacts introduced by noise injection.

4. A unified diversify-then-refine pipeline (DiverseVAR) that combines text-embedding annealing with scale-travel. The authors show it advances the diversity-quality Pareto frontier on both image generation (Infinity, Switti) and video generation (InfinityStar-8B).

Main Findings

• VAR models exhibit severely limited per-prompt diversity. For a fixed text prompt, Infinity and InfinityStar produce little variation across different random seeds, as shown qualitatively in the paper's figures.

• Text-embedding noise injection is the strongest of the explored diversity techniques. Of the options tested (CFG scheduling, noise injected into the text embedding, noise injected into the <SOS> token), injecting noise into the text embedding yields the best diversity gains, but this comes with sharp degradation in image quality and noticeable visual artifacts.

• CFG scheduling increases diversity but degrades image quality significantly. The authors evaluate several CFG schedules over the scale index, including piecewise constant and interpolation schedules, and report that the quality cost is substantial relative to the diversity gained.

• Scale-travel recovers quality while preserving most of the added diversity. On MJHQ-30K with Infinity, condition annealing improves LPIPS-MPD by +69.7% over the Infinity baseline, but drops ImageReward to 0.49. Adding scale-travel raises ImageReward by +120.41% relative to the noisy condition-annealed samples, at a cost of −14.29% in LPIPS-MPD. Relative to the base Infinity model, DiverseVAR gives a +45.45% improvement in LPIPS-MPD with only a −11.48% change in ImageReward.

• Quantitative image results (Table 1). On MJHQ-30K, Infinity reports FID 19.16, ImageReward 1.22, LPIPS-MPD 0.33, Vendi 4.10; condition annealing gives 15.53 / 0.49 / 0.56 / 7.68; adding scale-travel gives 15.28 / 1.08 / 0.48 / 6.07. On MS-COCO, Infinity reports 37.37 / 1.16 / 0.37 / 4.41; condition annealing gives 22.51 / 0.39 / 0.58 / 8.13; adding scale-travel gives 28.97 / 1.04 / 0.52 / 6.68.

• The method generalizes to a second image model. On MJHQ-30K, Switti reports FID 16.18, ImageReward 1.11, LPIPS-MPD 0.44, Vendi 4.79; condition annealing gives 21.34 / 0.63 / 0.58 / 7.34; adding scale-travel gives 17.30 / 0.93 / 0.56 / 6.68. On MS-COCO, Switti reports 25.92 / 1.18 / 0.46 / 5.32; condition annealing gives 25.26 / 0.78 / 0.59 / 7.74; adding scale-travel gives 23.47 / 1.05 / 0.57 / 7.22.

• Video results on InfinityStar-8B (Table 2). The baseline scores 0.270 on VBench-2.0 Diversity and 0.806 on VBench Quality Score. Condition annealing raises diversity to 0.627 (+132.2%) but lowers the quality score to 0.794. DiverseVAR raises diversity to 0.487 (+80.4%) with a change of −0.002 points in quality score (0.804).

• DiverseVAR is robust to sampling hyperparameters. The authors report that DiverseVAR improves the diversity-quality trade-off across all tested CFG weights (ω = 2.0, 4.0, 6.5, 9.0 for Infinity; 2.0, 4.0, 6.0, 9.0 for Switti), and outperforms simply varying temperature (τ = 0.5, 1, 2, 5, 10) or nucleus sampling parameters (p = 0.9, 0.95, 0.97, 1.0). Varying the nucleus sampling parameter p had little impact on diversity on its own.

• Prompt rewriting is complementary but different in kind. Prompt rewriting via an LLM changes the condition itself, whereas DiverseVAR aims to increase diversity under a fixed prompt. The authors observe that for highly detailed prompts, prompt rewriting has less room to introduce meaningful variation, and that it can help preserve image quality when combined with their method.

• Stated limitations. Inference cost is slightly higher than plain noise injection, and the most diverse results still suffer from visual artifacts in approximately 6% of generated samples.

Methodology in Plain English

The authors start from the observation that in VAR models the text condition dominates generation, so different random seeds barely change the output. They test two ways to loosen that dominance at inference time.

The first is CFG scheduling: rather than using a fixed guidance weight throughout generation, they vary it as a function of the scale index (the VAR analogue of the diffusion timestep). The second is condition annealing: injecting Gaussian noise into the condition signal, either the text embedding or the <SOS> token that initializes autoregressive generation, using an annealing schedule that starts with more noise and decays. Comparing these options on a diversity-quality plane, they find the text-embedding variant gives the best diversity, but it also leaves visible artifacts.

To clean those artifacts up, they introduce scale-travel. VAR builds an image as a pyramid of token grids, from a single coarse grid up to fine ones. The authors note that early scales already carry the coarse layout and semantics of the final image. So instead of simply continuing generation from a corrupted intermediate state, scale-travel takes the current multi-scale feature map, runs the model's multi-scale encoder on it (rather than on a ground-truth image), and keeps only the first m scales. Discarding the finer re-encoded scales effectively rewinds generation to a coarser scale. Generation then resumes normally from scale m + 1 to K without noise injection, reconstructing clean, coherent details on top of the preserved coarse structure. This is described as a minor modification of the standard multi-scale encoding algorithm.

The full pipeline is therefore two-stage: diversify with text-embedding annealing, then refine with scale-travel. In their default configuration, scale-travel starts at scale l = 8, where they observed VAR forms the global structure and semantics. They use a constant CFG scheduler and a cosine condition-annealing scheduler, with k_min = 1 and k_max varied over {2, 3, 4, 5, 6} for annealing and over {1, ..., 10} for CFG scheduling.

Why This Matters

DiverseVAR reframes diversity as a first-class objective for VAR models, a family that had been evaluated almost exclusively on quality. By showing that a test-time, training-free intervention can move the diversity-quality Pareto frontier for both images and video, the work opens a research direction alongside the equivalent literature that already exists for diffusion models. It also matters for test-time search and selection methods, which require a spread of candidate outputs to choose from.

Real-world applications:

• Creative and design workflows: designers and artists sampling multiple interpretations of a single prompt before picking one to refine.
• Content generation at scale: producing varied marketing visuals, storyboards, or concept art from one brief rather than repeated near-identical images.
• Video and media production: generating varied keyframes and shots for a given scene description using a model such as InfinityStar-8B.
• Test-time search and reward alignment: supplying diverse candidates for downstream selection or alignment procedures that would otherwise have degenerate input.

Industry relevance: the method requires no retraining, no fine-tuning, and no substantial extra compute at generation, so it can be retrofitted onto existing deployed VAR image and video models with only inference-time changes. The authors report that it generalizes across Infinity, Switti, and InfinityStar-8B, and remains effective across different CFG weights, temperatures, and nucleus sampling settings, which matters for production systems where those parameters vary.

Future Directions

• Reducing the residual artifact rate. The authors report that roughly 6% of samples from the most diverse setting still contain visual artifacts, so better refinement or artifact detection remains open.

• Lowering the inference overhead of refinement. The paper states that scale-travel costs slightly more at inference than plain noise injection, motivating cheaper refinement.

• Extending beyond the keyframe stage of video generation. DiverseVAR was applied only to the keyframe stage of InfinityStar's two-stage pipeline; whether it helps the temporal generation stage is not reported.

• Better understanding of which condition components drive diversity. The authors explicitly frame their work as a first systematic exploration for VAR, leaving the role of different conditioning mechanisms and alternative schedules as open questions. Details on the scheduling functions and ablations for k_max are deferred to the supplementary material.

Target Audience

This paper is most useful to researchers and engineers working on visual autoregressive generation, text-to-image and text-to-video systems, and inference-time sampling strategies. It is also relevant to practitioners who want to increase output variety in deployed VAR models without retraining, and to researchers studying diversity-quality trade-offs in generative models more broadly, including those coming from the diffusion-model literature on CFG annealing and CADS. Readers without background in autoregressive or diffusion sampling will find the method section harder going, since it relies on concepts such as classifier-free guidance, annealing schedules, and multi-scale latent tokenization.

Authors’ abstract

We introduce DiverseVAR, a framework that enhances the diversity of text-conditioned visual autoregressive models (VAR) at test time without requiring retraining, fine-tuning, or substantial computational overhead. While VAR models have recently emerged as strong competitors to diffusion and flow models for image generation, they suffer from a critical limitation in diversity, often producing nearly identical images even for simple prompts. This issue has largely gone unnoticed amid the predominant focus on image quality. We address this limitation at test time in two stages. First, inspired by diversity enhancement techniques in diffusion models, we propose injecting noise into the text embedding. This introduces a trade-off between diversity and image quality: as diversity increases, the image quality sharply declines. To preserve quality, we propose scale-travel: a novel latent refinement technique inspired by time-travel strategies in diffusion models. Specifically, we use a multi-scale autoencoder to extract coarse-scale tokens that enable us to resume generation at intermediate stages. Extensive experiments show that combining text-embedding noise injection with our scale-travel refinement significantly enhances diversity while minimizing image-quality degradation, achieving a new Pareto frontier in the diversity-quality trade-off. Project page: https://diverse-var.github.io