PTQ4ARVG: Post-Training Quantization for AutoRegressive Visual Generation Models

AI summary

Overview

Research area: Model compression for computer vision — specifically post-training quantization (PTQ) of autoregressive visual generation (ARVG) models.

Technical level: Advanced. The paper combines an analytical derivation of optimal scaling factors (Taylor expansion of quantization loss, differentiation, convex optimization) with architecture-level analysis of transformer models.

Scope: The paper identifies three quantization challenges specific to ARVG models and proposes a training-free PTQ framework, PTQ4ARVG, that quantizes the VAR, RAR, PAR, and MAR model families to 8-bit (W8A8) and 6-bit (W6A6) while preserving competitive generation quality.

What This Paper Is About

Autoregressive visual generation models can match or beat diffusion models at image generation, but they are large (for example VAR-d30 at 2B parameters, RAR-XXL at 1.5B, MAR-Huge at 1B, and PAR at 3B, with PAR-3B taking more than 3 seconds to generate a single image) and expensive to run. Post-training quantization can shrink and speed up such models without retraining, but the authors show that existing PTQ methods designed for LLMs, vision transformers, or diffusion models do not transfer well to ARVG. The goal is a training-free PTQ framework built around the specific properties of ARVG architectures.

Key Contributions

1. Identification of three ARVG-specific quantization challenges: (1) severe outliers at the channel-wise level in AdaLN-adjusted activations feeding MHSA and FFN; (2) highly dynamic activations at the token-wise level, caused by positional embedding information in the AdaLN input and by sensitive "sink" tokens from the conditional initial token; (3) mismatched distribution information at the sample-wise level, where activations are highly similar across input samples (especially unconditional ones), causing miscalibrated quantization parameters. The authors state these challenges appear across different layers, requiring layer-specific strategies.

2. Gain-Projected Scaling (GPS): a scaling strategy described as the first quantization scaling approach based on mathematical optimization. It expands the activation and weight quantization losses with Taylor series, defines the gain of scaling as the reduction in activation quantization loss minus the increase in weight quantization loss, and derives the optimal scaling factor in closed form by differentiation.

3. Static Token-Wise Quantization (STWQ): exploits two ARVG properties — fixed token sequence length and position-invariant distribution across samples — to assign static, fine-grained quantization parameters along the token dimension offline, distinguishing "sink" tokens from normal tokens in linear layers, with no online calibration overhead.

4. Distribution-Guided Calibration (DGC): measures each sample's distributional entropy with Mahalanobis distance and selects the top 50% highest-entropy samples as the calibration set, removing sample-wise redundancy. Together these form what the authors describe as the first comprehensive PTQ framework specifically for ARVG family models.

Main Findings

• 6-bit performance on RAR-B: PTQ4ARVG reaches FID 5.13, improving FID by 35.01 over the training-free OS+ and by 6.53 over the training-based OmniQuant. It also reports IS 206.17, sFID 12.68, and Precision 0.75, versus full precision (FP) at IS 292.80, FID 1.96, sFID 6.16, Precision 0.82.

• 8-bit results stay close to full precision: On VAR-d16, PTQ4ARVG reports IS 230.04, FID 4.06, sFID 12.23, Precision 0.79 (FP: 283.21 / 3.60 / 8.27 / 0.85). On VAR-d24 it reports IS 252.70, FID 3.36, sFID 13.24, Precision 0.77 (FP: 317.16 / 2.33 / 8.24 / 0.82). On RAR-XL at W8A8 it reports IS 304.18, FID 1.58, sFID 5.57, Precision 0.80 (FP: 308.54 / 1.54 / 5.31 / 0.80).

• Competing methods degrade sharply at low bit-width: On 6-bit RAR-B, QuaRot, RepQ*, OS+, and SmoothQuant all report FID above 40, and SVDQuant reports FID 125.51; on 6-bit MAR-H, SVDQuant reports IS 10.47 and FID 142.37, and on 6-bit PAR-XL-4×, SVDQuant reports FID 19.52. The authors note SVDQuant loses the advantage it shows on diffusion models because of ARVG's distinct activation distributions and autoregressive architecture.

• OmniQuant and SVDQuant fail at 6-bit on PAR and MAR: the paper reports that at 6-bit precision these methods fail on PAR and MAR, while PTQ4ARVG maintains competitive performance (for example PAR-XL-4× W6A6: IS 113.33, FID 12.87, sFID 6.80, Precision 0.62; MAR-B W6A6: IS 249.14, FID 2.99, Precision 0.78).

• Ablation confirms each component contributes: starting from a SmoothQuant baseline on 6-bit RAR-B (IS 31.04, FID 63.77, sFID 72.08, Precision 0.36), adding GPS yields IS 62.47, FID 36.51, sFID 24.53, Precision 0.46; adding STWQ yields IS 183.21, FID 6.67, sFID 12.74, Precision 0.71; adding DGC yields the full result.

• GPS beats other scaling methods on 6-bit RAR-B: SmoothQuant (IS 135.40, FID 10.26), RepQ* (92.44, 33.79), OS+ (161.63, 7.71), SQ+RepQ* (170.07, 7.43), and GPS (206.17, 5.13). Perturbing the GPS scaling factor by a random amount in the range [−0.3·s_GPS, +0.3·s_GPS] over 100 experiments produced worse quantization performance than the derived s_GPS.

• STWQ beats dynamic token-wise quantization without online overhead: on 6-bit VAR-d16, SQ+STWQ reports IS 151.60, FID 10.41, Precision 0.67 at 397.9 ms (2.922× speedup) versus SQ+DTWQ at IS 73.05, FID 30.14, Precision 0.49 at 473.9 ms (2.457× speedup). Full precision is 283.21 / 3.60 / 0.85 at 1163.0 ms (1.000×). The authors state that with batch size 100 and sequence length 256, DTWQ causes a 0.47× reduction in speedup compared to no token-wise quantization.

• Deployment gains: the 8-bit RAR-L and VAR-d20 models were deployed with a standard CUDA kernel on an RTX 3090 GPU at batch size 100; PTQ4ARVG achieves a 3.01× speedup and a 1.92× reduction in peak memory on VAR-d20 at sequence length 256.

• Calibration is cheap: the framework uses only 128 samples for calibration, and the authors state it introduces no additional inference overhead and needs no customized CUDA kernels.

• Remark 1 backed by statistics: when one activation channel's range exceeds another's, over 98% of channels satisfy s_i > s_j, and more than 99.5% satisfy R_x^i/s_i > R_x^j/s_j.

Methodology in Plain English

The authors first profiled ARVG models to find where quantization goes wrong, examining activations from a RAR-B block ("blocks.23") and across network layers. They found problems at three levels: some activation channels have extremely wide ranges; activation distributions shift strongly along the token dimension and include sensitive "sink" tokens; and different input samples produce very similar activations, so a naive calibration set poorly represents the real distribution.

They then attacked each level. For channel outliers, instead of heuristically balancing activation and weight ranges (as SmoothQuant or RepQ-ViT do) or learning scaling factors through backpropagation (as OmniQuant does), they wrote down an approximate expression for quantization loss using a Taylor expansion, expressed how a per-channel scaling factor changes both the activation loss and the weight loss, defined the net "gain" as the loss reduction minus the loss increase, and solved for the scaling factor that maximizes this gain by taking a derivative. The first channel (the one with the largest activation range) is fixed using s_1 = sqrt(R_x^1 / R_W^1); the remaining factors follow from the derived closed-form expression involving weight errors, activation errors, and the weights.

For token-wise variance, they exploited the fact that ARVG models always produce a fixed number of tokens and that token-position distributions are consistent across samples. That lets them compute percentile-based quantization parameters offline, once per token position, and separately for sink tokens versus normal tokens — rather than recomputing min-max ranges at every inference step as LLM-style dynamic quantization does.

For sample-wise mismatch, they score each candidate calibration sample by its Mahalanobis distance from the sample set mean and covariance, which measures how much the sample contributes to overall distributional entropy, and keep the top 50%.

Why This Matters

Impact on research: The paper reframes quantization for visual autoregression as a three-dimensional problem (channel, token, sample) rather than a direct port of LLM or diffusion techniques, and it provides a mathematical derivation for scaling factors in place of the empirical rules used by prior training-free methods. It also extends an under-explored area: the authors note quantization of ARVG is largely underexplored, and the closest prior work, LiteVAR, only assigns higher precision to quantization-sensitive layers rather than quantizing comprehensively.

Real-world applications (as framed by the paper's motivation — deployment on resource-constrained devices):

• Running image generation models on consumer or edge hardware where a 1B–3B parameter model and its memory footprint would otherwise not fit.
• Reducing inference latency for interactive image generation, where models such as PAR-3B take more than 3 seconds per image.
• Serving large numbers of image-generation requests where throughput and memory cost dominate operating expense.
• Fitting visual generation into multimodal pipelines alongside LLMs, given ARVG's architectural compatibility with language models.

Industry relevance: The framework is training-free (only 128 calibration samples), requires no customized CUDA kernels, and fuses scaling into weights offline, so it can be applied to already-trained checkpoints and deployed with standard kernels. That lowers the barrier for compression in production systems that cannot afford QAT's data and compute requirements. The reported 3.01× speedup and 1.92× peak-memory reduction on VAR-d20 give a concrete sense of the deployment payoff.

Future Directions

• Extending to lower bit-widths: the paper demonstrates 8-bit and 6-bit and mentions W4A8 tasks in the appendix, but the main text does not report how the framework holds up at 4-bit activation quantization, which remains an open question.

• Comparison gaps: the authors do not compare against QuaRot on PAR because PAR does not satisfy QuaRot's requirement that the number of heads and each head's dimension both be powers of 2. Whether a rotation-based or hybrid approach can be made applicable to such architectures is unresolved.

• Calibration selection beyond the top 50%: DGC uses a fixed 50% entropy threshold; the paper's own ablation varies calibration size, leaving open whether an adaptive or entropy-weighted selection rule would help further.

• Applicability to newer ARVG variants: the analysis is grounded in VAR, RAR, PAR, and MAR and their specific token prediction granularities; whether the same channel/token/sample challenges and the corresponding fixes transfer to future autoregressive visual architectures is untested.

Target Audience

Researchers and engineers working on model compression, efficient inference, or visual generation systems — particularly those who already understand transformer quantization basics and want the theoretical justification for scaling-based outlier suppression. It is also useful for practitioners deploying autoregressive image generation models on GPUs or memory-constrained hardware, since it reports concrete accuracy, speedup, and memory numbers on four model families. Readers without a background in quantization calibration or Taylor-expansion-based loss analysis will find sections 4.1 and the appendices demanding.

Authors’ abstract

AutoRegressive Visual Generation (ARVG) models retain an architecture compatible with language models, while achieving performance comparable to diffusion-based models. Quantization is commonly employed in neural networks to reduce model size and computational latency. However, applying quantization to ARVG remains largely underexplored, and existing quantization methods fail to generalize effectively to ARVG models. In this paper, we explore this issue and identify three key challenges: (1) severe outliers at channel-wise level, (2) highly dynamic activations at token-wise level, and (3) mismatched distribution information at sample-wise level. To these ends, we propose PTQ4ARVG, a training-free post-training quantization (PTQ) framework consisting of: (1) Gain-Projected Scaling (GPS) mitigates the channel-wise outliers, which expands the quantization loss via a Taylor series to quantify the gain of scaling for activation-weight quantization, and derives the optimal scaling factor through differentiation.(2) Static Token-Wise Quantization (STWQ) leverages the inherent properties of ARVG, fixed token length and position-invariant distribution across samples, to address token-wise variance without incurring dynamic calibration overhead.(3) Distribution-Guided Calibration (DGC) selects samples that contribute most to distributional entropy, eliminating the sample-wise distribution mismatch. Extensive experiments show that PTQ4ARVG can effectively quantize the ARVG family models to 8-bit and 6-bit while maintaining competitive performance. Code is available at http://github.com/BienLuky/PTQ4ARVG .