AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models

AI summary

Overview

Research area: Computer Vision — specifically post-training quantization (PTQ) for diffusion models, sitting at the intersection of generative modeling and model compression.

Technical level: Advanced. The paper derives gradient expressions for a multi-step denoising objective and analyzes memory complexity, though its core intuition can be stated simply.

Scope: The paper proposes AccuQuant, a PTQ calibration method that simulates multiple denoising steps at once so that quantization parameters account for errors accumulated across sampling steps, with a gradient approximation that reduces calibration memory complexity from O(n) to O(1).

What This Paper Is About

Diffusion models generate images by running a long sequence of denoising steps, and compressing them to low bit-widths is attractive because each step is expensive. The problem is that quantization error made at one step feeds into the input of the next step, so errors pile up along the sampling trajectory rather than staying local. Existing PTQ methods typically calibrate each denoising step independently and ignore this accumulation, which causes severe image-quality degradation — especially at low activation bit-widths. AccuQuant's goal is to calibrate quantization parameters while explicitly accounting for errors accumulated over several denoising steps, without paying a large memory cost.

Key Contributions

1. Multi-step calibration for diffusion quantization. AccuQuant groups consecutive denoising steps and minimizes the difference between the outputs of full-precision and quantized models across each group, so accumulated quantization error is directly accounted for during calibration. This contrasts with prior methods that imitate a single step at a time.

2. A gradient approximation technique with a new objective. The authors show that the term coupling consecutive steps is dominated by a scalar ratio of noise-scheduling coefficients, and derive a new loss whose gradient matches an efficient approximation of the full multi-step gradient. This reduces memory complexity from O(n) to O(1) in the number of denoising steps per group.

3. An analysis of quantization error in diffusion models. The paper analytically separates overall quantization error into a step error (from the estimated noise at the current step) and an accumulated error (carried along previous steps), and empirically shows the accumulated term grows drastically by comparison.

4. Extensive evaluation across tasks and models. Experiments cover unconditional, class-conditional, and text-to-image generation on CIFAR-10, LSUN-Churches, LSUN-Bedrooms, ImageNet, and MS-COCO, against Q-Diffusion, TFMQ-DM, TAC, PTQD, PTQ4DM, and PCR.

Main Findings

• Accumulated error dominates step error. In the DDIM sampling analysis on CIFAR-10, the accumulated error term increases drastically over denoising steps while the step error remains relatively constant. Tracking FID2FP32 every 5 steps with Q-Diffusion shows the score climbing along the sampling process, confirming that quantization errors accumulate.

• AccuQuant leads on FID2FP32. FID2FP32 — the FID between quantized-model outputs and full-precision outputs — is where the method shows its largest margins. On CIFAR-10 (DDIM, steps=100) at 4/8 bits, AccuQuant reports FID2FP32 of 1.15 versus 3.26 for Q-Diffusion and 1.52 for TFMQ-DM. At 6/6 bits it reports 3.30, versus 35.24 (Q-Diffusion) and 3.64 (TFMQ-DM).

• Large gains in low activation-bit settings. At 3/6 bits on CIFAR-10, AccuQuant reports FID 9.89 and FID2FP32 6.69, compared with FID 40.94 / FID2FP32 42.74 for Q-Diffusion and FID 29.29 / FID2FP32 27.11 for TFMQ-DM.

• Strong results at low bits on LSUN. On LSUN-Churches with LDM-8 (steps=500) at 3/8 bits, AccuQuant reports FID 5.04, sFID 10.95, FID2FP32 1.26, while Q-Diffusion reports FID 5.68 / FID2FP32 1.49 and TFMQ-DM reports FID 9.03 / FID2FP32 10.48. At 3/6 bits AccuQuant reports FID 8.91 and FID2FP32 5.79 versus FID 47.49 / FID2FP32 42.27 for Q-Diffusion.

• Class-conditional gains with matching metrics. On ImageNet 256×256 with LDM-4 (steps=20), at 3/8 bits AccuQuant reports FID 6.61, FID2FP32 3.86, LPIPS 0.2937, PSNR 19.9392, SSIM 0.7649. At 3/6 bits it reports FID 5.95, FID2FP32 6.31, LPIPS 0.3296, PSNR 19.0135, SSIM 0.7296. The full-precision reference is FID 11.13, sFID 7.85, IS 368.19, FID2FP32 0.00.

• Beats PCR at fewer bits on text-to-image. On MS-COCO 512×512 with Stable Diffusion v1.4 (steps=50), AccuQuant at 4/8 bits reports FID 22.73, FID2FP32 10.99, and CLIP score 26.85, while PCR at 4/8.4 mixed-precision reports FID 22.04, FID2FP32 14.25, and CLIP score 26.48. The full-precision CLIP score is 26.46.

• Group size is a trade-off. On CIFAR-10 (DDIM, 6/6-bit) FID2FP32 falls from 5.62 at group size 1 to 4.48 at size 2 and 3.30 at size 5, then rises to 4.30 at size 10, 14.19 at size 50, and 57.96 at size 100. The authors report that dividing timesteps into 10 to 20 groups yields consistently strong results, and use 20 groups for unconditional, 10 for class-conditional, and 25 for text-to-image generation.

• Gradient approximation helps rather than hurts. With the approximation, AccuQuant reports better FID and FID2FP32 at every tested bit-width (6/6, 4/8, 4/6, 3/8, 3/6) than without it, while using much less memory. The authors attribute this to the Jacobian term being small on average but highly variable (for example, at timestep 100 the full gradient is 1.0221 ± 0.112, the Jacobian term is −0.1236, and the scalar ratio is 1.1457), so omitting it stabilizes calibration.

• Qualitative improvement. Visual comparisons on LSUN-Bedrooms and LSUN-Churches at 3/8 bits, and text-conditioned generations from SD v1.4, show AccuQuant preserving details such as the shape of a hat and the pose of a subject better than Q-Diffusion and PCR. TAC results are absent where no source code was available.

Methodology in Plain English

The authors start from an observation about the sampling loop: each denoising step takes the previous step's output as input, so any error in that output is carried forward and compounded. They write down the quantized model's output at a step as the full-precision output plus two terms — one from the current step's noise-estimation error, and one from accumulated error — and show the accumulated term is the one that explodes.

Their fix is to change what calibration compares. Instead of comparing full-precision and quantized outputs at a single step, they break the denoising sequence into groups of M consecutive steps. For each group, they run the full-precision model and the quantized model through all M steps, but crucially each model uses its own previous output as input. They then adjust the quantization step-size for that group to shrink the gap between the two models' outputs at the end of the group. Calibration proceeds group by group along the sequence.

The naive version of this requires keeping every intermediate activation from all M steps to compute gradients, which is memory-heavy. The authors note that the mathematical relationship between consecutive outputs is a sum of a simple scaling term and a derivative of the noise network. Measuring these during calibration shows the simple scaling term dominates while the second term is small but noisy. Dropping the second term turns a long chain of stored derivatives into a single ratio of noise-schedule coefficients, and they design a loss function (using a stop-gradient operator) whose gradient matches this simplified expression. The result is the same calibration behavior with constant memory, regardless of group size.

Why This Matters

Impact on research. The paper reframes diffusion quantization as a trajectory-matching problem rather than a per-step approximation problem, and its error decomposition gives a concrete explanation of why per-step PTQ methods break down at low bit-widths. The gradient approximation also suggests that long chains of denoising steps can be differentiated cheaply in other contexts where multi-step generation is calibrated or fine-tuned.

Real-world applications:

• On-device or edge image generation, where low bit-widths are needed to fit diffusion models into limited memory and compute budgets.
• Text-to-image services that serve many concurrent requests and benefit from cheaper per-step inference on quantized weights and activations.
• Latent diffusion pipelines for high-resolution synthesis, such as the LSUN-Churches and LSUN-Bedrooms settings tested here.
• Video and audio generation, which the paper cites as diffusion applications but does not evaluate; text-to-image, class-conditional, and unconditional image generation are what the experiments cover.

Industry relevance. The method is post-training, meaning it calibrates quantization parameters using a small set of calibration samples and does not retrain network weights. For teams deploying large pretrained diffusion checkpoints, that is far cheaper than quantization-aware training and does not require access to the original training pipeline. The reported gains are largest exactly at the aggressive bit-widths that matter most for cost reduction.

Future Directions

• Relaxing the multi-step calibration requirement. The limitation section states that AccuQuant requires multiple denoising steps during calibration, and notes it may have limited efficacy in some settings even though it performs strongly in few-step regimes such as 20 steps on ImageNet. The visible text is truncated, so the full scope of this limitation is not reported here.

• Adaptive or automatic group sizing. Group size is treated as a hyperparameter balancing error capture against optimization stability, with performance degrading sharply at sizes of 50 and 100 on CIFAR-10. A principled way to choose group boundaries per model, dataset, and sampler is left open.

• Extending beyond the evaluated image models. Experiments cover DDIM, LDM, and Stable Diffusion v1.4. Whether the error-accumulation analysis and gradient approximation transfer to video and audio diffusion models, or to other samplers, is not reported.

• Combining with step-reduction methods. The paper distinguishes its work from approaches that reduce the number of denoising steps. Whether AccuQuant composes with those sampling accelerations is not evaluated.

Target Audience

Researchers and engineers working on model compression, efficient inference, and generative models — particularly those deploying diffusion models under tight memory or latency budgets. It is also relevant to practitioners of post-training quantization who want to understand why sequential generative models violate assumptions inherited from classification networks. Readers should be comfortable with diffusion sampling equations, quantization terminology (step-sizes, rounding, bit-widths), and gradient-based optimization of calibration objectives.

Authors’ abstract

We present in this paper a novel post-training quantization (PTQ) method, dubbed AccuQuant, for diffusion models. We show analytically and empirically that quantization errors for diffusion models are accumulated over denoising steps in a sampling process. To alleviate the error accumulation problem, AccuQuant minimizes the discrepancies between outputs of a full-precision diffusion model and its quantized version within a couple of denoising steps. That is, it simulates multiple denoising steps of a diffusion sampling process explicitly for quantization, accounting the accumulated errors over multiple denoising steps, which is in contrast to previous approaches to imitating a training process of diffusion models, namely, minimizing the discrepancies independently for each step. We also present an efficient implementation technique for AccuQuant, together with a novel objective, which reduces a memory complexity significantly from $\mathcal{O}(n)$ to $\mathcal{O}(1)$, where $n$ is the number of denoising steps. We demonstrate the efficacy and efficiency of AccuQuant across various tasks and diffusion models on standard benchmarks.