Generative Latent Coding for Ultra-Low Bitrate Image Compression

AI summary

Overview

• Research area: Learned lossy image compression, specifically generative/perceptual compression in the latent space of a vector-quantized autoencoder.
• Technical level: Advanced. The paper assumes familiarity with transform coding, entropy models, VQ-VAE latent spaces, adversarial and perceptual losses, and rate-distortion optimization.
• Scope: The paper proposes Generative Latent Coding (GLC), a codec that performs transform coding inside the latent space of a generative VQ-VAE rather than in pixel space, targeting high-realism and high-fidelity reconstruction at ultra-low bitrates.

What This Paper Is About

Most image codecs, including learned ones, compress images by transforming and quantizing pixels. The authors observe that pixel-space distortion does not align well with human perception, so pixel-space generative codecs such as MS-ILLM struggle at very low bitrates. GLC instead encodes an image into a perceptual, semantic latent space produced by a generative VQ-VAE, and then compresses that latent representation, aiming to keep both visual realism and fidelity while using fewer than 0.04 bpp on natural images and fewer than 0.01 bpp on facial images.

Key Contributions

1. A generative latent coding (GLC) scheme that moves transform coding out of pixel space and into the latent space of a VQ-VAE, which the authors describe as sparser, more semantic, and better aligned with human perception.
2. A categorical hyper module that models the hyper-information z with a discrete hyper codebook instead of the commonly used factorized prior, cutting the bit cost of that side information.
3. A code-prediction-based supervision used only during training (not at inference) to push the compressed latent toward greater semantic consistency.
4. Demonstrated performance and extensions: 45% bit reduction on CLIC2020 at the same FID as the previous state of the art, plus image restoration and style transfer applications built on the same pipeline.

Main Findings

• Ultra-low bitrates achieved: GLC maintains high visual quality at less than 0.04 bpp on natural images and less than 0.01 bpp on facial images.
• 45% bit savings on CLIC2020: GLC reaches the same FID as MS-ILLM while using 45% fewer bits; it is reported as superior on DISTS, FID and KID, and comparable on LPIPS to high-fidelity generative codecs such as HiFiC and MS-ILLM.
• Facial compression dominance: On CelebAHQ, GLC outperforms all compared methods across all metrics by a large margin; the authors note other methods fail to produce satisfactory reconstructions at around 0.01 bpp or lower.
• Transform coding beats indices-map coding: In the ablation on the FID–BPP curve on CLIC2020, indices-map coding causes a 66.2% performance loss relative to transform coding.
• Categorical prior helps: Switching from a factorized prior to the categorical hyper module yields a 17.7% improvement (with the categorical prior set as the 0% reference).
• Code prediction as supervision, not as a network module: Placing the code prediction module inside the network causes a 60.7% performance drop, whereas using it as auxiliary training supervision gives a 13.1% improvement.
• Patch attention matters at high resolution: On CLIC2020, global attention costs 20.8% BD-Rate relative to the 32×32 patch default; 64×64 costs 8.4% and 16×16 costs 1.8%.
• Restoration beats cascading: On CLIC2020, the GLC restoration application uses 0.0299 bpp (FID 8.62, DISTS 0.1081, 109M parameters) versus Restormer + GLC Codec at 0.0314 bpp (FID 10.79, DISTS 0.1174, 25M + 109M parameters).
• Lower latency than MS-ILLM: On CelebAHQ at 512×512, GLC reports 19.2 ms encoding and 26.6 ms decoding with 92M parameters versus MS-ILLM's 31.4 ms / 39.7 ms with 181M parameters; BD-DISTS is 0 lower for GLC versus 0.070 for MS-ILLM. On Kodak at 512×768, GLC reports 37.1 ms / 58.6 ms with 105M parameters (BD-DISTS 0) versus MS-ILLM's 41.8 ms / 53.5 ms with 181M parameters (BD-DISTS 0.047) and Text+Sketch's 2.0×10⁴ ms / 1.9×10⁴ ms with 409M parameters (BD-DISTS 0.140).
• Pixel-level metrics are unreliable at ultra-low bitrate: The authors argue PSNR, MS-SSIM and LPIPS prioritize pixel accuracy over semantics and texture realism, and prioritize DISTS, FID and KID instead; for Text+Sketch on Kodak they report PSNR = 11.97 dB and MS-SSIM = 0.3127 at BPP = 0.0289.
• Generalization limitation: GLC does not reliably reconstruct screen contents, such as straight grid lines in backgrounds, which the authors attribute to the generative latent autoencoder and training strategy.

Methodology in Plain English

GLC has two cooperating parts: a generative latent autoencoder and a compressor that operates on its latents.

1. Encode to a perceptual latent. An image is mapped by an encoder into a latent l in the codebook space of a generative VQ-VAE. The authors use a VQGAN-style structure for natural images (latent resolution 1/16 of the original, codebook size 16384) and a modified CodeFormer structure for faces (1/32, codebook size 1024). The discrete codebook acts as a variational bottleneck, producing a sparse, distortion-robust latent.
2. Transform-code the latent instead of index maps. Rather than transmitting the VQ index map, an analysis transform turns l into y, which is scalar-quantized to ŷ and entropy coded; a synthesis transform turns ŷ back into ĥ. The probability model for ŷ combines a categorical hyper module and a quadtree-partition-based spatial context module. A learned scaler q makes a single model support multiple rates.
3. Categorical hyper module. Instead of the usual factorized hyper prior, z is vector-quantized through a hyper codebook C_h, which the authors say makes z capture high-level semantics rather than low-level color/texture, so it costs far fewer bits. During inference the hyper indices are coded at a fixed length of log₂ M_h bits each.
4. Code-prediction supervision. An auxiliary code predictor takes the reconstructed latent and must predict the correct VQ indices, trained with cross-entropy plus L2 against l. This predictor is used only during training and is discarded at inference, so the decoder is not bottlenecked by the codebook.
5. Three-stage progressive training. Stage I trains the generative VQ-VAE with reconstruction, LPIPS perceptual (VGG features), adaptive Patch-GAN adversarial (λ_adv = 0.8) and codebook losses (β = 0.25). Stage II freezes the autoencoder and trains the transform coding module with a rate term plus the code-prediction loss (α = 0.5). Stage III fine-tunes everything jointly using a pixel-space version of the code-prediction loss (λ_code = 0.05) combined with reconstruction, perceptual and adversarial terms.
6. Latent patch attention. For high-resolution images, global attention in the latent is replaced by attention within 32×32 latent patches. Training uses ImageNet (stage I) and OpenImage (stages II–III) with 256×256 crops for natural images, and FFHQ at 512×512 for faces, optimized with AdamW at batch size 8.

Why This Matters

• Research impact: The paper argues that the choice of where transform coding happens (latent space versus pixel space) is a first-order design decision for perceptual compression. It reframes latent-space modeling, previously dominated by image generation and restoration work such as VQGAN and CodeFormer, as a compression paradigm, and reports a 45% bit reduction over MS-ILLM at equal FID.
• Real-world applications:
  • Ultra-low-bitrate transmission of natural images where bandwidth is the bottleneck.
  • Facial image compression for conferencing and communication, where the paper reports working below 0.01 bpp.
  • Joint restoration and compression, where a distorted photo can be compressed and then decoded as a clean image without extra model parameters.
  • Style transfer built directly into decoding, where the same codes produce an image in a different artistic style.
• Industry relevance: Lower bitrates at equal perceptual quality directly reduce storage and bandwidth costs; the reported lower latency and smaller parameter counts than MS-ILLM (92M vs 181M on faces; 105M vs 181M on Kodak) matter for deployment; rate-variable coding from a single model is a practical requirement for real codecs. The authors also note the pipeline is released as open code.

Future Directions

1. Improving generalization. The authors explicitly identify screen content as a failure case and propose improving the generative latent autoencoder or the training strategy to fix it.
2. Extending beyond natural and facial images. GLC is trained on specified datasets; whether the approach holds for other domains and content types is left open.
3. Better evaluation of perceptual quality. The paper argues PSNR, MS-SSIM and LPIPS are poorly suited to ultra-low bitrate evaluation and leans on DISTS, FID and KID; finding more reliable perceptual metrics remains an open problem.
4. Deeper integration with other vision tasks. The authors state they hope the versatility of the generative latent space "will foster connections between image compression and other vision tasks," having demonstrated only restoration and style transfer so far.

Target Audience

Researchers and engineers working on learned image compression, perceptual/generative compression, or neural codecs; practitioners who need ultra-low-bitrate compression for faces or natural images; and readers interested in how VQ-VAE latent spaces, entropy modeling and code-prediction transformers can be combined into a single system. Readers without a background in entropy coding, rate-distortion trade-offs and VQ-VAE latent modeling will find the method sections difficult.

Authors’ abstract

Most existing image compression approaches perform transform coding in the pixel space to reduce its spatial redundancy. However, they encounter difficulties in achieving both high-realism and high-fidelity at low bitrate, as the pixel-space distortion may not align with human perception. To address this issue, we introduce a Generative Latent Coding (GLC) architecture, which performs transform coding in the latent space of a generative vector-quantized variational auto-encoder (VQ-VAE), instead of in the pixel space. The generative latent space is characterized by greater sparsity, richer semantic and better alignment with human perception, rendering it advantageous for achieving high-realism and high-fidelity compression. Additionally, we introduce a categorical hyper module to reduce the bit cost of hyper-information, and a code-prediction-based supervision to enhance the semantic consistency. Experiments demonstrate that our GLC maintains high visual quality with less than 0.04 bpp on natural images and less than 0.01 bpp on facial images. On the CLIC2020 test set, we achieve the same FID as MS-ILLM with 45% fewer bits. Furthermore, the powerful generative latent space enables various applications built on our GLC pipeline, such as image restoration and style transfer. The code is available at https://github.com/jzyustc/GLC.