Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior

AI summary

Overview

• Research area: Learned image compression, with a focus on image compression for machines (ICM) and generative (diffusion-based) image coding.
• Technical level: Advanced. The paper assumes familiarity with learned codecs, rate-distortion optimization, latent diffusion models, ControlNet-style conditioning, and downstream vision benchmarks.
• One-sentence scope: The paper proposes Diff-ICMH, a generative image codec built on a pre-trained Stable Diffusion model that aims to serve many machine vision tasks and human viewers at once, using a Semantic Consistency loss and a Tag Guidance Module.

What This Paper Is About

Most image compression methods are optimized for either human viewing or machine analysis, not both. The authors argue that the two goals actually share a common foundation: preserving accurate semantic content matters for intelligent tasks and human understanding alike, while realistic image distributions also help machine feature extraction. Diff-ICMH is a single generative codec that tries to deliver both simultaneously, without any task-specific adaptation.

Key Contributions

1. A unifying perspective on codec design. The paper identifies semantic fidelity and perceptual realism as the two determinants that jointly govern performance for both machine tasks and human perception, framing the problem as one of preserving semantics while reconstructing realistic textures.
2. The Diff-ICMH framework. A generative compression method that decodes the bitstream into the VAE latent space of a pre-trained Stable Diffusion model, rather than into pixel space, and conditions reconstruction on the compressed latent through a ControlNet-like control module.
3. Semantic Consistency loss (SC loss). A training objective that projects ground-truth and decoded latents through a pre-trained diffusion model and maximizes cosine similarity between the resulting semantic representations, using the diffusion model's forward pass as a bridge into a shared semantic space.
4. Tag Guidance Module (TGM). A low-overhead mechanism that extracts word-level image tags, losslessly encodes their indices, and feeds them as text conditioning to both the diffusion model and the control module to activate generative priors.

Main Findings

• State-of-the-art or competitive across 10 downstream tasks. Diff-ICMH was evaluated on a diverse set of intelligent tasks spanning traditional perception (detection, instance segmentation, pose estimation, panoptic segmentation on COCO 2017), multimodal retrieval (Flickr30K), and MLLM-based understanding (referring expression on RefGTA, open-set segmentation on ADE20K), without task-specific adaptation.
• Strong advantage at very low bitrates. The paper reports significant advantages on Flickr30K cross-modal retrieval at 0.01–0.05 bpp and on ADE20K open-vocabulary segmentation at 0.02–0.1 bpp, attributed to the SC loss and TGM.
• Trade-offs in specific scenarios. Performance is described as comparable to methods such as DiffEIC on COCO pose estimation and RefGTA referring expression, which the authors attribute to inherent VAE latent space limitations for fine details and to domain shift from synthetic data, respectively.
• Signal fidelity metrics trail fidelity-optimized codecs. Diff-ICMH's PSNR and MS-SSIM scores fall behind traditional fidelity-oriented codecs, described as a common characteristic of perceptual-priority codecs. It is comparable to DiffEIC and outperforms PerCo on these fidelity metrics.
• State-of-the-art perceptual scores. Diff-ICMH achieves state-of-the-art FID and DISTS among all compared methods, and surpasses fidelity-oriented codecs (BPG, VTM-18.2) and other perception-oriented methods (HiFiC, PerCo) across LPIPS, FID, and DISTS, with particular strength at extremely low bitrates.
• Feature divergence grows with network depth for fidelity codecs. Using 1 minus cosine similarity on ResNet50 features, fidelity-oriented codecs (VTM-18.2, ELIC) showed smaller differences at the stem layer but significantly larger divergence at layer2 and layer4, while generative codes diverged less at deeper layers. In the appendix analysis, fidelity-oriented methods' differences reached a peak of approximately 0.5 at the deepest layer.
• Ablation: SC loss and TGM both help, and combine best. Their combination yielded the best performance, with around a 4 mAP gain over the baseline at approximately 0.025 Bpp on COCO 2017 object detection.
• SC loss hyperparameters matter. Weight λ_sem = 2.0 gave the optimal balance; applying SC loss to the middle block of the U-Net was optimal; and noise-free inputs (t = 0) gave the best rate-distortion performance, though all tested noise levels still improved over the baseline.
• Tag overhead is small. Using RAM++ (from Recognize Anything) with a maximum default vocabulary of 4585 tags, fixed-length coding of 13 bits per tag ID (accommodating a maximum of 8192 tags), and an average of 8.7 predicted tags per image measured over 500 randomly sampled COCO images, the average tag overhead is 13 × 8.7 = 113.1 bits per image. The abstract describes this as approximately 100 bits per image.

Methodology in Plain English

The input image is compressed into the latent space of a pre-trained Stable Diffusion model rather than into pixels. This latent space already encodes perceptually meaningful, compact information (Stable Diffusion uses 8×8 spatial downsampling), so optimizing fidelity there pushes the bitstream toward semantically coherent content instead of irrelevant pixel detail. Alongside the latent, a tag extractor (RAM++) produces word-level tags that are mapped to dictionary indices and losslessly coded into the same bitstream.

At the decoder, the compressed latent is fed into a control module adapted from ControlNet, while noisy latents enter the pre-trained diffusion model. The diffusion model predicts the noise, conditioned on the reconstructed latent and the tags, and after the standard reverse diffusion process the denoised latent is passed through the VAE decoder to produce the final image.

Training combines four terms: a rate loss on the quantized latents and hyperprior latents, an MSE reconstruction loss in the VAE latent space, the standard diffusion noise-prediction loss, and the SC loss, which compares semantic features of ground-truth and decoded latents inside the frozen diffusion model and maximizes their cosine similarity. The Stable Diffusion model and tag extractor stay frozen; only the encoder-decoder and control module are trained.

Setup details: training on LSDIR with 512×512 random crops; Stable Diffusion 2.1 as the pre-trained model; λ_dist = 1, λ_diff = 1, λ_sem = 2, with λ_rate in {2, 4, 8, 16, 32} to get different bitrates; Adam with β1 = 0.9 and β2 = 0.999; batch size 16; two training stages of 200K iterations each (learning rate 1e-4, then 5e-5). Tags are dropped with probability 0.1 during training; inference uses Classifier-Free Guidance with scale 5.0 and DDIM sampling with 50 steps, starting from pure Gaussian noise. Ablations used batch size 8, 80,000 iterations, and learning rate 1e-5. All training and inference ran on 4 NVIDIA A100 Tensor Core GPUs.

Why This Matters

Impact on research. The paper reframes machine-oriented and human-oriented compression as compatible rather than opposed, and provides the first (by the authors' account) broad empirical study of a single codec across 10 downstream tasks and multiple model families and backbones. It also supplies a concrete diagnostic tool — feature divergence across ResNet50 depths — for reasoning about why generative codecs help downstream models.

Real-world applications:

• Cloud-edge and bandwidth-constrained image delivery, where one bitstream can feed both human viewers and automated analysis pipelines.
• Large-scale image storage and archiving for datasets that must serve retrieval, detection, and captioning/understanding models at once.
• Content delivery platforms that need perceptual quality at extremely low bitrates (the paper highlights strong FID and DISTS at low BPP).
• Multimodal and MLLM-driven systems that consume images and must retain semantic content for referring expression and open-set segmentation.

Industry relevance. The approach avoids task-specific re-training or bitstream adaptation, which simplifies deployment: a single codec and single bitstream support many models. The tag overhead is small, and the tag dictionary approach (13 bits per tag ID) is straightforward to implement. The main practical barrier is decoding cost, since each denoising step requires a full network forward pass.

Future Directions

• Reducing decoding cost. The authors name computational complexity, driven by the iterative denoising process, as the primary limitation, and propose efficient sampling methods and distillation techniques to cut sampling steps.
• Closing the fine-detail gap. Performance is comparable to DiffEIC in cases such as COCO pose estimation, which the authors link to VAE latent space limits for fine details — an open question for latent-space codec design.
• Domain shift from synthetic data. The paper notes domain shift from synthetic data as a likely cause of relatively weaker results on the RefGTA referring expression task.
• Extension beyond the current task set. The evaluation covers 10 tasks; broader generalization across additional task families and models is left to future work. The paper does not report a study of how far the approach scales to other pre-trained diffusion backbones.

Target Audience

Researchers and engineers working on learned image compression, generative codecs, rate-distortion-perception trade-offs, and image compression for machines. It is also relevant to practitioners deploying shared codecs for human viewing and automated analysis, and to readers interested in how pre-trained diffusion priors can be reused as semantic feature extractors. The paper's density of benchmarks, model names, and training details makes it most accessible to readers already familiar with neural compression and diffusion models.

Authors’ abstract

Image compression methods are usually optimized isolatedly for human perception or machine analysis tasks. We reveal fundamental commonalities between these objectives: preserving accurate semantic information is paramount, as it directly dictates the integrity of critical information for intelligent tasks and aids human understanding. Concurrently, enhanced perceptual quality not only improves visual appeal but also, by ensuring realistic image distributions, benefits semantic feature extraction for machine tasks. Based on this insight, we propose Diff-ICMH, a generative image compression framework aiming for harmonizing machine and human vision in image compression. It ensures perceptual realism by leveraging generative priors and simultaneously guarantees semantic fidelity through the incorporation of Semantic Consistency loss (SC loss) during training. Additionally, we introduce the Tag Guidance Module (TGM) that leverages highly semantic image-level tags to stimulate the pre-trained diffusion model's generative capabilities, requiring minimal additional bit rates. Consequently, Diff-ICMH supports multiple intelligent tasks through a single codec and bitstream without any task-specific adaptation, while preserving high-quality visual experience for human perception. Extensive experimental results demonstrate Diff-ICMH's superiority and generalizability across diverse tasks, while maintaining visual appeal for human perception. Code is available at: https://github.com/RuoyuFeng/Diff-ICMH.