Progressive Learned Image Compression for Machine Perception

AI summary

Overview

Research area: Learned image compression, specifically progressive (fine granular scalability) coding adapted for machine perception rather than human viewing.

Technical level: Advanced. The paper assumes familiarity with learned image compression architectures (autoencoders, hyperpriors, entropy models), rate-distortion optimization, and downstream vision classifiers.

Scope: The paper proposes PICM-Net, a progressive learned image codec built on trit-plane coding that is adapted for downstream classification tasks via lightweight adapters plus a controller that decides per image when to stop decoding.

What This Paper Is About

Learned image codecs have largely been optimized for human visual quality, and a newer line of work re-optimizes them for machine vision tasks such as classification. Separately, progressive image coding (fine granular scalability) lets a single bitstream be decoded at multiple quality levels, which is useful when bandwidth fluctuates. The paper's goal is to combine both ideas: build the first progressive learned image codec whose coarse-to-fine bitstream is organized around what a downstream machine classifier needs, and add a mechanism that stops decoding as soon as the current reconstruction is good enough for a reliable prediction.

Key Contributions

1. PICM-Net, described by the authors as the first progressive image compression codec for machine perception, built on trit-plane coding with machine-aware symbol prioritization.

2. Machine-oriented progressive adaptation combining spatial-frequency modulation adapters (SFMA) and a hyper-synthesis low-rank adapter (HSLoRA), trained with a progressive decoding-aware objective that supervises the downstream task at randomly sampled trit-plane prefixes as well as at full reconstruction.

3. An adaptive decoding controller, a lightweight logistic regression model that predicts whether the current partial reconstruction is suitable for reliable inference and requests additional bits only when predicted suitability falls below a user-specified confidence threshold.

4. Extensive experiments comparing PICM-Net against human-oriented progressive codecs and machine-oriented non-progressive codecs, plus ablations on components, prioritization strategy, and controller behavior.

Main Findings

• Outperforms progressive baselines: Against human-oriented progressive codecs (PLONQ, DPICT, CTC, ProgDTD, Efficient-PIC, DeepHQ), PICM-Net consistently achieves superior rate-accuracy across all bitrates on both ResNet-50 and ConvNeXt. The authors attribute this to machine-oriented adaptation placing task-discriminative information in the early bits.

• Competitive with machine-oriented codecs: Compared to non-progressive machine-oriented codecs (TransTIC, AdaptICMH, SA-ICM, MPA), PICM-Net achieves competitive rate-accuracy on ResNet-50 while uniquely supporting fine-grained progressive decoding from a single bitstream. A moderate performance gap is observed on ConvNeXt, which the authors attribute to the adaptation modules being trained with ResNet-50 as the downstream classifier.

• Machine adaptation redistributes bits: Under the same 0.15 bpp budget, the human-oriented codec spreads bits broadly across the spatial domain, while the machine-oriented codec concentrates them on a sparser set of regions. The difference map shows extra bits going to areas highlighted by Grad-CAM on a pretrained ResNet-50, and fewer bits spent on task-irrelevant background.

• Controller gains grow with bitrate: With and without the adaptive decoding controller, PICM-Net shows nearly identical rate-accuracy at low bitrates; the controller yields substantially better performance at higher bitrates, because at low bitrates most images have not yet reached a reliable prediction.

• Per-image bit reallocation is real: At τ = 0.70 over 50K ImageNet-1K validation images, the controller saves bits on a large subset of images while spending more on others, producing a skewed per-image rate-difference distribution centered below zero. Partitioning into three equal groups by rate difference, the rate-saved group reaches 87.4% full-rate accuracy while the rate-spent group reaches only 57.2%.

• Easy images stop early, hard images continue: The rate-saved group stops after roughly the first 4% of the progressive trit-plane trajectory on average, whereas the rate-spent group continues to roughly 67% on average. At the sample level, an easy image is correctly classified at 0.053 bpp, while a hard image's suitability stays below threshold until near the final level at 0.435 bpp.

• Accuracy saturates while PSNR does not: For rate-saved images, accuracy saturates early and further decoding has diminishing returns, whereas PSNR keeps rising monotonically with bitrate across all groups. This confirms that bits removed by the controller improve reconstruction fidelity but do not change the downstream decision.

• Quantitative controller gains: In the practical operating region (accuracy ≥ 70%), the controller achieves a BD-rate of −21.6% relative to the decoder without the controller. At desired confidences τ of 70%, 73%, and 75%, average bitrate is reduced by 18.7%, 23.9%, and 26.4% respectively. Outside this region the benefit is limited: at very low bitrates, the fixed threshold may trigger premature stopping, causing an accuracy decrease of up to 1.91 %p relative to static decoding.

• Ablation on components: Adding SFMA alone yields the largest gain, −66.9% BD-rate in the operating region. HSLoRA further reduces BD-rate by 1.9% without changing accuracy at the full decoding level. Removing progressive decoding-aware training causes the rate-accuracy curve to collapse across all bitrates, with average accuracy over progressive bitrates falling from 66.5% to 54.1%.

• Prioritization trade-off: Expected variance-based symbol ordering gives only a marginal BD-rate gain of −0.4%, but is 20.7× slower for symbol-order prioritization and 2.28× slower overall (21.0 ms vs. 9.2 ms per image). Sigma-based ordering is therefore adopted as the default.

Methodology in Plain English

The researchers start from an existing human-oriented transformer-based codec (a pretrained TransTIC base codec with M = 192 latent channels and N = 128 hyperprior channels) and keep its weights completely frozen. On top of this backbone they add progressive coding and machine adaptation.

For progressive coding, each quantized latent coefficient is broken into ternary digits, called trits, and transmitted plane by plane. How many planes a coefficient gets depends on its predicted scale parameter: coefficients with wider predicted distributions are allotted more planes. Coarse planes arrive first, so a decoder can reconstruct an approximate image after only part of the bitstream. Within each plane, symbols are ordered by their scale parameter.

For machine adaptation, two small trainable modules are inserted. SFMA modules sit inside the encoder and decoder and modulate features in both spatial and frequency domains, so the codec can suppress task-irrelevant detail and emphasize semantic content. HSLoRA is a 1×1 convolutional low-rank adapter (rank 16) applied only to the hyper-synthesis network, which outputs the mean and scale of the Gaussian used for coding the latent; the hyper-analysis network is left untouched because its output is entropy-coded with a fixed factorized density model that cannot be refit under the frozen-backbone design. Three SFMA modules go in each of the encoder and decoder, and one HSLoRA module in the hyperprior decoder, for 300,080 trainable parameters — 3.84% of the full network.

Training uses a rate-distortion objective that combines bitrate with cross-entropy from a frozen ResNet-50 classifier and an MSE regularizer (λ_task = 0.8, λ_mse = 0.0025). Crucially, the training samples a random trit-plane prefix at each iteration, so the objective covers the whole progressive decoding trajectory rather than only the full reconstruction. The adaptation modules are trained on 80,000 images sampled from the ImageNet-1K training set for 60 epochs using AdamW with learning rate 10⁻⁴, weight decay 10⁻⁴, and cosine annealing to η_min = 10⁻⁶.

Finally, the adaptive decoding controller is a logistic regression model fitted on a 12-dimensional feature vector derived from the classifier's output at each decoding level — softmax-based statistics, logit-based statistics, and energy-based statistics — with a binary label indicating whether the prediction was correct. It is trained on 5,000 images from the ImageNet-1K training set compressed at every progressive level; a separate controller is trained for ConvNeXt. At inference, the decoder walks up the levels and stops as soon as the predicted suitability crosses the threshold τ.

Evaluation follows the settings in AdaptICMH: ImageNet-1K validation set, images resized to 256×256 for compression and center cropped to 224×224 for evaluation, top-1 accuracy, using ResNet-50 and ConvNeXt from the timm library, with all experiments on NVIDIA A100-SXM4-40GB GPUs. All codecs are implemented with the CompressAI library.

Why This Matters

Impact on research: The paper opens a gap the authors identify as previously unexplored — progressive, machine-oriented compression. It shows that task-driven adaptation implicitly reorders which parts of a progressive bitstream matter, and that the human assumption "more bits are always better" does not hold for machines. It also provides a concrete recipe for adapting a frozen human codec with a very small number of trainable parameters.

Real-world applications:

• Autonomous driving, where bandwidth fluctuates and a perception model may need only enough bits to classify or interpret a scene.
• Surveillance systems, where many cameras stream to a downstream analyzer and per-image early stopping reduces aggregate transmission cost.
• Remote sensing, where imagery is downlinked under constrained bandwidth and consumed automatically.
• Bandwidth-constrained edge deployments where a single bitstream must serve receivers at different quality or confidence needs.

Industry relevance: The controller's ability to trade bitrate against a user-specified confidence threshold (18.7%, 23.9%, and 26.4% average bitrate reduction at τ = 70%, 73%, and 75%) speaks directly to operating points that streaming and perception pipelines need to tune. Because only 3.84% of the network's parameters are trained, the approach is also cheap to build on top of an existing deployed codec.

Future Directions

• Classifier-agnostic adaptation: The adaptation modules are trained with a specific downstream classifier (ResNet-50), and the ConvNeXt results show a moderate gap; making adaptation generalize to unseen classifiers is left as future work.

• Beyond classification: The authors propose extending the framework to dense machine tasks such as object detection and segmentation.

• Low-bitrate controller behavior: The fixed threshold can trigger premature stopping at very low bitrates, causing accuracy decreases of up to 1.91 %p relative to static decoding — a limitation that motivates better or adaptive thresholding.

• Prioritization efficiency: Expected variance-based ordering offers only a −0.4% BD-rate gain at a 20.7× cost in prioritization time and 2.28× in total decode time, leaving open the question of whether a cheaper ordering can capture that gain.

Target Audience

Researchers and engineers working on learned image compression, scalable or progressive coding, and machine-oriented or task-aware compression. It will also interest practitioners building bandwidth-constrained perception pipelines (autonomous driving, surveillance, remote sensing) and anyone studying how compression interacts with downstream model confidence. Readers need a background in deep learning and image compression codecs; the paper is not introductory.

Authors’ abstract

Recent advances in learned image codecs have extended from human perception toward machine perception However, progressive image compression with fine granular scalability (FGS)-which enables decoding a single bitstream at multiple quality levels-remains unexplored for machine-oriented codecs. In this work, we propose PICM-Net, a progressive learned image compression codec for machine perception built on trit-plane coding. Starting from a human-oriented codec, we adapt it for machine perception via a spatial-frequency modulation adapter (SFMA), a hyper-synthesis low-rank adapter (HSLoRA), and progressive decoding-aware training, and analyze how task-driven adaptation alters symbol prioritization for progressive transmission. To further support real-world deployment, we introduce an adaptive decoding controller that dynamically determines the necessary decoding level at inference time, requesting additional bits only when the current suitability is insufficient for the desired confidence level. Extensive experiments demonstrate that PICM-Net achieves efficient and adaptive progressive transmission while maintaining strong downstream classification performance. Our code is available at https://github.com/kjungwoo03/PICM-Net.