Benchmarking and Enhancing VLM for Compressed Image Understanding

AI summary

Overview

• Research area: Computer vision and multimodal machine learning — specifically vision-language models (VLMs) applied to compressed images, sitting at the intersection of image compression (codecs) and vision-language understanding.
• Technical level: Intermediate. The paper is readable for someone familiar with basic deep learning and image compression concepts; the method section involves knowledge distillation and conditioning embeddings, but the findings are presented through benchmark tables and rate-metric curves.
• Scope (one sentence): The paper introduces a large-scale benchmark for measuring how well VLMs understand compressed images across 11 codecs, 9 VLMs and 7 tasks, decomposes the resulting performance loss into an information gap and a generalization gap, and proposes a lightweight adapter that improves performance by 10%–30%.

What This Paper Is About

Vision-language models are normally trained and evaluated on high-bitrate, clean images, but real deployments often involve images that have been heavily compressed for transmission or storage. The paper asks a question nobody had systematically answered: how much does image compression actually hurt a VLM's ability to understand an image, and can anything be done about it? The authors build the first comprehensive benchmark for this setting (over one million compressed images) and then propose a small, codec-agnostic adapter that recovers much of the lost accuracy.

Key Contributions

1. The first comprehensive benchmark for VLMs on compressed images. It spans 11 widely used codecs across three categories (traditional: JPEG, HM, VTM; learning-based: ELIC, TCM, MLICpp; generative: HiFiC, MS-ILLM, DiffEIC, RDEIC, StableCodec), 9 VLMs from 1B to 32B parameters, 7 evaluation tasks, and more than one million compressed images, each codec applied at four different bitrate levels.

2. A decomposition of the performance gap into two parts. The paper formally separates the drop in benchmark score into an information gap — the part of the loss that is caused by irreversible data loss during compression and cannot be fixed by adapting the model — and a generalization gap — the part caused by the VLM's failure to generalize to compressed inputs, which is recoverable.

3. A universal, lightweight VLM adapter. A unified conditional vision encoder that takes codec-identity and distortion-level information as conditioning, trained by distilling the original uncompressed-image vision encoder into a compressed-image encoder. It improves performance across codecs and bitrates by 10%–30%.

4. Empirical evidence of generalization. The adapter is trained on only three codecs (JPEG, ELIC, MS-ILLM) but is shown to improve results on unseen codecs (HM, MLICpp, DiffEIC) and on a different VLM (InternVL3), and it is compared against image-coding-for-machines (ICM) methods such as TIC and TransTIC.

Main Findings

• VLMs struggle with heavily compressed images. Across all 11 codecs, the rate-metric curves for compressed images fall below the uncompressed baseline. When the bitrate drops below 0.1 bpp, VLMs struggle to maintain accurate semantic understanding and task performance, as compression artifacts distort crucial visual details.

• Stronger VLMs mostly handle compression better. Relative performance rankings on compressed images stay consistent with rankings on uncompressed images. However, Janus-pro shows the smallest performance decrease across all compression conditions on the same task, indicating the best resistance to compression — and this resilience is independent of a model's absolute performance.

• Generative codecs are better for semantic preservation at low bitrates. Diffusion-based codecs such as RDEIC and StableCodec reconstruct semantically consistent images at low bitrates and sit in the upper-left corner of the rate-metric plots, making them more suitable for VLM-facing tasks. The same generative codecs perform poorly on fine-grained tasks such as OCRBench, which are known to yield inferior results on text.

• Scaling laws do not hold for distortion robustness. Testing InternVL3 at 1B, 2B and 8B across three codec types, the authors find that increasing model size does not consistently reduce compression-induced degradation, breaking the expected scaling law for generalization to unseen inputs.

• Human-vision quality metrics only partially predict VLM performance. Comparing task scores against PSNR, LPIPS, DISTS and FID, high pixel-level scores do not always translate into strong VLM task performance. For the fine-grained OCRBench task, PSNR correlates most strongly; for coarse-grained tasks (MMBench, MME, SEEDBench, GQA), DISTS or FID has the largest absolute correlation coefficient.

• The two gaps can be measured empirically. On SEEDBench with QwenVL2.5, JPEG produced a performance gap of 13.25, of which 9.5 was information gap and 3.75 generalization gap; ELIC produced 8.53 (4.33 information, 4.2 generalization); ILLM produced 17.68 (15.26 information, 2.42 generalization). On POPE, JPEG's gap of 36.29 split into 6.81 information and 29.48 generalization, ELIC's 9.8 into 3.9 and 5.9, and ILLM's 24.81 into 12.19 and 12.62. The paper notes these values are an empirical lower bound on the generalization gap under a given optimization setup, not a fixed property.

• The adapter delivers large, consistent gains. Against the original compression results with QwenVL2.5-3B, the adapter yields BD-Metric improvements of POPE 12.62, SEEDBench 12.88, GQA 11.63, MMBench 14.91, OCRBench 52.51 and MME 285.4 for JPEG; ELIC and ILLM show over 10% improvement on the POPE metric.

• Gains transfer to unseen codecs and models. Using HM, MLICpp and DiffEIC as stand-ins for JPEG, ELIC and MS-ILLM, the adapter improves all unseen codecs. MLICpp achieves larger gains than ELIC itself on SEEDBench. DiffEIC improves on POPE, OCRBench and MMB but drops very slightly on SEEDBench, MME and GQA — the authors attribute this to MS-ILLM being GAN-based while DiffEIC is diffusion-based, creating a larger structural gap between the codec families. InternVL3-1B also improves across all six benchmarks.

• Adding adapter correction on top of ICM methods helps further. On POPE, BD-Metric went from 0.00 for TIC and 0.18 for TransTIC to 2.43 for TIC-Adapter and 3.02 for TransTIC-Adapter, suggesting that addressing the generalization gap and the information gap together gives the best result.

• Conditioning information matters. Ablations show that even with no conditional metadata (a "blind" setting) the adapter still improves results, but the no-condition variant can degrade as bitrate increases. Adding distortion-level conditioning stabilizes gains across compression levels, and adding codec-identity conditioning yields the best overall numbers (e.g., JPEG POPE: 11.86 with no metadata, 12.22 with distortion level only, 12.43 with codec only, 12.62 with both).

Methodology in Plain English

The authors first assemble a test suite. They take existing VLM evaluation benchmarks (POPE with 5127 images, COCO-Caption with 5000, OCRBench with 1000, GQA with 398, MME with 1187, MMBench with 4329, SEEDBench with 14232) and re-compress the underlying images with 11 codecs at four bitrate levels each, deliberately targeting bitrates below 0.3 bpp. They run 9 open-source VLMs over all of this using the VLMEvalKit evaluation toolkit and treat each model's uncompressed-image score as its ceiling, so all results are reported as degradation rather than absolute accuracy.

To explain why performance drops, they split the loss into two pieces. Imagine the perfect model for a given compressed image. The gap between the uncompressed score and what even a perfectly adapted model achieves on the compressed image is the information gap — the data is simply gone. The remaining gap between that ideal adapted model and the actual VLM is the generalization gap. Operationally, they estimate these by fine-tuning the VLM separately on JPEG, ELIC and ILLM compressed data and seeing how much of the loss can be recovered.

For the fix, they build an adapter rather than fine-tuning the whole model. Because VLM vision encoders use Vision Transformers, the adapter injects two extra pieces of information into the encoder: which codec was used (one-hot encoded) and what compression level it used. These are mapped through an embedding layer into a codec-condition embedding (a 12-dimensional embedding for 3 codecs × 4 bitrates), which is added to the rotary positional embedding of the image patches. The resulting conditional vision encoder is trained by distillation: it must produce features from the compressed image that match the features the original encoder produces from the uncompressed image, minimizing the mean squared error between the two. Training used 4 NVIDIA A100 GPUs, random 336×336 crops, batch size 24, 100k iterations, and an initial learning rate of 1×10⁻⁴, on over 11w COCO images compressed with the three codecs.

Why This Matters

Impact on research. The paper reframes the problem of "VLMs on compressed images" as two separable failure modes, giving the community a way to distinguish losses that require better compression algorithms from losses that require better models. It also shows that scaling model size alone does not solve distortion robustness, and that existing human-perception quality metrics are imperfect proxies for machine understanding — both of which challenge assumptions in the image and video coding for machines literature.

Real-world applications:

• Transmitting images over bandwidth-constrained networks (mobile, satellite, remote sensing) where heavy compression is unavoidable but the receiver is a VLM rather than a human.
• Storage and retrieval pipelines where large image archives are stored at low bitrates.
• Cost reduction in cloud multimodal services, since the adapter is lightweight and does not require re-training the full VLM.
• Any deployment where the compression metadata may be unknown, since the ablation shows the adapter still helps in a blind, no-metadata setting.

Industry relevance. The adapter's ability to improve results on codecs it was never trained on, and across VLM backbones (Qwen2.5-VL and InternVL3), makes it a practical drop-in component rather than a codec-specific engineering fix. The paper also situates itself relative to MPEG's Video Coding for Machines (VCM) and Feature Coding for Machines (FCM) standardization efforts.

Future Directions

• Extending the benchmark to proprietary VLMs. The authors explicitly state that their experiments do not include the latest proprietary models due to API cost constraints and that they plan to extend the study.
• Bridging the information gap. Since information loss during compression is irreducible by model adaptation, closing it requires better codecs — the paper points toward compression algorithms optimized jointly with machine perception rather than human perception.
• Handling larger codec-family gaps. The DiffEIC result (gains on POPE, OCRBench and MMB but slight drops on SEEDBench, MME and GQA) suggests that conditioning designed around one codec family does not fully transfer to structurally different families, leaving room for broader codec coverage in adapter training.
• Extending beyond still images. The paper's framing toward semantic compression and video/feature coding standards implies open questions about temporal compression and feature-domain coding for VLMs.

Target Audience

This paper is most useful to researchers and engineers working on vision-language models, image/video compression, or coding for machines — particularly those building systems where images are transmitted or stored at low bitrates before being interpreted by a model. It is also relevant to practitioners evaluating VLM robustness, and to standards-oriented readers interested in how machine perception should influence codec design. Readers should have basic familiarity with VLMs, image codecs and evaluation benchmarks, but the high-level findings are accessible without a deep compression background.

Authors’ abstract

With the rapid development of Vision-Language Models (VLMs) and the growing demand for their applications, efficient compression of the image inputs has become increasingly important. Existing VLMs predominantly digest and understand high-bitrate compressed images, while their ability to interpret low-bitrate compressed images has yet to be explored by far. In this paper, we introduce the first comprehensive benchmark to evaluate the ability of VLM against compressed images, varying existing widely used image codecs and diverse set of tasks, encompassing over one million compressed images in our benchmark. Next, we analyse the source of performance gap, by categorising the gap from a) the information loss during compression and b) generalisation failure of VLM. We visualize these gaps with concrete examples and identify that for compressed images, only the generalization gap can be mitigated. Finally, we propose a universal VLM adaptor to enhance model performance on images compressed by existing codecs. Consequently, we demonstrate that a single adaptor can improve VLM performance across images with varying codecs and bitrates by 10%-30%. We believe that our benchmark and enhancement method provide valuable insights and contribute toward bridging the gap between VLMs and compressed images. The source code is available at https://github.com/bblgbr/CompressVLMBench.