CARES: Context-Aware Resolution Selector for VLMs

AI summary

Overview

Research area: Computer vision and multimodal machine learning — specifically inference efficiency for vision-language models (VLMs).

Technical level: Intermediate. The core idea is intuitive (show the model only as many pixels as the question needs), but the paper assumes familiarity with tokenization, vision encoders, prefill stages, and standard VLM benchmarks.

Scope: The paper introduces CARES, a small plug-in module that predicts the minimum image resolution needed to answer a given image-query pair, cutting visual token counts and prefill compute by roughly 65–85% across nine benchmarks without meaningfully hurting accuracy.

What This Paper Is About

Modern VLMs process every image at high or native resolution to be safe across unknown tasks, which causes visual tokens to dominate the input — up to 99% of prefill tokens — and drives up latency and cost. The authors argue that not every query needs that much visual detail: "What breed is the dog?" and "What name is on the collar?" have very different resolution requirements. CARES is a lightweight, model-agnostic preprocessing step that looks at the image and the query together and predicts the smallest resolution that will still let a downstream VLM answer correctly.

Key Contributions

1. Formalizes query- and image-conditioned resolution selection as a distinct efficiency lever that operates before tokenization, complementing existing post-tokenization token pruning and merging methods.
2. Introduces a supervision strategy based on multi-resolution rollouts of a pretrained VLM plus a convergence criterion (ANLS threshold plus a margin), producing a per-example "sufficient resolution" label without exhaustive search.
3. Proposes the CARES architecture: a frozen compact VLM proxy (~350M parameters, SmolVLM-500M truncated at layer 16) feeding a lightweight classifier over discrete resolutions, deployed as a continuous selector via probability-weighted interpolation. A second autoregressive variant based on Granite-Docling-258M with LoRA is also presented.
4. Demonstrates broad empirical gains: across nine multimodal benchmarks and four target VLMs (Granite-Vision 3.3-2B, InternVL3-8B, Qwen2.5-VL-72B, GPT-4o), prefill compute drops 65–85% — up to 78% on average — with at most sub-point accuracy changes.

Main Findings

• Large compute savings with accuracy parity: Averaged across models and datasets, prefill FLOPs fall 65–85%. For example, Granite-Vision-2B drops from 0.59 to 0.60 macro score at −63% cost; InternVL3-8B holds 0.77 at −64%; Qwen2.5-VL-72B edges from 0.79 to 0.80 at −70%; GPT-4o stays at 0.68 versus 0.69 at an estimated −55% dollar cost.
• Real latency gains, not just theoretical ones: On DocVQA, time-to-first-token for Qwen2.5-VL-7B falls from 435.7 ms (native) to 270.1 ms with CARES, and Granite-Vision 3.3-2B falls from 228.6 ms to 108.9 ms.
• Continuous resolution beats a discrete menu: Probability-weighted interpolation over the discrete resolution set reduces FLOPs by 63% versus 46% for discrete prediction at Granite-Vision-2B and InternVL3-8B, with no accuracy loss; on Qwen2.5-VL-72B it reaches −80%.
• Sufficiency labels generalize across teacher models: Comparing labels from Granite-Vision-2B and Qwen3-VL-235B on 1,000 shared examples, the two agree on the same sufficient resolution for more than 95% of cases (Pearson correlation 0.908, mutual information 1.116), suggesting "resolution adequacy" is a broadly shared property rather than a quirk of one annotator.
• Intermediate features outperform final-layer features: Extracting the hidden state at layer 16 rather than the output layer improves selection accuracy by roughly one percentage point while using only about half the LLM — SmolVLM reaches 63.3% versus 62.3%, Qwen2.5-3B reaches 67.2% versus 66.2%.
• A three-way resolution menu beats a binary one downstream: Binary {384, 1024} classification scores higher on the classification task itself (96.2% vs 67.2%) but produces worse downstream benchmark performance (0.76 vs 0.80), because finer granularity gives better routing control.
• Adaptive routing beats any fixed resolution: Against fixed-resolution baselines, CARES reaches 94.80 accuracy where native is 95.50, while 768 and 384 fixed settings fall to 93.35 and 89.00 — showing the gains come from selective escalation, not from VLMs simply tolerating downscaling.
• Label smoothing helps continuous deployment: Adding 0.05 label smoothing during training improves OCRBench performance for Qwen2.5-VL-7B (0.821 vs 0.811) while slightly lowering expected FLOPs (−63.8% vs −60.5%), by better calibrating the probabilities used for interpolation.
• Proxy-target mismatch is not a major problem: Swapping Qwen2.5-VL features for much smaller SmolVLM features produces nearly identical results across tested benchmarks, indicating the decision relies on coarse joint image-text cues rather than tight feature alignment.

Methodology in Plain English

The authors start by defining what "sufficient resolution" means. They take a dataset of image-query pairs and, for each one, run a pretrained VLM at three fixed resolutions: 384, 768, and 1024 pixels. They score each answer against the ground truth using ANLS (a normalized string-similarity metric suited to OCR-style answers). The label is the lowest resolution whose score clears a threshold of 0.85 and where no higher resolution improves the score by more than a small margin of 0.1. This turns an expensive search problem into a cheap annotation procedure, and the margin prevents rewarding trivial gains.

To build the selector, they take a small pretrained VLM (SmolVLM-500M) and cut off its upper layers, keeping roughly the first half. They feed it the image at low resolution (384) together with the query text, and grab the hidden state of the last token at layer 16 — a compact representation of the joint image-query context. A small classifier head then predicts a probability distribution over the three resolution choices. The whole base model stays frozen; only the head is trained, on 80,000 samples drawn evenly from TextVQA, ChartQA, DocVQA, and LLaVA-Multi. Training uses cross-entropy with 0.05 label smoothing for six epochs.

At inference, the discrete distribution is converted into a continuous resolution by taking a probability-weighted average of the three candidate resolutions. This smooth estimate is then used to resize the image before it is handed to the target VLM. Because CARES only touches the input image, the downstream VLM needs no retraining, no architectural changes, and no access to its internals — it works even for API-only models like GPT-4o. An alternative autoregressive instantiation fine-tunes Granite-Docling-258M with LoRA to emit dedicated resolution tokens, then applies the same interpolation rule.

Why This Matters

Impact on research. The paper reframes input resolution itself as a controllable efficiency lever — one that sits upstream of, and is orthogonal to, the dominant research direction of pruning or merging visual tokens after the vision encoder has already produced them. It also provides a reusable supervision recipe: turning a downstream task metric into per-example resolution labels via multi-resolution rollouts. The cross-teacher agreement result gives the community reason to believe these labels capture something stable rather than model-specific noise.

Real-world applications:

• Document AI pipelines handling invoices, forms, receipts, and scanned letters, where many queries (e.g., "who is the contact person?") are answerable from a thumbnail while others demand full detail.
• Chart and figure question answering in analytics dashboards, where simple trend questions and precise value-reading questions coexist in the same traffic stream.
• Mobile and edge deployment, where memory and compute budgets make high-resolution visual token counts a hard blocker for on-device multimodal assistants.
• API cost control for hosted models, where the paper's GPT-4o results show that routing easy queries to smaller images translates directly into dollar savings on inference.

Industry relevance. Cloud providers and API vendors pay for every visual token they prefill, and latency-sensitive products (search, assistants, agentic workflows) feel the cost most acutely. Because CARES is a drop-in front-end that does not require access to the target model's weights, it can be layered onto existing deployments — including closed commercial models — without a migration. The distribution of predicted resolutions (per-instance rather than per-task) also means operators get finer-grained control than a single global downscaling policy.

Future Directions

• Extending beyond single images: The current setup handles one image and one turn only. Multi-image inputs, video, and streaming interaction raise new questions about whether resolution should be chosen jointly across frames and whether temporal redundancy changes the sufficiency criterion.
• Joint resolution and tiling selection: CARES currently picks a resolution, but AnyRes and dynamic tiling introduce a second decision axis. Choosing both together could yield further savings or better fidelity on dense layouts.
• Domains that need very fine cues: The authors acknowledge that dense OCR, medical imagery, and similarly detail-critical tasks risk under-allocation, since the frozen proxy may not surface the subtle signals those domains depend on.
• Robustness, safety, and hardware-level trade-offs: The paper explicitly leaves unstudied how the selector behaves under adversarial prompts, model perturbation at inference, annotation noise, and the detailed cost-latency behavior across different hardware backends.

Target Audience

Practitioners and researchers working on multimodal model efficiency will get the most from this paper: ML engineers deploying VLMs in production and paying for visual tokens, systems researchers interested in adaptive computation, and document-AI teams whose workloads mix coarse and fine-grained queries. It is also useful for anyone studying supervision strategies for routing or early-exit policies, since the labeling procedure generalizes beyond resolution. Readers new to VLMs can follow the high-level argument — only a portion of the pixels in an image ever matter for a given question — but will need some background in tokenization and vision encoders to follow the implementation details.

Authors’ abstract

Large vision-language models (VLMs) commonly process images at native or high resolution to remain effective across tasks. This inflates visual tokens ofter to 97-99% of total tokens, resulting in high compute and latency, even when low-resolution images would suffice. We introduce \emph{CARES}-a \textbf{C}ontext-\textbf{A}ware \textbf{R}esolution \textbf{S}elector, a lightweight preprocessing module that, given an image-query pair, predicts the \emph{minimal} sufficient input resolution. CARES uses a compact VLM (350M) to extract features and predict when a target pretrained VLM's response converges to its peak ability to answer correctly. Though trained as a discrete classifier over a set of optional resolutions, CARES interpolates continuous resolutions at inference for fine-grained control. Across five multimodal benchmarks spanning documents and natural images, as well as diverse target VLMs, CARES preserves task performance while reducing compute by up to 80%.