SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs

AI summary

Overview

Research area: Vision-language models (VLMs), test-time scaling, visual grounding, and context engineering for multimodal inference.

Technical level: Advanced. The paper assumes familiarity with VLMs, chain-of-thought reasoning, reinforcement learning for reasoning models, KV-caches, LoRA fine-tuning, and grounding-based coordinate prediction.

Scope: The paper proposes and evaluates SPARC, a two-stage prompting and training pipeline that separates visual perception (region localization) from reasoning in vision-language models, and measures its accuracy, token efficiency, and robustness against the "thinking with images" paradigm.

What This Paper Is About

Current multimodal chain-of-thought methods ("thinking with images") interleave visual operations such as zooming and cropping with long text reasoning traces, entangling perception and reasoning in one unstructured context. This is expensive, brittle, and depends on reinforcement learning with hand-crafted rewards. SPARC instead splits VLM inference into two explicit stages: a first stage that localizes question-relevant image regions, and a second stage that reasons only over those high-resolution crops to produce the final answer.

Key Contributions

1. SPARC, a prompting scheme that enables reliable test-time scaling of perception, in zero-shot mode and with very small computational overhead.
2. Demonstration that SPARC enables asymmetric compute allocation between perception and reasoning, allowing a targeted self-consistency mechanism that scales more favorably than naive ensembling.
3. Evidence that decoupling visual reasoning into separate perception and reasoning stages allows efficient training of the perception model without degrading the reasoning model's original capabilities.
4. A definition and analysis of Implicit Relevance Detection (IRD) as the first-stage task, distinguishing it from standard Referring Expression Comprehension.

Main Findings

• Accuracy gains over monolithic and grounding baselines: SPARC improves Qwen3-VL 4B on the V* VQA benchmark by 6.7 points and surpasses "thinking with images" by 4.6 points in an OOD setting with a 200× lower token budget.
• In-domain and out-of-distribution averages: In Table 1, the in-domain (ID) metric is the mean over V*, HRBench-4K and HRBench-8K, and the OOD metric is the average over the XLRS remote sensing benchmark. Qwen3-VL 4B goes from native 41.7 / 48.8 / 72.6 (256 / 512 / Full) to SPARC 51.0 / 60.6 / 74.8 on ID, and from 46.2 / 48.4 / 53.5 to 48.7 / 52.9 / 54.8 on OOD. Qwen3-VL 8B goes from 41.4 / 49.4 / 79.0 to 45.4 / 54.8 / 79.5 on ID, and from 47.5 / 47.9 / 53.2 to 52.4 / 53.3 / 56.0 on OOD. Molmo2 4B goes from 48.1 / 53.2 / 60.8 to 48.7 / 57.0 / 62.9 on ID, and from 38.1 / 39.1 / 39.9 to 40.8 / 42.1 / 43.2 on OOD. Molmo2 8B goes from 45.8 / 52.4 / 57.8 to 47.4 / 55.0 / 59.1 on ID, and from 37.5 / 39.4 / 39.1 to 39.1 / 39.9 / 39.0 on OOD.
• Perceptual self-consistency scaling: Running self-consistency over eight rollouts of the IRD step with a shared KV-cache creates only a few additional text tokens and one additional crop, but boosts performance of the full pipeline by up to 9.3%. Table 2 shows monotonic gains from 1 to 8 rollouts; for Qwen3-VL 4B with Weighted Boxes Fusion at 8 rollouts, scores reach 55.7 / 67.0 / 82.0, and the average number of final crops with 8 rollouts at 256 resolution is only 3.30.
• Low-resolution crops can substitute for high resolution: In the crop-overlap ablation, a model operating at 256-pixel resolution with only 20% overlap with the ground truth already surpasses a 512px model without object localization. At high overlap ratios, the 256px model converges to the performance of the full-resolution model. The same pattern is replicated with Molmo2-4B.
• OOD efficiency on remote sensing: On XLRS, whose average image dimension is 8500 × 8500 pixels, Molmo2 operating at 256-pixel resolution with crops surpasses the standard model prompted at full resolution, while processing approximately 0.1% of the visual tokens required for a naive full-resolution forward pass.
• Low-resolution training works best: In Table 3, the model trained at the lowest resolution (SPARC SFT 256) achieves the highest accuracy across most test settings. For Qwen3-VL 8B, SPARC SFT 256 reaches 53.1 / 64.3 / 82.4 versus 45.4 / 54.8 / 79.5 for the SPARC baseline. The sole exception is Molmo2-4B, where the fine-tuned model performs comparably to the baseline at lower resolutions, attributed to a distillation bottleneck from a relatively weak Molmo2-8B teacher.
• Crop count falls with resolution: As input image size increases from 256 to Full resolution, the average number of final crops consistently decreases, which the authors hypothesize reflects higher IRD confidence and greater consensus among rollouts.
• No degradation reported elsewhere: The authors present targeted studies (Appendices A.7 and A.12) showing SPARC has no detrimental influence on less perception-oriented benchmarks and does not suffer from "tunnel vision" or context collapse on queries requiring global or relational understanding.
• Instability in the baseline: The authors report unstable behavior with the Qwen3-VL family, where "thinking with images" sometimes does not even reach the native model performance.

Methodology in Plain English

The approach is a two-step prompting protocol applied to an off-the-shelf VLM.

First, the model is asked to output only the coordinates (for Qwen3-VL, bounding boxes) or points (for Molmo2) of image regions relevant to the question. This stage is called Implicit Relevance Detection, because the target region is not explicitly named as in referring expression comprehension — the model must infer latent visual relevance from a high-level reasoning prompt. For Molmo2, a squared 256 × 256 crop is extracted from each point; crops are taken from the original full-size image and downsized if they exceed the target resolution to avoid hacking. Evaluation is run at full resolution and at 256 and 512 longest image side, at 4B and 8B model sizes, with greedy decoding.

Second, the extracted crops are appended to the prompt and the model answers the original question conditioned on them.

For scaling, the authors run N independent IRD rollouts at a non-zero temperature of 0.7 and aggregate them with Weighted Boxes Fusion, which computes a weighted average of overlapping predictions rather than discarding them as Non-Maximum Suppression does. Boxes with at least 50% intersection over union are merged; distinct non-overlapping boxes are forwarded to the reasoning stage. Results are reported for N = 4 and N = 8.

For training, the authors generate synthetic spatial annotations from the DeepEyes dataset. Bounding-box annotations come from Qwen3-VL 235B-A22B, extracting crop coordinates from intermediate tool calls and keeping only traces that yield a correct final answer, yielding approximately 23,000 samples. Point-based annotations come from the Molmo2 8B variant, using the two-step pipeline and the same filtering protocol, yielding approximately 14,000 samples. A LoRA adapter is trained exclusively for the detection phase with supervised fine-tuning for two epochs under a standard autoregressive next-token prediction objective, at three resolution scales, and is activated only during perceptual search at test time.

The empirical motivation comes from an ablation that shifts crop centers relative to ground-truth bounding boxes and measures downstream VQA accuracy as a function of overlap.

Why This Matters

The work argues that unstructured multimodal chains of thought entangle perception and reasoning, and that structured, modular context composition produces better and cheaper results. It shifts the emphasis from inducing latent reasoning traces through reinforcement learning to explicit context engineering over a shorter, more focused visual context.

Real-world applications:

• Document understanding, one of the domains the authors list as a motivation for detailed visual perception.
• Outdoor robotics, where high-resolution scenes contain small task-relevant objects.
• Satellite and remote sensing image analysis, which the paper evaluates directly through the XLRS suite with images averaging 8500 × 8500 pixels.
• Any deployment where a model must handle distribution shift, since SPARC allows prioritizing perceptual processing under such shift.

Industry relevance: The method is training-free in its zero-shot form, works on off-the-shelf models with native spatial grounding, shares visual KV-caches between stages, and truncates context to avoid the quadratic costs of entangled chains of thought. It also lowers the fine-tuning requirement from reinforcement learning with hand-crafted rewards and custom reward shaping to standard supervised fine-tuning of a lightweight LoRA. The paper states that targeted training for IRD is extremely efficient in terms of both data and training time because only a small number of tokens for crop coordinates must be rolled out.

Future Directions

1. Exploring iterative zooming pipelines, extending the two-stage structure into repeated localization and inspection rounds.
2. Extending the framework to video inputs, where a lightweight perceptual circuit could perform spatio-temporal object tracking to isolate salient clips.
3. Applying targeted visual augmentations or super-resolution techniques directly to the extracted crops before the reasoning phase to further maximize downstream reasoning accuracy.
4. Removing the reliance on VLMs with native spatial grounding ability. The authors identify this as a current limitation and note that applying SPARC to purely text-generative VLMs would require integrating a dedicated external object detection model.

Target Audience

Researchers and engineers working on vision-language models, multimodal reasoning, test-time compute scaling, and visual grounding. It is most useful to readers already familiar with chain-of-thought methods, self-consistency, and fine-tuning pipelines who want a modular, compute-aware alternative to monolithic "thinking with images" systems. Practitioners deploying VLMs on high-resolution or out-of-distribution imagery, such as remote sensing or document analysis, will also find the efficiency analysis directly relevant.

Authors’ abstract

Despite recent successes, test-time scaling -- i.e., dynamically expanding the token budget during inference as needed -- remains brittle for vision-language models (VLMs). Unstructured visual reasoning chains entangle perception and reasoning, leading to long, disorganized contexts where small perceptual mistakes may cascade into completely wrong answers. Reasoning also requires expensive reinforcement learning with hand-crafted rewards. Here, we introduce SPARC (Separating Perception And Reasoning Circuits), a modular framework that explicitly decouples visual perception from reasoning. Inspired by sequential sensory-to-cognitive processing in the brain, SPARC implements a two-stage pipeline where the model first performs explicit visual search to localize question-relevant regions, then conditions its reasoning on those regions to produce the final answer. This separation enables independent test-time scaling with asymmetric compute allocation (e.g., prioritizing perceptual processing under distribution shift), and supports selective optimization (e.g., improving the perceptual stage alone when it is the bottleneck for end-to-end performance). It also accommodates compressed contexts by running global search at lower image resolutions and allocating high-resolution processing only to selected regions, thereby reducing visual token count and compute. SPARC outperforms monolithic baselines and strong visual-grounding approaches across challenging visual reasoning tasks, such as improving Qwen3VL 4B on the $V^*$ VQA benchmark by 6.7 points and surpassing "thinking with images" by 4.6 points in an OOD setting with a $200\times$ lower token budget.