Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models

AI summary

Overview

• Research area: Multimodal large language models (MLLMs) — specifically efficiency and model downscaling, sitting at the intersection of computer vision and language modeling.
• Technical level: Intermediate. The paper is readable without deep math, but it assumes familiarity with vision-language architectures (vision encoder plus projector plus LLM), visual instruction tuning, and the two-stage perception-then-reasoning (Prism) framework.
• Scope: A systematic study of what breaks when the language-model backbone of a multimodal system is shrunk, plus a training method (visual extraction tuning) and a combined pipeline (Extract+Think) that target the resulting bottlenecks.

What This Paper Is About

There is strong demand for small, efficient multimodal models that can run in constrained settings, but nobody has carefully characterized what actually degrades when you shrink the language model inside them. The authors ask which capabilities suffer most under LLM downscaling, and whether the damage comes mainly from weakened visual reasoning or also from a more fundamental loss of visual perception. They answer that perception is an underappreciated bottleneck, then propose a two-stage framework that trains a small perception module to extract instruction-relevant visual details and a small reasoning module to think step by step over them.

Key Contributions

1. A controlled downscaling study. The authors train multimodal models across Qwen3 LLM sizes (8B, 4B, 1.7B, 0.6B) with a fixed SigLIP vision encoder, a 2-layer MLP connector, and LLaVA-OneVision-style Higher AnyRes with Bilinear Interpolation, over a broad set of visual instruction tuning datasets with matching benchmarks. They find that LLM downscaling disproportionately harms vision-centric tasks rather than abilities inherited from the base LLM.

2. A decoupled perception/reasoning analysis. Using the Prism framework, they split each question into a perception stage (a VLM extracts relevant visual information) and a reasoning stage (an LLM produces the answer), then downscale each module independently. Isolating perception shows sharp drops that often match or exceed the drops from isolating reasoning.

3. Visual extraction tuning. A training paradigm in which the model is explicitly taught to generate the fine-grained visual details relevant to a given instruction, unifying the many perceptual skills that visual instruction tuning otherwise spreads across heterogeneous tasks.

4. Extract+Think. A final two-stage system combining a visually-extraction-tuned perception module with a chain-of-thought reasoning module, achieving strong performance with far fewer parameters and far less visual training data than comparable baselines.

Main Findings

• Downscaling hits vision-centric tasks hardest, not base-LLM tasks. Moving from 8B to 0.6B leaves most tasks with only modest decline, but Grounding drops 48% and NIGHTS (Perceptual Similarity) declines 38%. The tasks that rely heavily on the base LLM — ScienceQA for knowledge, GQA for general abilities — are comparatively unaffected.

• The more a task depends on visual information, the more it suffers from downscaling. Plotting the 8B-to-0.6B performance gap against the gap between performance with and without visual input (at 8B) yields an approximately linear trend across the full set of datasets. The exception is ImageNet, where the small model performs strongly but blind performance is near zero — likely because the required perception is simple and the task makes up a large portion of the visual instruction tuning data.

• Perception is a central bottleneck, not just reasoning. In the decoupled setup, downscaling the perception module from 8B to 0.6B causes an average accuracy drop of 0.15 for in-domain data and 0.07 for out-of-domain data. Even on tasks targeting visual reasoning (Instance Reasoning, Logical Reasoning), downscaling perception has an impact comparable to or exceeding downscaling reasoning.

• A theoretical explanation for the perception bottleneck. The authors invoke the Quantization Model of neural scaling laws: skills can be "quantized" into discrete chunks, and scaling laws limit how many a model can effectively learn. Visual instruction tuning requires many diverse perceptual skills, so smaller models learn fewer of them; the authors hypothesize the bottleneck arises from the heterogeneity of how perception is learned.

• Captioning is a helpful but insufficient baseline. Post-training the perception module on ALLaVA-4V (a 950K caption dataset) mitigates downscaling effects. For a 0.6B perception module, in-domain performance moves from 77.6 to 82.8 and MMStar from 40.4 to 44.0; for 1.7B, in-domain goes from 80.3 to 84.4 and MMStar from 44.4 to 49.0. But captioning does not teach question-relevance or domain-specific visual concepts.

• Visual extraction tuning adds substantial gains on top of captioning. Using a pipeline built on Qwen3VL-8B applied to 382K training samples, in-domain performance increases by 5.2 for a 0.6B perception LLM and 4.1 for a 1.7B LLM; MMStar improves by 3.6 and 4.6 respectively.

• Chain-of-thought helps reasoning without any visual supervision. Enabling Qwen3 thinking mode with a 4096-token thinking budget (with NoWait to limit self-reflection) improves out-of-domain performance across all LLM sizes. In-domain gains are concentrated at intermediate scales (4B and 1.7B), while 8B and 0.6B perform similarly with or without CoT — suggesting CoT helps but does not fully resolve the reasoning bottleneck at small scale.

• Extract+Think is extremely parameter- and data-efficient. The smaller variant (0.6B perception, 1.7B reasoning) reaches 80.3 in-domain average and 46.6 MMStar average, surpassing the largest PrismCaptioner model with a perception module roughly 12x smaller and a reasoning module 41x smaller. It outperforms LLaVA-OneVision-0.5B by 12.9% in-domain and 19.5% on MMStar while using 73% fewer visual samples. The from-scratch Extract+Think variant (0.4M visual samples) improves over LLaVA-OneVision-0.5B by 9.7% in-domain while using 95% fewer visual samples.

• The trend generalizes across model families. Additional analysis on MMStar with InternVL2.5 (8B to 2B) and Gemma3 (12B to 4B) shows consistent accuracy drops when downscaling the perception module (4.33 and 5.26, respectively, versus 3.30 for the authors' controlled Qwen3 8B to 1.7B setup).

Methodology in Plain English

The authors first build a controlled testbed rather than comparing off-the-shelf models, so that architecture and data stay fixed while only LLM size changes. They pair the Qwen3 family (8B, 4B, 1.7B, 0.6B) with a SigLIP vision encoder and a simple two-layer MLP connector, pretrain the connector on BLIP558K, then run visual instruction tuning on 574K single-image examples followed by 309K multi-image examples plus 150K randomly sampled single-image examples. They pick datasets that come with both training data and benchmark evaluations, spanning single-image tasks (OCR-VQA, VQAv2, ImageNet, VizWiz, ScienceQA, TextVQA, GQA, Grounding) and multi-image tasks (Text-Rich VQA, Spot the Difference, Image Edit Instruction, Multi-Image-VQA, Visual Story Telling, Puzzles, Perceptual Similarity).

To separate perception from reasoning, they adopt the Prism two-stage framework. A question is first converted offline into a question-specific instruction telling the model what visual elements to look for; that instruction plus the image goes into a perception module, which produces a text description of the relevant visual details; a reasoning module then answers from that description. Because the question-specific instructions are generated offline by one fixed model (Qwen3-8B), the questions stay identical across all comparisons, so any difference must come from the module being downscaled. Evaluation uses the multiple-choice AutoConverter format plus the out-of-domain MMStar benchmark.

To fix the perception bottleneck, the authors generate their own training data: each visual instruction example is converted into a declarative statement, which is turned into a prompt asking for fine-grained visual details relevant to that statement; Qwen3VL-8B generates the target description. This yields 382K visual extraction samples that the perception module is post-trained on.

Finally, for reasoning they turn on Qwen3's thinking mode and cap the thinking budget at 4096 tokens, then combine the extraction-tuned perception module with this step-by-step reasoning module. They present two size configurations — 0.6B perception with 1.7B reasoning, and 1.7B perception with 4B reasoning — with the reasoning module deliberately larger than the perception module.

Why This Matters

Research impact. Prior work gave conflicting signals about whether perception is sensitive to model scale — some studies said scale barely affects perception, others found perception-heavy tasks like OCR and Chart VQA highly sensitive. This paper provides a controlled decoupled analysis that isolates perception from reasoning while holding the training mixture and architecture fixed, reframing perception as a first-class bottleneck in small multimodal models rather than an assumed-safe component. It also shows that a data- and parameter-efficient training recipe can partially close that gap.

Real-world applications (implied by the tasks and settings studied, not separately evaluated in the paper):

• On-device visual assistants, where a small LLM backbone must run locally but still answer questions about images.
• Document and text-rich image understanding, such as OCR-VQA, DocVQA, and Text-Rich VQA.
• Visual grounding and referring-expression tasks (RefCOCO, RefCOCO+, RefCOCOg), where perception failures are especially costly.
• Multi-image workflows such as image editing instructions, spot-the-difference, and visual storytelling, where the model must track changes across images.

Industry relevance. The results speak directly to the tradeoff engineers face when deploying multimodal systems under memory, latency, and cost constraints. The finding that perception, not just reasoning, degrades under downscaling shifts where designers should spend their limited parameter and data budget. Extract+Think's headline efficiency claims — 12x smaller perception module and 41x smaller reasoning module than the largest PrismCaptioner, and 95% fewer visual training samples than LLaVA-OneVision-0.5B for the from-scratch variant — suggest a practical route to compact generalist models.

Future Directions

• Explore downscaling across a broader range of model sizes than the Qwen3 8B/4B/1.7B/0.6B sweep used here.
• Compare downscaling of visual representations (the vision encoder) against downscaling of the language model, which this work holds fixed.
• Incorporate data size as a variable to study how downscaling behavior varies across different data scales.
• Investigate visual extraction tuning more directly against standard visual instruction tuning, and test whether the paradigm remains effective with larger language models.

Target Audience

Researchers and engineers working on multimodal models who care about efficiency: people building small VLMs, practitioners deciding where to spend parameters and training data under deployment constraints, and anyone studying scaling behavior or failure modes of vision-language systems. The paper is also useful to readers interested in perception-versus-reasoning decompositions of multimodal capability, since it provides both a diagnostic framework and a recipe. Readers without background in vision-language architecture will need to pick up the basics of connectors, visual instruction tuning, and two-stage perception-reasoning pipelines first.

Authors’ abstract

Scaling up multimodal models has enabled remarkable advances in visual understanding and reasoning, but practical demands call for smaller, efficient systems. In this work, we conduct a principled analysis of downscaling intelligence in multimodal models, examining how reduced large language model (LLM) capacity affects multimodal capabilities. Our initial findings reveal an interesting trend: LLM downscaling disproportionately affects visual capabilities, rather than abilities inherited from the LLM. We then examine whether this drop mainly reflects the expected decline in visual reasoning or a more fundamental loss of perceptual abilities. Isolating the effect of LLM downscaling on perception, we find performance still drops sharply, often matching or exceeding the impact on reasoning. To address this bottleneck, we introduce visual extraction tuning, which explicitly trains the model to extract instruction-relevant visual details consistently across tasks. With these extracted visual details, we then apply step-by-step reasoning to generate answers. Together, these components form our Extract+Think approach, setting a new standard for efficiency and performance in this space.