VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs

AI summary

Overview

• Research area: Computer Vision / multimodal large language models (MLLMs) — specifically visual token efficiency and serving infrastructure.
• Technical level: Advanced.
• Scope: The paper introduces VisionWeave, a training method that gives MLLMs a native ability to choose, per spatial block, whether to represent visual content with fine-grained tokens or learned coarse-grained tokens, evaluated on eight benchmarks plus an SGLang deployment.

What This Paper Is About

Multimodal large language models encode images and video as dense sequences of fixed-size patch tokens, which becomes expensive at high resolution and long video length, yet visual information is unevenly distributed — some regions need detail, others do not. Existing token pruning and merging methods apply preset compression ratios, discard task-relevant content, and are awkward to deploy in modern serving engines. VisionWeave's goal is to make the choice of where and at what granularity to compress a learned, content-adaptive capability of the model itself.

Key Contributions

1. Elastic visual representation weaving as a native capability. The authors state this is the first effort to establish content-adaptive visual granularity in frontier MLLMs through large-scale training, validated on Qwen3.5-4B and scaled to Qwen3.8-27B with over 30K A100 GPU-hours.
2. A gated spatial pooler. A learned pooling module maps each 2×2 neighborhood of ViT patch features to one pooled feature; passing it through the native merger produces coarse-grained tokens placed at block-center MRoPE coordinates, complementary to the native fine-grained tokens in the same coordinate frame.
3. A granularity router. A learned module probes ViT features at depths 0, L//2 and L with local and global cross-attention blocks and predicts a per-block probability of using the coarse representation, routing at a threshold of 0.5 by default.
4. A three-stage self-distillation recipe and an SGLang integration. Training uses only the model's own native forward pass as teacher, and the deployment work adds support for block-center RoPE coordinates and mixed-granularity sequences to the SGLang serving engine.

Main Findings

• Content-adaptive savings with near-native quality: On Qwen3.8-27B at a 512-token-per-image budget, VisionWeave saves 16.5–55.1% of visual tokens across eight benchmarks, averaging 43.0% savings while preserving 98.9% of native performance; average performance loss is 1.08% on Qwen3.8-27B and 0.90% on Qwen3.5-4B.
• Fixed-budget baselines degrade: FastV† and VisionZip retain 88.2% and 87.5% respectively at a fixed 50% savings target, with average losses of 11.82% and 12.46% on Qwen3.8-27B (7.96% and 9.08% on Qwen3.5-4B).
• Task-dependent allocation: Savings are smaller on information-dense tasks and larger on video — on Qwen3.8-27B, 16.5% savings on InfoVQA (3.40% loss) and 27.3% on DocVQA (1.23% loss), versus 49.6–54.2% across three video benchmarks with losses at or below 0.30%. VisionWeave loses 3.17% on ScreenSpotV2 while saving 55.1%; VisionZip loses 47.88% there.
• Matching savings ratios is not enough: When FastV† and VisionZip are granted VisionWeave's per-dataset savings ratios, VisionWeave still has the highest average score — 72.59 versus 72.20/71.90 on Qwen3.5-4B, and 76.73 versus 70.10/70.33 on Qwen3.8-27B.
• Serving gains on SGLang: 2.30× end-to-end throughput (0.90 to 2.07 req/min), mean TTFT reduced 54.4% (169.90 s to 77.54 s), P95 TTFT reduced 57.1% (281.41 s to 120.59 s), mean TPOT reduced 60.6% (355.64 ms to 140.29 ms), P95 TPOT reduced 59.6% (715.58 ms to 289.19 ms).
• Inference-time control: Lowering the routing threshold from 0.5 to 0.25 increases savings from 54.2% to 66.0% on Video-MME with Qwen3.8-27B at only a 0.37-point loss; on Qwen3.5-4B savings rise from 51.4% to 62.1% while preserving the score of 67.22. At t=1 no tokens are saved.
• Gated pooling beats pixel-unshuffle: At 75% token savings after Stage 1 on Qwen3.5-4B, the gated pooler scores higher on all eight benchmarks, including gains of 2.64 points on DocVQA and 2.77 points on VideoOCR.
• Learned allocation beats shuffled allocation: Shuffling block-wise routing scores within each input while preserving the number of coarse selections reduces mean scores by 2.31 and 5.36 points on Qwen3.5-4B and Qwen3.8-27B; excluding ScreenSpotV2, the decreases are 1.30 and 1.53 points.
• Training dynamics: In Stage 2, mean router entropy falls from 0.658 to 0.427 nats between the first and last 100 steps; from step 50 to 2,500 the validation share of blocks with routing probability in [0.4, 0.6) drops from 64.8% to 14.6%, while the share at probability ≥ 0.9 reaches 28.3%.
• Temporal coverage: With a 256-frame cap, Qwen3.8-27B VisionWeave outperforms the native model capped at 128 frames by 1.20 points on LongVideoBench while using fewer visual tokens.

Methodology in Plain English

The method adds two small learned modules to an existing MLLM and trains them without any external labels, using the model's own unmodified forward pass as the teacher.

First, a gated spatial pooler takes each 2×2 group of ViT patch features, normalizes them, computes a shared context across the group, and uses channel-wise softmax gates (plus a learned value refinement) to produce one pooled feature. Feeding this pooled feature through the model's existing merger yields a coarse-grained token that stands in for four fine-grained ones. Because the final layers of the gate and value networks are zero-initialized, the pooler starts out as plain mean pooling. Coarse tokens get a positional coordinate at the geometric center of the four tokens they replace, so everything lives in the same MRoPE frame.

Second, a granularity router maintains one learnable query per spatial block. The query attends first to its own 4×4 patch features and then to the whole frame, at three ViT depths (0, L//2, L), giving six cross-attention blocks in total. A linear head turns the final query state into two logits, and the block uses its coarse token if the coarse probability exceeds the threshold (0.5 by default).

Training proceeds in three self-distillation stages, all using a forward KL divergence over the teacher's top-512 vocabulary entries at temperature 1, off-policy, on responses generated by the frozen native model:

1. Stage 1 trains only the pooler with routing bypassed (all blocks coarse, 25% of tokens), so the student learns to produce good coarse representations.
2. Stage 2 trains only the router, replacing hard selection with a differentiable soft-mixing surrogate that interpolates fine and coarse features with separately handled directions and magnitudes, plus a balance loss (weight 0.02, target coarse fraction ρ = 0.8).
3. Stage 3 trains the pooler and the LLM together on the hard-routed mixed-granularity sequences actually used at inference, with the ViT, merger and router frozen.

For deployment, the authors integrate into SGLang by doubling position coordinates and halving rotary frequencies to represent block-center half-integer positions in integer-indexed RoPE caches, and by attaching layout metadata to visual responses so mixed-granularity layouts are finalized before scheduling while preserving dynamic batching, KV-cache management and CUDA-graph decoding. The 0.78 million training samples span image understanding (LLaVA-OneVision, 246K), video understanding (LLaVA-Video-178K, 281K) and GUI grounding (UGround, 250K), and stages are run in BF16 with Adam (β1 = 0.9, β2 = 0.999, ε = 10⁻⁸), zero weight decay and cosine decay; stage learning rates are 1×10⁻⁴, 1×10⁻⁵ and 3×10⁻⁶ over 5,257, 2,500 and 1,000 steps respectively.

Why This Matters

Impact on research. The paper argues that representational granularity should be treated as a learnable degree of freedom rather than a fixed design choice, and shows a large-scale training path for making it native to frontier MLLMs. It also provides evidence that content-adaptive savings outperform fixed-ratio pruning and merging on information-dense and grounding tasks, and that learned routing placement matters beyond just the number of tokens retained.

Real-world applications:

• Long-video and streaming analysis, where the reported savings reach 49.6–54.2% on video benchmarks and serving throughput more than doubles under a 256-frame workload.
• GUI agents and screen understanding, where fixed-budget pruning collapsed on ScreenSpotV2 (47.88% loss for VisionZip) while VisionWeave lost 3.17% with 55.1% savings.
• Document and infographic processing, where adaptive savings drop to 16.5–27.3% to protect fine text detail instead of discarding it.
• Cost-sensitive multimodal serving, where inference-time threshold control lets one deployed model trade quality for cost without retraining.

Industry relevance. The SGLang integration and measured latency gains (mean TTFT down 54.4%, mean TPOT down 60.6%) make the approach directly relevant to operators of multimodal inference infrastructure, not only to model researchers.

Future Directions

• More comprehensive exploration of architectural choices, which the authors say is limited by the substantial computational cost of training.
• Broader investigation of on-policy versus off-policy self-distillation, building on their finding that on-policy training in Stage 3 of Qwen3.5-4B gave no consistent gains.
• Understanding and further exploiting the routing distributions, such as the observed concentration of routing probabilities at the extremes during Stage 2.
• Extending elastic visual weaving to other frontier MLLM architectures and vision backbones; note that the paper's stated results are for Qwen3.5-4B and Qwen3.8-27B, and the limitations section in the provided content is truncated.

Target Audience

Researchers and engineers working on multimodal LLM efficiency, visual token compression, and LLM serving systems, as well as practitioners who deploy vision-language models at scale and need to reason about the quality-cost trade-off. Readers should be comfortable with ViT patch representations, MRoPE positional encoding, self-distillation, and inference-serving concepts such as TTFT and TPOT.

Authors’ abstract

Multimodal large language models have become the dominant paradigm for visual understanding, but incur substantial costs by encoding inputs into dense, fixed-size patch tokens. However, visual information is unevenly distributed: some regions require fine-grained detail, while others admit compact representations. Downsampling sacrifices this detail, while existing token pruning and adaptive approaches remain limited in content-adaptive granularity, task generalization, and integration with modern MLLMs and serving infrastructure. Overcoming these limitations calls for foundation models that learn, end to end, where-and at what granularity-to allocate visual representations, a native capability we term elastic visual representation weaving. We introduce VisionWeave, establishing this capability in frontier-level MLLMs through large-scale training. It combines two components: a gated spatial pooler constructs coarse-grained representations alongside native fine-grained representations within a shared MRoPE coordinate, while a granularity router learns their content-adaptive allocation. Through self-distillation alone, we validate this capability on Qwen3.5-4B and scale to Qwen3.8-27B with over 30K A100 GPU-hours. Based on Qwen3.8-27B, VisionWeave adaptively adjusts token savings to visual content, saving 43.0% tokens on average while retaining 98.9% native performance across eight benchmarks, versus only 88% performance preserved for token pruning baselines with a fixed 50% savings target. Extensive evaluations confirm robust efficiency-quality trade-offs across diverse tasks, resolutions and video frames. When deployed on SGLang serving engine, our method achieves a 2.3x throughput gain while reducing mean TTFT by 54.4% and mean TPOT by 60.6%. Together, we believe these results position elastic visual weaving as a promising capability for next-generation multimodal models.