ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs

AI summary

Overview

Research area: Multimodal large language models (MLLMs), specifically scaling methods that increase computation without increasing backbone parameter count.

Technical level: Advanced. The paper assumes familiarity with Vision Transformer (ViT) / LLM architectures, attention key-value (KV) prefixes and caches, supervised fine-tuning (SFT) recipes, mixture-of-experts (MoE) routing, and FLOP/latency/memory profiling.

Scope: The paper introduces and evaluates ParVL, a framework that reuses a shared ViT and LLM backbone across multiple parallel branches so that extra computation can be allocated independently between the vision and language components.

What This Paper Is About

Most ways of making multimodal models stronger either add parameters (costing memory) or add sequential inference steps such as chain-of-thought (costing latency). Neither changes the fixed ratio of computation spent inside the vision encoder versus the language decoder, which is decided at design time. ParVL instead reuses the same ViT and LLM weights across several parallel branches, each distinguished only by small learned prefixes, so that the number of vision branches (P_v) and language branches (P_l) can be varied independently. The paper asks: given a fixed backbone parameter budget, where should the extra shared-backbone computation go?

Key Contributions

1. The ParVL framework. A parameter-sharing, parallel-scaling architecture for MLLMs that adds prefix-conditioned ViT and LLM branches over shared backbones, with modality-specific token-wise MLP aggregation. The authors state this is the first study to jointly scale the ViT encoder and LLM decoder through parallel parameter reuse.
2. A systematic vision–language allocation study. Nine branch-count configurations across P_v, P_l in {1, 2, 4}, covering balanced, vision-heavy, and language-heavy settings, showing that the preferred allocation varies by benchmark rather than by domain label alone.
3. Controlled comparisons at three scales. Same-recipe comparisons at 1B, 2B, and 8B against single-branch baselines, with total parameters increasing by at most 4% across the evaluated configurations.
4. Cost characterization. Latency and memory profiling, a comparison of KV prefixes versus LoRA for parameterizing branches, and an exploration of sparse top-1 routing over the trained branches.

Main Findings

• Parallel branches beat the same-recipe single-branch baseline. At 1B, the 9-benchmark average rises from 49.6 (P_v=1, P_l=1) to 50.5 at 4:4; the language-heavy 1:4 configuration reaches 50.3. At 2B the balanced 2:2 configuration gives 54.7 versus 54.4, and at 8B it gives 63.0 versus 62.5.
• The largest configuration is not always best. At 1B, General peaks at 41.1 under 2:2, above the 40.3 single-branch baseline, but falls below that baseline under 1:4 and 4:4. The 4:4 configuration trails the best observed score by 2.1 on General and 0.4 on OCR.
• Gains concentrate in math and OCR/document tasks. At 1B, Math rises from 26.0 to 28.0 at 4:4, and OCR from 75.6 to 76.3 at 2:4. At 2B, 2:2 improves Math by 0.3 and OCR by 0.5, while General moves from 46.4 to 46.1. At 8B, 2:2 improves General, Math, and OCR by 0.9, 0.8, and 0.1 respectively.
• Allocation preferences are task-specific, not domain-specific. MathVision favors language-heavy 1:4, LogicVista favors vision-heavy 4:1, and MathVista favors 4:4, improving on 1:1 by 3.6, 2.7, and 4.1 respectively. Within OCR/document tasks, ChartQA and DocVQA favor 2:4 while TextVQA and OCRBench favor 4:4.
• More branches do not monotonically help. At P_l=2, increasing P_v from 1 to 2 or 4 lowers the 9-benchmark average from 49.9 to 49.4 and 49.3; raising P_l to 4 recovers it to 49.9 at 2:4 and 50.5 at 4:4.
• KV prefixes outperform LoRA for branch differentiation. Under P_v=4, P_l=1, a 72-token visual prefix is best on General (41.7) and OCR (75.8), while 128 tokens is best on Math (27.6) and overall (50.2). The authors select 96 tokens as a compromise, matching the 72-token overall score of 50.1 while improving Math by 0.7 and using 25% fewer visual prefix tokens than the 128-token setting. The best LoRA variant (rank 16, General 41.3, OCR 75.7, overall 49.8) remains 0.4 below the best prefix result.
• Sparse routing nearly matches dense aggregation. From a checkpoint trained at 4:4, a learned router activating one ViT–LLM branch pair per sample scores 50.2 overall versus 49.5 for random pair selection and 50.5 for dense 4:4 aggregation, with gains concentrated in Math (26.6 to 27.5) and OCR (75.1 to 75.8) while General stays at 38.8.
• Latency grows more slowly than memory. At batch size 1, dense 4:4 uses 1.04x the latency and 1.23x the peak memory of 1:1. At batch size 8, standard 4:4 rises to 1.22x latency and 2.12x memory. A Shared KV variant, which mean-reduces ordinary-token caches across LLM branches while retaining branch-specific prefix K/V, lowers the batch-8 memory ratio to 1.47x but raises latency to 1.62x, with only a small drop in the 9-benchmark average.

Methodology in Plain English

The researchers start from an existing multimodal model (InternVL3.5 at 1B, 2B, and 8B) and keep its ViT and LLM weights shared. Instead of copying the model, they run several parallel "streams" through the same weights. To keep the streams from being identical, each one gets its own small learned key/value prefix at every attention layer — a short block of extra context prepended to the keys and values only, while the query path and all projection matrices stay shared. Prefixes are the only branch-specific backbone parameters.

After the vision branches run, a small two-layer MLP looks at all branches' features for each token and produces weights over them, which are then used to average the branches (a smoothing coefficient, set to λ_v = 0 and λ_l = 0.1, blends these weights toward uniform). The fused visual tokens go through the standard class-token removal, pixel shuffle, and connector into the LLM's hidden space, then are fed identically to every language branch. The language branches' outputs are fused the same way before the shared language-modeling head.

Everything — shared backbones, prefixes, aggregators, and connector — is trained jointly with full-parameter SFT using a length-normalized next-token loss (weighting each sequence by 1/sqrt(number of valid tokens)). Training uses roughly 13B tokens, a 1/20 subsample of the InternVL3.5 SFT collection (4.6M instances), for one epoch. The authors then sweep P_v and P_l over {1, 2, 4} and score nine benchmarks, while separately profiling latency and memory on an H200 GPU.

Why This Matters

This work reframes a design choice that most multimodal models make silently — how much computation the vision encoder gets relative to the language decoder — as an explicit, tunable axis. Because shared-backbone branches can execute concurrently, the extra computation need not translate into proportional wall-clock latency growth when hardware allows. The finding that no single branch configuration wins across all nine benchmarks is a direct argument against any single fixed vision/language split.

Real-world applications suggested by the task-level results:

• Document and OCR-heavy pipelines (e.g., ChartQA, DocVQA, OCRBench), where the study finds limited but measurable gains from specific allocations such as 2:4 or 4:4.
• Mathematical and logical reasoning over diagrams (e.g., MathVista, MathVision, LogicVista, WeMath), where the best allocation differs per benchmark — language-heavy for MathVision, vision-heavy for LogicVista.
• General multimodal assistants, where the balanced 2:2 configuration produced the highest General score at 1B (41.1) and also improved General at 8B.
• Low-concurrency or single-request serving, where within-request parallelism is described as most attractive because it provides hardware concurrency when batching across requests is not available.

Industry relevance: the at-most-4% parameter overhead, the shared-KV memory option, and the top-1 router that reduces execution from 4:4 to 1:1 suggest a deployment path where one trained checkpoint can be run densely for quality or sparsely for cost. The authors caution that all profiling results are specific to the tested hardware (NVIDIA H200) and workload.

Future Directions

• Extend parallel scaling from SFT to pretraining, to see how joint vision–language scaling behavior emerges rather than only how an existing checkpoint is adapted.
• Test whether the modest gains at larger scales reflect the restricted data budget. Training used only a 1/20 subsample of the InternVL3.5 SFT collection and a single data mixture; controlled data-scaling experiments are named as needed.
• Validate across broader backbones and larger scales, since only InternVL3.5 at 1B, 2B, and 8B was studied.
• Combine with token-efficient visual backbones based on adaptive token pruning or patch merging, which the authors suggest could offset multi-branch computation cost, and evaluate on real deployment workloads.

Target Audience

Researchers and engineers working on multimodal model architecture, compute-efficient scaling, and inference serving. It is most useful to readers already comfortable with transformer attention, KV caching, and SFT pipelines who want a concrete, controlled study of vision-versus-language compute allocation. Readers looking for an off-the-shelf accuracy improvement should note the gains are modest (for example, 49.6 to 50.5 on the 9-benchmark average at 1B) and that no single configuration wins everywhere. Code is available at https://github.com/YangYangGirl/ParVL.

Authors’ abstract

Existing scaling strategies for Multimodal Large Language Models (MLLMs) typically expand either model parameters or sequential inference computation, incurring substantial memory or latency overhead. More importantly, most existing methods fail to alter the rigid, fixed computation allocation between the Vision Transformer and the Large Language Model components, limiting task-specific optimization. To address this, we introduce the Parallel Vision-Language (ParVL) scaling framework for MLLMs, which scales parallel computation by reusing the existing ViT and LLM backbone parameters across multiple vision and language branches. This framework raises a central question: given a fixed backbone parameter budget, how should additional shared-backbone computation be allocated between the vision and language modalities? We instantiate each parallel computational stream with branch-specific prefix parameters over a shared backbone, and train the entire model end-to-end via full-parameter supervised fine-tuning on roughly 13B tokens. We systematically study the computation-allocation trade-off between the ViT encoder and LLM decoder. ParVL improves overall multimodal performance over same-recipe single-branch baselines, and the best evaluated vision--language allocation varies across tasks. Code is available at https://github.com/YangYangGirl/ParVL.