NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints

AI summary

Overview

• Research area: Multimodal large language models (MLLMs) — computer vision combined with language modeling; specifically the "native" end-to-end training paradigm versus the dominant "compositional" paradigm.
• Technical level: Advanced. The paper deals with scaling laws, mixture-of-experts routing, visual encoder depth/width trade-offs, and large-scale pre-training recipes.
• Scope in one sentence: This paper studies how to design and scale natively trained (end-to-end) multimodal LLMs when training data is practically limited rather than infinite, and builds a model called NaViL from the resulting findings.

What This Paper Is About

Most MLLMs today are built compositionally: a visual encoder and a language model are pre-trained separately and then stitched together with extra multimodal training. Because the two halves are trained separately, it is hard to reason about how performance scales when you grow either half, so the authors instead study models where vision and language are optimized jointly, end-to-end. The core problem they address is whether this native approach can actually be competitive with top-tier compositional MLLMs at a realistic data budget, and what the right architecture and scaling choices are under that constraint.

Key Contributions

1. A systematic exploration of the design space for native MLLMs under data constraints — covering LLM initialization, mixture-of-experts (MoE) usage, and visual encoder architecture — resulting in an "optimal meta-architecture" that balances performance against training cost.
2. A study of scaling properties showing that the visual encoder and the LLM have positively correlated optimal sizes: the best visual encoder size grows roughly in proportion to LLM size on a log scale.
3. The proposal of NaViL, a native MLLM built with a simple, cost-effective recipe, evaluated on 14 multimodal benchmarks and reported as competitive with existing MLLMs using roughly 600M pre-training image-text pairs.
4. Large-scale pre-training and fine-tuning experiments, including a NaViL-9B variant based on Qwen3-8B, plus an attention-map analysis comparing small (150M) and large (1.2B) visual encoders.

Main Findings

• LLM initialization matters enormously: A model trained from scratch on multimodal data performed significantly worse than one initialized from a pre-trained LLM, and required over 10x more data to reach comparable validation loss. In zero-shot captioning the gap persisted even when the non-initialized model saw much more data, which the authors attribute to the lower textual quality and diversity of multimodal corpora compared with LLM pre-training data.
• MoEs accelerate convergence at no extra activation cost: Adding modality-specific MoE experts (with the number of activated experts set to one) let the model reach the same validation loss with only 1/10 of the data, without increasing training or inference cost. The authors note that using only a feed-forward-network expert caused a significant feature-scale mismatch between visual and language modalities, so they also introduced modality-specific attention experts (separate qkvo projection layers) with unified global attention.
• Visual encoder depth and width are surprisingly forgiving: With the LLM and visual encoder parameter count held fixed at 600M, depth was swept over {3, 6, 12, 24, 48} and width over {4096, 2880, 2048, 1472, 1024}. Only extreme configurations performed worse. Shallow encoders converged faster in the early phase (less than 30M data), but that advantage faded with more data, and deeper encoders were slightly better on zero-shot captioning.
• Scaling the LLM follows conventional scaling laws: Sweeping LLM sizes {0.5B, 1.8B, 7B} with a fixed 600M visual encoder, validation loss decreased log-linearly with LLM size.
• Scaling the visual encoder hits a ceiling: With visual encoder sizes {75M, 150M, 300M, 600M, 1.2B, 2.4B}, gains diminished progressively and beyond a certain size became marginal, indicating the multimodal model's upper performance limit is constrained by the LLM's capacity rather than the encoder's.
• Optimal encoder size tracks LLM size: Defining the optimal encoder as the smallest one whose loss difference versus an encoder twice its size is less than λ = 1% of the loss achieved with the 75M encoder, the log of the optimal encoder size scaled linearly with the log of the LLM size. This contradicts compositional practice, where a single pre-trained visual encoder is reused across all LLM scales.
• Encoder size changes attention behavior: Visualizing attention maps in the 1.8B LLM with 150M versus 1.2B encoders, shallow layers showed local attention patterns in both cases. With the 1.2B encoder, shallow-layer visual tokens attended more to global information, and attention between visual and text tokens in the first layer was significantly higher than with the 150M encoder — earlier cross-modal interaction.
• Reported benchmark performance of NaViL-2B: On the MLLM benchmark table, NaViL-2B (2.4B activated parameters) reports an average of 67.1, with MMVet 78.3, MMMU 41.8, MMB 71.2, MME 1822, MathVista 50.0, OCRBench 796, and CCBench 83.9. The compositional counterpart InternVL-2.5-2B reports an average of 67.0 (MMVet 60.8, MMMU 43.6, MMB 74.7, MME 2138, MathVista 51.3, OCRBench 804, CCBench 81.7). The best prior native model shown, Mono-InternVL (1.8B activated), reports an average of 56.4.
• Reported VQA performance of NaViL-2B: On the visual question answering table, NaViL-2B reports an average of 75.1, with TextVQA 76.9, SQA-I 95.0, GQA 59.8, DocVQA 85.4, AI2D 74.6, ChartQA 78.0, and InfoVQA 56.0. InternVL-2.5-2B reports an average of 76.5. The authors state NaViL still leaves a gap to the best compositional MLLMs, and attribute that to instruction data quality and LLM strength.
• Training-token efficiency: The paper contrasts training tokens: NaViL-2B is listed at 0 tokens for training the ViT and 800B for training the MLLM (800B total), and NaViL-9B at 0 and 450B (450B total). For comparison, Qwen2.5VL is listed as unknown for ViT and 4.1T for the MLLM (>4.1T total), and InternVL2.5-8B as >3.3T for the ViT and 140B for the MLLM (>3.5T total).

Methodology in Plain English

The authors first fix a general "meta-architecture": a visual encoder made of transformer layers, an LLM, and MoE layers injected into the LLM. The visual encoder takes an image, splits it into patches, processes them with bidirectional attention, and its output is downsampled and projected into the LLM's feature space.

To find good design choices, they train models on web-scale noisy image-caption pairs using next-token prediction and image captioning, and compare configurations by validation teacher-forcing loss on a held-out subset, plus zero-shot captioning. They vary one axis at a time: whether the LLM starts from a pre-trained checkpoint (InternLM2-Base) or from scratch, whether the LLM has MoE layers or not, and how the visual encoder splits a fixed parameter budget between depth and width.

For the scaling study, they scale one component while holding the other fixed, then scale both together and look for the encoder size beyond which doubling it buys less than a small threshold of improvement. Finally, they assemble the best choices into NaViL.

NaViL's recipe has two stages. Stage 1 pre-trains on 500 million image-text pairs — 300 million sampled from Laion-2B, Coyo-700M, Wukong and SA-1B, and 200 million where captions were synthesized by InternVL-8B. During that sub-stage the text parameters are frozen and only vision-specific parts (visual encoder, MLP projector, MoE visual experts) are trainable. A second sub-stage trains on 185 million high-quality samples mixing multimodal alignment data and pure language data, and unfreezes the textual parameters inside the self-attention layers. Stage 2 is supervised fine-tuning on 68 million higher-quality multimodal samples with all parameters unfrozen.

NaViL-2B is built on InternLM2-1.8B, has 4.2B total parameters and 2.4B activated parameters (of which 0.6B is the visual encoder). NaViL-9B is built on Qwen3-8B with 9.2B activated parameters, 1.2B of which belongs to the visual encoder. The model accepts images of any resolution, pads them to multiples of 32, uses patch-embedding stride 16, bidirectional attention with 2D-RoPE in the encoder and causal attention with 1D-RoPE in the LLM, and inserts special tokens to mark image boundaries, row ends, and scale ends. A "visual multi-scale packing" step feeds progressively downsampled copies of an image to the encoder at inference, with downsampling rate √2/2. Global batch size was 7000 in the first pre-training stage and 4614 in the second.

Why This Matters

For research, the paper challenges a widely used assumption in MLLM design — that one frozen, pre-trained visual encoder can serve every LLM scale. It offers evidence that the right encoder size depends on the LLM size and grows with it, and it argues that native training removes the bias introduced by training vision and language with different objectives (contrastive or self-supervised for vision, next-token prediction for language). It also reframes the native-versus-compositional comparison under a realistic data budget rather than an assumed infinite one.

Real-world applications implied by the benchmarks and tasks used:

• Optical character recognition and document understanding (OCRBench, DocVQA, InfoVQA), where any-resolution input handling is directly relevant.
• Chart and infographic comprehension (ChartQA, InfoVQA).
• Image captioning and general visual question answering (TextVQA, ScienceQA-IMG, GQA, AI2D).
• Multi-image and real-world comprehension tasks via MMBench, MME, MMVet, MMMU and MathVista.

For industry, the key claim is cost: a native model with zero separately trained ViT tokens and 800B total MLLM tokens for NaViL-2B, or 450B for NaViL-9B, versus the >4.1T and >3.5T totals listed for two compositional systems. If those efficiency claims hold, native training is a shorter path to a capable multimodal model for teams without massive separate vision and language pre-training pipelines.

Future Directions

• Scale beyond 9B. The paper explicitly states that, due to limited computation, it only investigates scaling properties up to 9B parameters, and suggests future experiments at 30 billion, 70 billion, and 100 billion parameters to validate the trend.
• Extend beyond vision and language. The authors note the work covers only visual and linguistic modalities and call for research on broader modality combinations and more general design principles.
• Close the remaining gap to the best compositional MLLMs. The authors attribute NaViL's residual gap to instruction-data quality and LLM capability, implying higher-quality instruction data and stronger base LLMs as the next lever.
• Complete and publish the full-scale comparisons. The provided content truncates the NaViL-9B benchmark table (Table 4), so the per-benchmark numbers for that model are not reported here.

Target Audience

Researchers and engineers working on multimodal LLM architecture, pre-training pipelines, and scaling laws, especially those with constrained compute or data budgets. It is also relevant to practitioners deciding how large a visual encoder to pair with a given language model, and to students who already understand transformer basics and want a worked example of how architecture and scaling studies are designed and interpreted.

Authors’ abstract

Compositional training has been the de-facto paradigm in existing Multimodal Large Language Models (MLLMs), where pre-trained vision encoders are connected with pre-trained LLMs through continuous multimodal pre-training. However, the multimodal scaling property of this paradigm remains difficult to explore due to the separated training. In this paper, we focus on the native training of MLLMs in an end-to-end manner and systematically study its design space and scaling property under a practical setting, i.e., data constraint. Through careful study of various choices in MLLM, we obtain the optimal meta-architecture that best balances performance and training cost. After that, we further explore the scaling properties of the native MLLM and indicate the positively correlated scaling relationship between visual encoders and LLMs. Based on these findings, we propose a native MLLM called NaViL, combined with a simple and cost-effective recipe. Experimental results on 14 multimodal benchmarks confirm the competitive performance of NaViL against existing MLLMs. Besides that, our findings and results provide in-depth insights for the future study of native MLLMs.