Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers

AI summary

Overview

Research area: Multimodal Large Language Models (MLLMs) — specifically, the architecture question of how visual features from a hierarchical vision transformer (ViT) should be wired into the layers of a language model.

Technical level: Advanced. The paper assumes familiarity with ViT/LLM layer stacks, cross-attention, softmax routing, residual gating, and standard MLLM training recipes (two-stage pre-training plus supervised fine-tuning).

Scope: The paper proposes Dynamic Gated Pathways (DGP), an all-to-all routing mechanism connecting every ViT layer to every LLM layer, and evaluates it on LLaVA-1.5-7B/13B, Qwen3-VL-2B, and LLaVA-NeXT across eleven multimodal benchmarks plus ablations, efficiency measurements, and routing visualizations.

What This Paper Is About

Most MLLMs take the output of one fixed ViT layer — usually the last or second-to-last — and feed it into the language model once, so the LLM sees only a single level of visual abstraction even though its own layers process language at very different depths. The authors argue the real open question is where to connect: which visual granularity each LLM layer should receive, and whether that choice should be made adaptively rather than fixed in advance. DGP answers this by letting every LLM layer dynamically choose its own weighted blend of all ViT layers, conditioned on its current hidden state.

Key Contributions

1. A reframing of visual injection around "where to connect." The paper argues that fixed single-layer injection and predefined sparse or hierarchical ViT-to-LLM wiring both fail to match the differentiated demands of hierarchical ViT representations and layer-wise LLM semantics.
2. DGP (Dynamic Gated Pathways), an all-to-all cross-layer connection mechanism. Every ViT layer is a candidate for every LLM layer, and each LLM layer produces its own routing distribution over the ViT hierarchy, conditioned on its own semantic state, then injects the fused result through a gated residual connection.
3. An entropy regularizer on the batch-averaged routing distribution that prevents routing from collapsing onto a few visual layers, applied with λ = 0.01 during pre-training and disabled (λ = 0) during supervised fine-tuning.
4. Empirical validation plus interpretability analysis. Consistent gains over LLaVA-1.5 baselines and over prior multi-layer fusion methods, generalization to Qwen3-VL-2B and LLaVA-NeXT, and layer-wise routing visualizations showing that different LLM layers genuinely select different visual granularities.

Main Findings

• Gains over LLaVA-1.5-7B: DGP reaches 64.6 on GQA (+2.6), 71.0 on SQA^I (+4.2), 79.7 on VQAv2 (+1.2), 53.1 on VizWiz (+3.1), 61.5 on SEED (+2.9), 67.5 on SEED^I (+1.4), 66.5 on LLaVA^W (+3.1), 33.8 on MM-Vet (+3.3), 66.7 on MMB (+2.4), 60.5 on MMB^CN (+2.2), and 87.0 on POPE (+1.1).
• Gains over LLaVA-1.5-13B: 65.5 on GQA (+2.2), 73.0 on SQA^I (+1.4), 80.7 on VQAv2 (+0.7), 57.4 on VizWiz (+3.8), 63.0 on SEED (+1.4), 69.0 on SEED^I (+0.8), 70.0 on LLaVA^W (−0.7), 37.5 on MM-Vet (+2.1), 70.1 on MMB (+2.4), 65.4 on MMB^CN (+1.8), and 87.6 on POPE (+1.7). LLaVA^W is the one benchmark where the 13B variant drops.
• Best among multi-layer fusion baselines on six benchmarks: Against MMFuser, Dense Connector, DeepStack-L, SparseCut, IGVA, and TGIF, DGP attains the best score on GQA, SQA^I, SEED, LLaVA^W, MM-Vet, and MMB^CN (7B setting).
• Generalizes across frameworks: On Qwen3-VL-2B (SigLIP2-L encoder + Qwen3-1.7B), DGP improves GQA from 63.9 to 65.4, SQA^I from 85.3 to 86.9, MMB from 76.4 to 77.4, and MM-Vet from 49.8 to 50.9. On LLaVA-NeXT (CLIP-L + Vicuna-7B with AnyRes tiling), it improves GQA from 64.0 to 64.8, SQA^I from 69.5 to 72.3, MMB from 66.5 to 68.7, and MM-Vet from 33.1 to 34.6.
• The gain comes from routing, not from "more layers": A uniform average over the 24 ViT layers hurts — GQA falls to 60.2, VQAv2 to 72.0, MM-Vet to 25.3, MMB to 57.6, MMB^CN to 44.9, POPE to 77.2. Gated routing alone recovers and surpasses the baseline (GQA 64.1, SQA^I 70.5, MM-Vet 32.4), and adding the entropy regularizer yields the full DGP numbers.
• Layer-dependent routing is non-uniform and multimodal: Middle ViT layers are activated most frequently, but routing curves peak at different positions — LLM layer 1 favors shallow-to-middle ViT layers, LLM layer 20 leans toward shallow features, and LLM layers 22 and 32 peak at mid-deep and deep ViT layers respectively.
• Routing adapts to the task and to the individual sample: At LLM layer 21, LLaVA^W and VizWiz allocate more mass to mid-level features, while SQA^I also puts noticeable weight on shallow ViT layer 2. Within a single benchmark, per-sample routing distributions vary substantially.
• Routing adapts to the question for the same image: Attribute-recognition and small-region questions activate shallow or shallow-to-middle ViT features; global-semantic and abstract-reasoning questions peak at mid-deep or deep ViT layers; open-ended "describe in detail" prompts widen into a broad multimodal distribution.
• Overhead is modest: Compared with LLaVA-1.5-7B (6,759.4M params, 9,168.3 GFLOPs, 15.51 GB peak memory), DGP uses 7,117.3M params (+5.29%), 9,541.2 GFLOPs (+4.07%), and 15.49 GB (−0.13%). For 13B (13,047.3M params, 17,731.2 GFLOPs, 27.34 GB), DGP uses 13,604.9M params (+4.27%), 17,718.8 GFLOPs (−0.07%), and 28.47 GB (+4.13%).
• Qualitative improvement: In case studies on math reasoning, OCR understanding, and fine-grained visual reasoning, LLaVA-1.5 misses critical visual evidence or hallucinates content (for example, misreading menu prices or inventing a "calzone"), while DGP grounds answers in recovered multi-granularity cues.

Methodology in Plain English

The setup is deliberately simple to describe. The vision encoder is run once, and the output of every one of its 24 layers is cached — both the patch tokens and the [CLS] token, which summarizes the whole image at that layer's level of abstraction. The initial input to the LLM still uses the penultimate layer's patches, projected as usual, so the standard pipeline is unchanged at entry.

The novelty is what happens inside each LLM layer. A single learnable "glimpse" token is placed immediately before the first answer token, so that — under causal attention — it can see all preceding visual and textual context. That token's hidden state at layer l becomes a query. The 24 [CLS] summaries (projected into the LLM's hidden space, with layer-ID embeddings added) serve as shared keys and values. A cross-attention module shared across all LLM layers compares the query against every visual-layer summary, and then a layer-specific small MLP turns the result into a softmax distribution over the 24 ViT layers — the routing weights.

Those weights do two things. First, they blend the patch features position by position, so each spatial location gets a fused feature drawn from whichever visual depths that LLM layer prefers. Second, after a layer-specific projector, the fused representation is added back to the LLM's visual token hidden states through a gated residual connection with a learnable scalar passed through a sigmoid, so the model also controls how much visual information is injected at that depth.

Two design details keep this stable. The cross-attention is shared while the routing MLPs are not — the sharing gives every LLM layer the same fair query interface to compare visual layers, while the per-layer MLPs let each layer express its own preference. And an entropy term on the batch-averaged routing distribution (weight λ = 0.01 in pre-training, λ = 0 in fine-tuning) rewards spread-out routing early on, so the distribution does not collapse onto a handful of visual layers before it has had a chance to specialize.

Training follows the LLaVA-1.5 recipe: CLIP-ViT-L/14-336 kept frozen, Vicuna backbone, pre-training on LLaVA-558K and fine-tuning on LLaVA-mixed-665K, AdamW with cosine scheduling, warmup ratio 0.03, batch sizes 256/128, learning rates 1e-3/2e-5, one epoch per stage, a routing MLP hidden dimension of 1024, and a residual gate initialized to 0.3. Unusually for this kind of adapter work, the LLM itself is optimized jointly with the new modules in both stages, because DGP touches hidden states at every layer.

Why This Matters

Impact on research. The paper moves a design decision that is usually made by convention — which ViT layer to tap — into a learned, per-layer, per-sample decision. It also supplies a clean negative result that the field needs: naively averaging or concatenating more visual layers is not automatically better, and can be substantially worse. The routing heatmaps give a diagnostic handle on why a given LLM depth benefits from a given visual granularity, which is rare in multimodal adapter work, and the gains hold across two additional architectures (Qwen3-VL-2B and LLaVA-NeXT), suggesting an architectural principle rather than a LLaVA-specific trick.

Real-world applications:

• Document and menu OCR assistants — the case studies show errors in reading prices and printed text, a failure mode directly relevant to receipts, forms, and signage.
• Accessibility tools for blind and low-vision users, where open-ended real-world scene description is the task; VizWiz, drawn from that use case, improves by 3.1 points on the 7B model.
• Hallucination-sensitive assistants such as POPE-style object-presence checking and detailed image captioning, where grounding in fine-grained visual evidence matters more than fluent prose.
• Scientific and technical figure understanding, where SQA^I improves by 4.2 points and the routing analysis shows shallow layer 2 carrying weight for fine-grained scientific detail.

Industry relevance. The overhead is 5.29% additional parameters and +4.07% FLOPs on the 7B model (with peak memory essentially unchanged at −0.13%), and the 13B model shows −0.07% FLOPs. That is a small price for double-digit relative gains on several benchmarks, which makes the approach plausible as a drop-in modification to an existing LLaVA-style training pipeline rather than a from-scratch architecture. The code is released at https://github.com/YingyingYan1/DGP.

Future Directions

• Video and multi-frame inputs. The paper evaluates only single images; whether routing over visual hierarchies remains meaningful (or becomes redundant) when the temporal axis already supplies multi-granularity information is untested.
• Beyond decoder-only, projector-based MLLMs. The related work separates projector-based designs (LLaVA, Qwen-VL, InternVL) from cross-attention designs (Flamingo), but all experiments are in the former family; transplanting DGP's routing into gated cross-attention layers is an open question.
• Pushing the overhead down further. The 4-5% parameter increase comes from per-layer routing MLPs and projectors; sharing or factorizing these, or pruning to a sparse subset of ViT layers after training, is a natural compression step.
• Using routing as a diagnostic or control signal. Since routing shifts with the question and with the sample, one could imagine auditing MLLM failures by inspecting which visual granularity a wrong answer relied on — but the paper demonstrates the visualization without turning it into a tool.
• Sensitivity to the entropy weight and gate initialization is deferred to the appendix (Appendix C), so the paper's main text does not establish how brittle the λ = 0.01 setting is.

Target Audience

Researchers and engineers working on multimodal LLM architecture, especially those building or fine-tuning LLaVA-style models who have hit a ceiling with single-layer visual injection. It is also useful for readers interested in adaptive computation and routing — the gating mechanism is a clean, well-ablated instance of content-dependent layer selection. Practitioners who need to justify an architecture change with concrete efficiency numbers (parameters, FLOPs, peak memory at batch size 1) will find Table 3 directly useful. It is not an introductory paper: readers should already understand how a ViT layer stack, a decoder-only LLM, and softmax cross-attention work.

Authors’ abstract

A central yet under-explored question in Multimodal Large Language Models (MLLMs) is where to connect---how visual information from a hierarchical vision encoder should be wired into the layer-wise semantics of a large language model. Mainstream MLLMs inject a fixed single-layer visual representation into the LLM and treat all LLM layers as a single consumer, ignoring that different layers demand different visual granularities. Recent multi-layer fusion methods either compress hierarchical features on the encoder side or rely on predefined sparse or hierarchical layer-to-layer connections, and therefore stop short of modeling adaptive cross-layer interactions between the ViT and LLM hierarchies. We address this gap with DGP (Dynamic Gated Pathways), a routing mechanism that establishes all-to-all connections between every ViT layer and every LLM layer, with each LLM layer dynamically gating its preference over visual layers conditioned on its own semantic state. Built on top of LLaVA-1.5-7B and 13B, DGP delivers consistent gains over the LLaVA-1.5-7B baseline (e.g., +4.2 on SQA$^{I}$, +3.3 on MM-Vet, +3.1 on VizWiz and LLaVA$^{W}$) and surpasses prior multi-layer visual fusion methods on most benchmarks. Layer-wise routing analyses further show that different LLM layers do select different visual granularities on demand, supporting both the effectiveness and interpretability of the proposed dynamic gated pathways. The code is available: https://github.com/YingyingYan1/DGP.