Rethinking Projector Training in Multimodal LLMs

AI summary

Overview

• Research area: Multimodal large language models (vision-language models), specifically the projector/connector alignment stage that links a frozen vision encoder to a frozen LLM.
• Technical level: Intermediate. The motivation and empirical findings are accessible, but the proposed method relies on Gaussian optimal transport, PCA subspaces, and Bures-Wasserstein geometry.
• Scope: The paper empirically re-examines what projector alignment actually accomplishes and proposes PORTAL, a training-free closed-form projector built only from per-modality summary statistics, evaluated across seven (vision encoder, LLM) pairs and 16 benchmarks.

What This Paper Is About

Standard multimodal LLM pipelines (LLaVA, Molmo, InternVL, QwenVL) first train a projector on large image-caption datasets to align a vision encoder with a frozen language model, then jointly train the full model. The role of that first stage is poorly understood, and prior work disagrees about it: Prismatic reports it can be skipped, Cambrian-1 reports that scaling its data to 2.5M pairs helps, and MM1 argues resolution and data mixing matter more than the projector itself. This paper argues that projector alignment is only a coarse, geometric conditioning step with a broad solution space, and shows that it can be replaced outright by a closed-form construction requiring no paired captions and no gradient steps.

Key Contributions

1. An empirical characterization of projector alignment as coarse alignment. The authors show that using only 10 to 20% of the LCS-558k image-caption corpus recovers most downstream gains, and that projectors trained on entirely different datasets can be linearly interpolated while largely preserving performance.
2. PORTAL (Projector via Optimal-transport ALignment). A training-free, closed-form projector initialization computed as the Bures-Wasserstein optimal transport map between Gaussian approximations of vision and language feature distributions on a shared top-k principal subspace. It needs only per-modality means, principal directions, and spread along each direction.
3. A cross-backbone and cross-encoder evaluation. PORTAL is tested across six LLM backbones from 0.5B to 8B parameters (Vicuna 1.5 7B, LLaMA 3.1 8B, Qwen 2.5 7B, LLaMA 3.2 1B, Qwen 2.5 0.5B, OLMo 2 1B) and two vision encoders (CLIP ViT-L/14-336, SigLIP-SO400M) on 16 benchmarks, against both a 558k-trained baseline and a random-initialization baseline.
4. A theory sketch for why moment matching suffices. The paper provides Proposition 1, a spectrum-to-ω bound showing that when token norm spread is small, the dispersion of token directions is fixed by the covariance alone, which PORTAL matches.

Main Findings

• Alignment data saturates early. On the LLaVA 1.5 setup with CLIP ViT-L/14-336, Vicuna 1.5 7B, and a 2-layer MLP projector (with MobileLLaMA 1.4B substituted for the data-fraction sweep), only 10 to 20% of LCS-558k captures most of the downstream benefit, and all four Cambrian-1 categories plateau by 30 to 40%. ShareGPT-1.2M, more than twice as large, does not do better: its average peaks at 38.1 with 40% of its data.
• Projectors trained on different data are linearly interpolable. Interpolating W_α = (1−α)W_LCS + αW_ShareGPT at α ∈ {0, 0.25, 0.5, 0.75, 1} largely preserves performance, and every interior blend outperforms both endpoints on the aggregate score, peaking at α = 0.5. Category optima differ: OCR peaks closer to the ShareGPT endpoint, Knowledge closer to the LCS endpoint.
• Initialization, not data, selects the loss basin. A second pair of projectors trained on the same LCS-558k data from two different random initializations is not linearly interpolable: captioning CIDEr drops from about 80 at both endpoints to 0.7 at α = 0.5.
• PORTAL matches the trained baseline without paired data. Across the seven settings, PORTAL exceeds the trained baseline in three: LLaMA 3.2 1B/CLIP reaches 106.2%, Qwen 2.5 0.5B/CLIP reaches 106.1%, and LLaMA 3.1 8B/CLIP reaches 102.7%. It stays close on the rest: Vicuna 1.5 7B/CLIP 98.8%, Vicuna 1.5 7B/SigLIP 99.3%, Qwen 2.5 7B/CLIP 98.3%, and OLMo 2 1B/CLIP 96.3%.
• PORTAL closes the collapse of a random projector. On SigLIP, the Random baseline falls to 82.5% of the trained projector while PORTAL reaches 99.3%.
• Losses concentrate on OCR-heavy benchmarks. The two settings below the trained baseline's bootstrap confidence interval, Qwen 2.5 7B/CLIP and OLMo 2 1B/CLIP, lose most on OCR and Chart tasks — exactly the categories that drive most of the Stage-I gain over no pretraining, suggesting paired captions teach character-level spatial structure that marginal statistics cannot recover. Category-level, PORTAL retains on average 100.7% (Knowledge), 102.0% (General), 102.4% (OCR), and 100.4% (Vision-Centric) of the trained baseline across all seven configurations.
• PORTAL is not uniquely dependent on its unpaired approach. Other unpaired distribution-matching methods, including CORAL and Gromov-Wasserstein alignment, also recover most of the performance of paired Stage-I training.
• Alignment compute drops sharply. Used as a warm start on Vicuna 1.5 7B/CLIP, PORTAL reaches the 558k-trained baseline at roughly 4k captions, about 135× fewer projector alignment FLOPs. With as few as 10k paired captions it exceeds the full 558k-trained baseline and widens the gap up to 100k.
• PORTAL benefits a simpler projector more. On LLaMA 3.2 1B/CLIP with a single linear layer instead of the 2-layer MLP, PORTAL reaches 111.8% of the trained baseline against Random's 94.2%, widening the gap from 6.3 to 17.6 percentage points. With the C-Abstractor, which has approximately 19× more parameters than the 2-layer MLP, PORTAL is essentially indistinguishable from conventional Stage-I training (42.93 vs. 42.98 average).
• The image pool and rank are not critical. A small COCO val2017 pool of 1024 images matches or exceeds a Cambrian-737k subset spanning 9 sources with 4096 images on 12 of 15 benchmarks (44.0 vs. 43.4 average). Varying k over {64, 128, 256, 512, 1024} shows every value beating the trained baseline by at least about 4%: flat at roughly 43.3 from k = 64 to k = 256, peaking at k = 512 (44.2), and dipping marginally at k = 1024 (44.0).
• The embedding table is the best target. On LLaMA 3.1 8B/CLIP, targeting input embeddings (ℓ = 0) reaches 102.3% of the trained baseline (Avg 52.5), ahead of ℓ = 4 at 101.9% (52.3) and ℓ = 8 at 101.4% (52.0), with Random at 100.2% (51.4) and trained at 51.3.
• Directional dispersion moves toward the target. On three (CLIP, LLM) pairs, PORTAL reduces cosine dispersion ω from 0.23 at the source to 0.09–0.10, toward the target's 0.05–0.06, while a random projector leaves ω at the source value.
• Robustness beyond the default setting. Under larger-scale Stage-II instruction tuning with 3.2M samples, PORTAL slightly outperforms the conventionally Stage-I-trained initialization, and on the POPE object hallucination benchmark it matches or exceeds the trained Stage-I baseline in every setting.

Methodology in Plain English

The authors start by auditing the alignment stage rather than trying to improve it. They train projectors on progressively smaller fractions of image-caption data and measure what downstream performance does. They then train projectors on two different corpora from the same initialization and blend their weights, and separately train two projectors from different random seeds, to see whether the resulting weights can be mixed without losing performance.

From these observations they build a replacement. Instead of learning the projector from image-caption pairs, they treat each modality as a roughly Gaussian cloud of features. They compute the mean, the top-k principal directions, and the spread along each of those directions — separately for vision features and for LLM input embeddings. They then write down the closed-form optimal transport map between these two Gaussian approximations, restricted to the shared principal subspace. Plugged into the standard LLaVA 2-layer MLP form, this gives W_1 and b_1 directly, with the second layer fixed to the identity and its bias to zero. The result is a projector computed with a single PCA per side, with no captions and no gradient descent.

To justify why matching only means and covariances is enough, the authors note that LLMs normalize tokens before attention, so attention depends on token directions rather than lengths. They prove a bound showing that when all tokens have similar norms, the variance of the cosine similarity between two tokens is determined by the covariance through the effective dimension. Finally, they insert PORTAL projectors into the standard LLaVA pipeline and run identical LLaVA-665k Stage-II instruction tuning for every configuration, so that only the initialization differs.

Why This Matters

The paper separates two things that the field has often conflated: making visual features distributionally compatible with an LLM's input space, and learning fine-grained image-text correspondences. If the first stage is mostly the former, then a large amount of caption data and compute is being spent on a step with a closed-form substitute.

Recommended impact on research:

• It reframes conflicting results in prior work (Prismatic, Cambrian-1, MM1) as a single phenomenon rather than a disagreement.
• It supplies a controlled baseline — random, PORTAL, and trained projectors under identical Stage-II tuning — that future connector papers can compare against.
• It ties multimodal alignment to the classical optimal transport and second-order statistics literature (CORAL, Bures-Wasserstein, Gromov-Wasserstein), giving the alignment stage a clearer theoretical handle.
• It suggests the OCR and Chart gap is the specific frontier where paired supervision still earns its cost.

Real-world applications implied by the work:

• Cheaper construction of multimodal assistants and document-understanding models where the alignment stage's caption corpus and GPU time are a bottleneck.
• Rapid adaptation of a frozen vision encoder to a new or swapped LLM backbone when no aligned image-caption dataset exists for that pairing.
• On-device or privacy-sensitive deployments, since source statistics can be estimated from a small unlabeled image pool (1024 COCO images sufficed) rather than a curated caption corpus.
• Lower-cost warm starting of existing training pipelines, reaching baseline quality at roughly 4k captions instead of 558k.

Industry relevance: teams training vision-language models spend substantial data-collection and compute budget on projector alignment. A closed-form initialization that matches a 558k-caption-trained baseline, reduces alignment FLOPs by roughly 135×, and works across LLM sizes from 0.5B to 8B and across CLIP and SigLIP encoders suggests a practical cost reduction for anyone assembling a multimodal pipeline, particularly where a paired caption corpus is unavailable or expensive to license.

Future Directions

• Understanding where marginal statistics fail. The paper leaves unanswered exactly which aspects of OCR and Chart tasks paired supervision captures that covariance matching does not; the authors identify this as the regime needing further study.
• Richer connectors and targets. The study is limited to simple projector architectures and single-image inputs with frozen backbones during alignment; extending it to richer connectors, long-context grounding, and multi-image reasoning is proposed.
• Additional modalities. Whether the coarse-alignment view holds for audio, video, and 3D inputs is explicitly listed as open, along with the paper's own noted limitations in Appendix A.
• Reconciling initialization and data. The cross-seed interpolation failure (CIDEr falling from about 80 to 0.7 at α = 0.5) shows initialization selects the loss basin while training data selects the solution within it; how to exploit that structure — for instance, whether PORTAL's closed-form solution lies in a more favorable basin — is left open.

Target Audience

Researchers and engineers working on multimodal LLM training pipelines, especially those responsible for vision-language alignment stages and connector design. The paper is most valuable to readers already familiar with the LLaVA-style two-stage recipe, since it presupposes knowledge of projector alignment, instruction tuning, and benchmarks such as GQA, MMMU, DocVQA, and ChartQA. Readers interested in optimal transport, training-free adaptation, and compute-efficient training will also find the closed-form construction directly useful, particularly given the self-contained PyTorch implementation in Appendix F.

Authors’ abstract

Multimodal LLMs are typically trained by first aligning a vision encoder with a language model via projector training on large image-caption datasets, followed by joint training of the full multimodal model. Despite its widespread use, the role of the projector training stage remains poorly understood. We find that (i) using only $10$ to $20\%$ of the data for training the projector already recovers most downstream performance gains, and (ii) projectors trained on different datasets can be linearly interpolated while largely preserving performance. These results suggest that projector training mainly performs a coarse alignment, which many different projectors achieve equally well. Motivated by these findings, we propose PORTAL, a training-free projector initialization that requires neither paired image-caption data nor gradient-based optimization, but only per-modality summary statistics. PORTAL computes an optimal-transport-based map between Gaussian approximations of the vision and language feature distributions on a shared principal subspace, so that the projector training stage can be skipped entirely. Without any paired data, PORTAL matches the standard pipeline trained on 558k image-caption pairs across six LLM backbones, two vision encoders, and 16 benchmarks. It outperforms that pipeline in $3$ of $7$ (vision encoder, LLM) settings and reaches an average $98\%$ of its score across the other four.