Multimodal Foundation Models

The cheapest way to look multimodal is to accept two inputs and answer from one of them. A model that never sees the image at all answers 48.21% of the questions in the original VQA dataset correctly. Goyal, Khot, Summers-Stay, Batra and Parikh measured that language-only baseline in 2017.

That number is the floor a visual question answering system has to beat before any of its accuracy can be credited to vision. Their response was not a warning but a dataset. VQA v2 is balanced: more than 443K train, 214K val and 453K test (question, image) pairs. For every question they collected a complementary image that yields a different answer. When the same question has two answers depending on which picture accompanies it, the language shortcut stops paying.

Architecture determines where modalities can interact and which computations can be reused. Independent encoders, a shared embedding space, cross-modal fusion and unified generation are four places to put the join. External perception tools are the fifth, and the one most often treated as a stopgap rather than a design choice.

It is worth seeing what a specialised perception component actually buys. Whisper was trained on 680,000 hours of weakly supervised audio: 117,000 hours of it covering 96 other languages, and 125,000 hours of X-to-English translation data. Radford and colleagues published it in 2023. They matched it against wav2vec 2.0 Large (no LM), a supervised LibriSpeech model that comes within 0.1% of it on LibriSpeech test-clean. On the thirteen other speech recognition datasets in their Table 1 — Artie, Common Voice, Fleurs En, TED-LIUM, CHiME6, VoxPopuli En, CORAAL, AMI IHM, Switchboard, CallHome, WSJ, AMI SDM1 and LibriSpeech Other — the zero-shot model made 55.2% fewer errors on average, with per-dataset relative error reductions from 16.1% to 74.7%. The paper puts it in one sentence: “Despite their very close performance on the reference distribution, the zero-shot Whisper model achieves an average relative error reduction of 55.2% when evaluated on other speech recognition datasets.”

Two systems that tie on the reference distribution can differ by half their errors everywhere else. That gap is an argument for wiring a dedicated perception component in beside a unified model, rather than assuming the unified model has absorbed it.

Each pattern trades retrieval speed against how deeply the modalities interact and how freely the model can generate. A dual encoder encodes each modality separately and compares vectors. That makes embeddings cacheable and large collections searchable, at the cost of fine-grained interaction. A unified token model puts several modalities in one sequence or decoder. It buys flexible mixed-modal generation, and it inherits the tokenization, serving and modality-imbalance problems that come with it.

The middle pattern — fusion or cross-attention — has a well-documented instance. Flamingo conditions a frozen language model on vision by inserting trainable GATED XATTN-DENSE cross-attention blocks: every fourth layer for Flamingo-9B, every seventh for Flamingo-80B. The tanh gating is initialised at 0, so the pretrained language model starts out unchanged and the visual pathway has to earn its influence during training. Alayrac and colleagues state the payoff precisely: “On 6 of these 16 tasks, Flamingo also outperforms the fine-tuned state of the art despite using only 32 task-specific examples, around 1000 times less task-specific training data than the current state of the art.”

Note what that schedule implies. Cross-attention is not sprinkled everywhere. It is inserted at a chosen interval, gated shut at initialisation, and the resulting cost is paid per pair rather than once per item. That is the real shape of the tradeoff a dual encoder declines to make.

Paired captions, transcripts, demonstrations, and contrastive examples teach correspondence between modalities. The pairs may be noisy, incomplete, culturally narrow, or dominated by obvious associations. At frontier scale the noise is deliberate. ALIGN trained a simple dual encoder with a contrastive loss on 1.8 billion image alt-text pairs, kept intentionally dirty, with only minimal frequency-based filtering. Jia and colleagues say why in one line: “Here, for the purpose of scaling, we trade quality for scale by relaxing most of the cleaning steps in the original work.” The trade worked on the metrics it was aimed at. ALIGN reached 76.4% top-1 zero-shot accuracy on ImageNet, and beat the previous state of the art by more than 7% on most zero-shot and fine-tuned R@1 metrics on Flickr30K and MSCOCO.

CLIP was trained on “a dataset of 400 million (image, text) pairs collected from the internet”. The entire pre-training objective was to predict which caption goes with which image. Benchmarked on over thirty existing vision datasets, it transferred without dataset-specific training. On ImageNet it matched the accuracy of the original ResNet-50 zero-shot, using none of that model's 1.28 million labeled examples.

Then ask what all that correspondence knows about word order. Winoground is 400 hand-curated examples: 800 unique captions and images, 1600 image-text pairs, 800 correct and 800 incorrect pairings. Both captions of a pair contain a completely identical set of words in a different order. MTurk humans score 89.50 text / 88.50 image / 85.50 group. Random chance is 25.00 / 25.00 / 16.67. The strongest model, VinVL, reaches 37.75 text / 17.75 image / 14.50 group — top of the table on the image and group metrics, and still below the 16.67 group chance baseline. UNITER-large edges it on text alone, at 38.00. Thrush and colleagues are blunt about the sweep: “We probe a diverse range of state-of-the-art vision and language models and find that, surprisingly, none of them do much better than chance.”

Billions of pairs buy association. On 400 items where the words are held fixed and only their arrangement changes, that association does not survive.

Use interventions that reveal whether each modality contributes as intended. Establish unimodal baselines on the same split and task. Create counterfactual pairs by swapping images, captions, audio or frames while preserving superficial format. Test missing inputs. Inspect localization by requiring supporting regions, timestamps, spans or retrieved evidence. Slice by modality quality across blur, noise, accents, compression, language and device conditions. Compare end-to-end outcomes rather than an aggregate benchmark.

Missing inputs are not a corner case, and the cost of leaving fallback behavior undefined has been measured. Ma and colleagues tested Transformer multimodal models with only 30% of text available at test time. Early fusion fell from 55.3 to 31.2 on MM-IMDb, from 91.9 to 65.9 on UPMC Food-101 and from 70.2 to 60.2 on Hateful Memes. Late fusion fell from 54.9 to 31.0, from 91.8 to 69.1 and from 64.5 to 57.8. Read those pairs across datasets and the ranking of the two fusion strategies flips: “What surprised us is that the optimal fusion strategy is dataset dependent even for the same Transformer model; there does not exist a universal strategy that works in general cases.” There is no architecture you can pick once to be safe here. Degraded-mode behavior has to be measured on your data.

The last step — comparing end-to-end outcomes rather than aggregate scores — has its own reference point. MMMU is 11.5K college-level multimodal questions covering six disciplines, 30 subjects, 183 subfields and 30 highly heterogeneous image types, from charts and diagrams to maps, tables, music sheets and chemical structures. Across 28 open-source LMMs plus the proprietary GPT-4V and Gemini, the headline result is modest: “Even the advanced GPT-4V and Gemini Ultra only achieve accuracies of 56% and 59% respectively, indicating significant room for improvement.” A system that fusion has genuinely improved should show it on the decision you care about, not only on a number in that range.

Multiple modalities can resolve ambiguity, support richer interfaces, and enable new forms of generation. They also add obligations: to align the modalities, to keep them in step, to store them, to evaluate them, and to protect what they reveal about people.

That last obligation is concrete. Deep learning models predict a patient's self-reported race from medical images at AUC 0.91-0.99 on x-ray, 0.87-0.96 on chest CT and 0.81 on mammography. Gichoya, Banerjee, Bhimireddy and colleagues reported that in The Lancet Digital Health in 2022. The obvious explanations do not hold. Body-mass index reaches AUC 0.55, disease distribution 0.61, breast density 0.61. The signal survived image corruption, cropping and noising, across all anatomical regions and frequency bands. Their interpretation states the consequence: “However, our finding that AI can accurately predict self-reported race, even from corrupted, cropped, and noised medical images, often when clinical experts cannot, creates an enormous risk for all model deployments in medical imaging.”

A modality can carry an attribute that the experts supervising the model cannot see in it. The supervision therefore cannot catch the model using it.

The next lesson focuses on generated and edited media, where realism can be mistaken for evidence and provenance must be handled separately from visual quality.