Multimodal Fusion and Cross-Attention Architectures

Fusion location controls how early one modality can change the representation of another.

The third pattern has a controlled comparison behind it. Force all cross-modal exchange through a handful of latent units — B = 4 of them, in the audio-visual model — and fusion gets better, not merely cheaper. The narrow channel requires “the model to collate and condense the most relevant information in each modality and only share what is necessary”. Nagrani and five colleagues introduced these attention bottlenecks in 2021. The restriction “improves fusion performance, at the same time reducing computational cost”, they report, with state-of-the-art results on Audioset, Epic-Kitchens and VGGSound.

Both halves of that sentence carry a number. The paper's introduction gives the first: “On the Audioset dataset, we outperform the current state of the art by 5.9 mAP (12.7% relative improvement).” The second came from Google Research. The design cut FLOPs by 50% against a vanilla multimodal transformer.

The width of the channel then mattered far less than its existence. Sweeping the bottleneck across 4, 36, 64, 256 and 1024 tokens moved accuracy by less than 0.5 mAP. At the earliest fusion layer (Lf = 0), bottleneck fusion beat vanilla cross-attention by over 2 mAP at less than half the compute. Four tokens were enough, and a thousand were not better. Restricting the channel improved the fusion. It did not merely make it cheaper.

Vision, audio, text, and sensor encoders produce states with different scales, resolutions, and meanings. A bridge projects or resamples those states into a form another module can consume.

The bridge can become the true bottleneck. Its token count, normalization, temporal aggregation, and training objective determine which modality details survive.

Flamingo fixes that token count and publishes the number. A frozen vision encoder reaches a frozen 70B-parameter language model through a Perceiver Resampler. In the paper’s words: “It takes as input a variable number of image or video features from the vision encoder and produces a fixed number of visual outputs (64), reducing the computational complexity of the vision-text cross-attention.” Whatever the resolution, however long the video, exactly 64 tokens arrive. What the bridge drops, the 70B backbone behind it never sees. DeepMind published the design in 2022. With that bridge the 80B model beat the fine-tuned state of the art on 6 of the 16 benchmarks tested, from only 32 task-specific examples.

The bridge is also the part a reproduction has to get right. A different group built OpenFlamingo in 2023. It reuses the same trainable Perceiver resampler over a frozen CLIP ViT-L/14, and reaches 80-89% of the corresponding Flamingo performance on seven vision-language datasets. The encoders changed. The interface did not.

Cross-attention is not inherently symmetric. The query stream determines what is requested.

The delicate step is the last one, where retrieved evidence re-enters the query stream. Flamingo injects its 64 resampled visual tokens into the frozen language model through GATED XATTN-DENSE blocks, inserted between the frozen LM layers. Queries come from the text stream. Keys and values come from the visual stream. Each block is gated by a tanh gate initialised at 0, so at initialisation the conditioned model reproduces the original language model exactly.

The paper's own ablation prices that choice. Remove the 0-initialised tanh gating and training destabilises: “Without it, we see a drop of 4.2% in our overall score.”

How retrieved evidence is merged back into the residual stream is load-bearing, and the cost has been measured. It is not plumbing. The design was reproduced in the open. Hugging Face built the same interleaved cross-attention into IDEFICS, an open-access model whose name expands to “Image-aware Decoder Enhanced à la Flamingo with Interleaved Cross-attentionS”.

The network can only fuse evidence that has been represented and synchronized appropriately.

Broadcasting has already measured how much audio-video misalignment a person notices. The tolerance is an order of magnitude tighter than intuition suggests. Recommendation ITU-R BT.1359-1, from 1998, records “that subjective evaluations show that detectability thresholds are about +45 ms to –125 ms and acceptability thresholds are about +90 ms to –185 ms on the average, a positive value indicates that sound is advanced with respect to vision”. It recommends an overall tolerance not exceeding +90 ms or -185 ms. EBU Recommendation R37-2007 is tighter still. Each stage may run 5 ms early to 15 ms late. At any output intended for emission, sound before picture must stay within 40 ms and sound after picture within 60 ms.

A one-second offset is not a marginal error against those numbers. It is roughly twenty times the threshold at which an average viewer detects that something is wrong.

Design tests around examples whose answer changes when one modality changes.

One such test already exists, built and scored. Winoground is 400 hand-curated examples — 800 unique captions and images, forming 1,600 image-text pairings. The two captions of a pair contain an identical set of words in a different order. The words cannot separate them. Only the image can decide which caption is correct.

Crowdworkers scored 89.50 text / 88.50 image / 85.50 group. Random chance is 25.00 / 25.00 / 16.67. The best model tested, VinVL, managed 37.75 / 17.75 / 14.50. Every model evaluated came in below chance on the group score.

The diagnosis matters as much as the score. Diwan and four colleagues at UT Austin re-analysed those failures independently in 2022, and located them exactly where this section tells you to look: “Our analysis suggests that a main challenge in visuolinguistic models may lie in fusing visual and textual representations, rather than in compositional language understanding.” The conflict cases and counterfactual pairs below are how you reach that verdict on your own system.

Perceiver-style architectures use a fixed or bounded latent array as queries into a much larger input collection. Repeated latent processing then concentrates computation on the smaller internal workspace. The approach supports mixed modalities and very long raw inputs, but the cross-attention bottleneck can discard rare detail. Latent count, query design, and readout queries require task-specific ablation.

The reference design is the Perceiver itself. Jaegle and five colleagues published it in 2021. It uses “an asymmetric attention mechanism to iteratively distill inputs into a tight latent bottleneck, allowing it to scale to handle very large inputs”. The image model “obtains performance comparable to ResNet-50 and ViT on ImageNet without 2D convolutions by directly attending to 50,000 pixels”. The workspace is bounded on purpose. The input is not.

A product may receive no audio, a broken sensor, unreadable text, or a delayed image. Zero-filling can look like a valid observation unless missingness is represented separately.

The cost of leaving that policy undesigned has been measured. Show a ViLT-based model only 30% of the text at test time and it loses 43.6% on MM-IMDb, 28.3% on UPMC Food-101 and 14.2% on Hateful Memes. Under severe missingness it scored worse on MM-IMDb and UPMC Food-101 than the unimodal model it was meant to improve on. Ma and four colleagues published those numbers in 2022, and their abstract is blunt: “Unsurprising, we find Transformer models are sensitive to missing modalities while different modal fusion strategies will significantly affect the robustness.” They found no universal answer either. The best fusion strategy is dataset-dependent. A separate group at UC Riverside and the Air Force Research Laboratory reported the same fragility in 2024, across five tasks and seven datasets.

Options include modality dropout during training, learned missing tokens, independent fallback heads, or conditional routing. The output should expose reduced evidence when the absent stream matters.