Multimodal Item Understanding

A classifier learned to recognise horses by reading a printed mark of provenance. Not in a marketplace — in a benchmark. The images came from PASCAL VOC 2007, 9,963 of them, and about a fifth of the horse pictures carried a source tag. A Fisher-vector model found the tag and used it. In 2019 a paper in Nature Communications asked what those horse predictions actually rested on, and answered.

“The Fisher vector classifier trained on the PASCAL VOC 2007 data set focuses on a source tag present in about one-fifth of the horse figures. Removing the tag also removes the ability to classify the picture as a horse. Furthermore, inserting the tag on a car image changes the classification from car to horse.” — Lapuschkin and colleagues, 2019

A correctly classified image of a Ferrari became a horse because the tag was pasted onto it.

Everything a recommender does with product photographs sits downstream of that. The tag is a fact about who published the picture, not about what is in it. The classifier could not tell the two apart.

Encoders first, one per modality. Then a decision about how to combine them.

What the stack has to survive is a missing modality: the listing with no photograph, the video with no transcript. The cost of not surviving it has been measured. A 2022 study ran a ViLT backbone on MM-IMDb. With both modalities complete it scored 55.3 F1-Macro. With only 30% of text available at test time it scored 31.2. The image-only model scored 35.0.

The multimodal system fell below its own single-modality baseline the moment one input thinned out. That is why masks and fallback encoders sit inside the stack, not around it.

Text, image, audio, video and structured attributes can improve new-item retrieval and fine-grained similarity. Encoders may be trained separately, jointly, or aligned in a shared space. Each modality captures item properties and production artifacts at the same time. Style, camera, language, provider budget, watermark, platform formatting — any of these can become an unintended recommendation feature.

The watermark on that list is not a hypothetical. The source tag that carried a Fisher-vector classifier's horse predictions was exactly that: a mark of who published the image, learned as though it were a mark of what the image contained. It was an artifact of the dataset. Nobody noticed until someone asked what the classifier was using.

Style bias rewards whoever can afford better production. Unsafe semantic inference reads attributes out of media that the media does not carry. Both start in the same place: an encoder trusted to have understood the item rather than described it. One of the two now has a statutory boundary drawn around it. The other has a published size.

Where the modalities meet decides what breaks when one of them is missing. Early fusion allows the richest interaction and tolerates a missing input least. Late fusion is modular, easy to ablate, and gives up the fine interactions. Gated fusion learns which modality to trust, and can learn a shortcut instead. Two published tables put numbers under that paragraph.

The first is about whether to add a modality at all. Facebook AI compared late-fusion multimodal networks against their best single modality on Kinetics top-1 validation accuracy. Audio plus RGB scored 71.4 against RGB alone at 72.6, a loss of 1.2. Audio plus optical flow scored 58.3 against optical flow alone at 62.1, a loss of 3.8. Gating rescued none of it: an SE-gate reached 71.4 and an NL-gate 72.0, both still below the 72.6 unimodal baseline. Their 2020 abstract says it plainly: “In our experiments, however, we observe the opposite: the best uni-modal network often outperforms the multi-modal network.”

The ranking between the arrangements is not stable either. In the 2022 missing-modality study, with only 30% of text available, early fusion won on MM-IMDb, 31.2 against 31.0, and on Hateful Memes, 60.2 against 57.8. Late fusion won on UPMC Food-101, 69.1 against 65.9. The authors were blunt: “What surprised us is that the optimal fusion strategy is dataset dependent even for the same Transformer model; there does not exist a universal strategy that works in general cases.”

A fusion choice is a result to be measured on this catalogue. It is not a design principle to be imported.

Modality baselines come first, one per source. Text only, image only, structured attributes, collaborative signal.

Step four is the one teams skip, because documenting pretraining exposure sounds like a paragraph rather than a measurement. It can be done as a measurement, and there is a worked example. In 2021 CLIP's authors ran a de-duplication analysis between the 400-million-pair internet set the model was pretrained on and its downstream evaluations. Of 35 datasets studied, 9 had no detected overlap at all. “There is a median overlap of 2.2% and an average overlap of 3.2%.” Accuracy shifted by more than 0.1% on only 7 datasets. Just 2 of those were statistically significant after Bonferroni correction, and the largest detected improvement was 0.6% on Birdsnap.

A method, a coverage count and a bounded effect. That is what a documented exposure limit looks like, and it is reproducible on a catalogue.

Drift monitoring closes the sequence. Encoders, preprocessing, catalog and creative all move underneath a model that was frozen months earlier.