PROMISE: Prompt-Attentive Hierarchical Contrastive Learning for Robust Cross-Modal Representation with Missing Modalities

AI summary

Overview

Research area: Multimodal / cross-modal representation learning (computer vision and natural language), specifically robustness to missing modalities.

Technical level: Advanced. The paper assumes familiarity with contrastive learning objectives (NT-Xent), multi-head attention, prompt tuning, and frozen vision-language backbones such as CLIP-ViT-Large-Patch14.

Scope in one sentence: The paper proposes PROMISE, a framework that combines modality-specific prompt pools with a prompt-attention mechanism and a two-level contrastive objective to produce robust image-text representations when large fractions of modality data are missing.

What This Paper Is About

Multimodal models that combine language and images degrade sharply when some inputs are missing, because representations learned from complete data do not match those learned from incomplete data. PROMISE addresses this by generating representations for the absent modality from the modality that is still available, using learned prompts and attention, and then forcing those generated representations to stay semantically consistent with real ones. The goal is a single model that trains and tests directly under high missing rates (the paper's main setting is η = 70%) without any pre-training on complete data.

Key Contributions

1. A new technical paradigm that combines prompt learning with hierarchical contrastive learning to handle robust cross-modal representation learning under high missing rates.
2. A Prompt Attention mechanism that extracts semantic information from available modalities to generate high-quality, semantically consistent representations for missing ones, using modality-specific prompt pools paired with multi-head attention.
3. A dual-level contrastive learning strategy — Fusion-driven Nexus Contrastive Learning (FNCL) for cross-modal semantic consistency and Cohesion-driven Core Contrastive Learning (CCCL) for intra-modal discriminative power — integrated with the prompt learning component.
4. Extensive benchmarking and ablation on four datasets (UPMC Food-101, MM-IMDb, Hateful Memes, N24News), including component-wise ablations, missing-rate sweeps, parameter sensitivity, and t-SNE visualization of the learned representation space.

Main Findings

• Strong results at a 70% missing rate. On UPMC Food-101, PROMISE reaches accuracy of 79.97, 88.10, and 83.22 across the three reported test configurations, versus MPVR's 73.85, 86.09, and 77.49. On MM-IMDb it reaches F1-Macro of 49.62, 54.29, and 52.19, versus MPVR's 39.19, 46.30, and 42.41. On Hateful Memes it reaches AUROC of 63.63, 64.40, and 67.16, versus MPVR's 61.01, 62.34, and 63.53.
• Largest gains in the balanced-missing setting. The paper reports that PROMISE's biggest improvements over state-of-the-art methods occur in the balanced scenario, where each modality is missing 35% of its data.
• Modality importance differs by dataset. PROMISE benefits most from balanced modality information on Hateful Memes, while on UPMC Food-101 and MM-IMDb it performs better when the text modality is complete — the authors read this as text playing a crucial role in those tasks.
• All three components are needed. In the component ablation, the full configuration scores 66.56 F1 and 65.47 ACC on Hateful Memes and 68.53 F1 and 68.89 ACC on N24News. Configurations with components removed fall as low as 56.50 F1 / 56.89 ACC on Hateful Memes and 63.69 F1 / 64.01 ACC on N24News. Adding either FNCL or CCCL alone yields only a modest increase; the synergy appears only when both are used together.
• Different contrastive levels matter in different missing patterns. In missing-image and missing-text cases, the CCCL-only model clearly outperforms the MPVR baseline, while the FNCL-only model performs slightly below the full model — showing that inter-modal consistency becomes especially important when a whole modality is absent. In the missing-both case, the full model beats FNCL-only, which beats CCCL-only, and all exceed the baseline; the gap between CCCL-only and FNCL-only widens compared with single-modality missing cases.
• Parameter sensitivity is non-monotonic. Across 1 to 8 layers and prompt dimensions from 16 to 128 on Hateful Memes (70% missing rate in training and testing), the best configuration is 6 layers with prompt dimension 16, reaching 67.17%. A 3-layer model with dimension 16 reaches a near-optimal 66.46%. Moderate dimensions such as 32 or 48 also perform strongly, while performance typically declines beyond 48; 1- and 2-layer models generally underperform, and 7-8 layer models give variable results.
• t-SNE shows recovered cluster structure. On four N24News categories (Fashion & Style, Theatre, Food, Health), PROMISE with only 30% text plus 100% image data achieves embedding separation comparable to the upper bound of complete modalities (100% image and 100% text), and clearly better than the image-only lower bound.
• Robustness across missing rates from 10% to 100%. On N24News, Hateful Memes, and UPMC Food101, PROMISE consistently outperforms MPVR across the full missing-rate sweep, and it maintains this advantage as the proportion of missing data grows.

Methodology in Plain English

Authors’ abstract

Multimodal models integrating natural language and visual information have substantially improved generalization of representation models. However, their effectiveness significantly declines in real-world situations where certain modalities are missing or unavailable. This degradation primarily stems from inconsistent representation learning between complete multimodal data and incomplete modality scenarios. Existing approaches typically address missing modalities through relatively simplistic generation methods, yet these approaches fail to adequately preserve cross-modal consistency, leading to suboptimal performance. To overcome this limitation, we propose a novel multimodal framework named PROMISE, a PROMpting-Attentive HIerarchical ContraStive LEarning approach designed explicitly for robust cross-modal representation under conditions of missing modalities. Specifically, PROMISE innovatively incorporates multimodal prompt learning into a hierarchical contrastive learning framework, equipped with a specially designed prompt-attention mechanism. This mechanism dynamically generates robust and consistent representations for scenarios where particular modalities are absent, thereby effectively bridging the representational gap between complete and incomplete data. Extensive experiments conducted on benchmark datasets, along with comprehensive ablation studies, clearly demonstrate the superior performance of PROMISE compared to current state-of-the-art multimodal methods.