Inference-Time Dynamic Modality Selection for Incomplete Multimodal Classification

AI summary

Overview

Research area: Multimodal deep learning (MDL) and multimodal fusion, with applications spanning natural image classification, marketing data, and medical imaging (cardiac disease classification). The paper sits at the intersection of incomplete-modality learning, dynamic neural networks, and information theory.

Technical level: Advanced. The method rests on a mutual-information lower bound, class-prototype density estimation, Bregman-divergence distances, and a transformer-based multimodal architecture. The paper is written for readers comfortable with these tools.

One-sentence scope: The paper proposes DyMo, an inference-time framework that dynamically decides which recovered (imputed) modalities to fuse for each individual test sample, in order to resolve the trade-off between discarding missing modalities and imputing them.

What This Paper Is About

Multimodal models normally assume every modality is present at test time, but real data frequently has missing modalities due to different collection protocols across centers, sensor failures, or transmission errors. Existing fixes fall into two camps: recovery-based methods impute the missing modalities (which can inject blurry, corrupted, or semantically misaligned content), and recovery-free methods simply ignore them (which throws away task-relevant signal). The authors call this the discarding-imputation dilemma and aim to move beyond it by selecting, per sample at inference time, only the recovered modalities that actually add task-relevant information.

Key Contributions

1. Framing the discarding-imputation dilemma. The authors state they are the first to investigate this dilemma in incomplete MDL and to introduce dynamic neural networks to address it.
2. DyMo, an inference-time dynamic modality fusion framework. It adaptively fuses recovered modalities using a selection algorithm formulated on multimodal task-relevant information gain, rather than discarding or blindly imputing.
3. A theoretically grounded reward function (MTIR). Because the true data distribution is unknown at inference, the authors derive a lower bound connecting mutual information I(Y;Z) to the test-time cross-entropy loss, and use loss decrease as a tractable proxy for information gain. An intra-class similarity (ICS) calibration term refines the reward, using an asymmetric α that is capped at 1 to keep the model conservative toward synthetic modalities.
4. A flexible multimodal network and a tailored training strategy. The architecture predicts from arbitrary modality combinations (dummy tokens fill missing positions, attention masks prevent distortion), and training samples random modality subsets with an auxiliary missing-agnostic contrastive loss.

Main Findings

• Large gains under severe missingness. DyMo achieves 13.12% higher accuracy on PolyMNIST with 80% missing modalities and 4.11% higher accuracy on DVM when the full table is missing, compared with static and dynamic fusion SOTAs (CONCAT, QMF, DynMM, PDF).
• Consistent improvement over incomplete MDL SOTAs. Against recovery-based methods (MultiAE, MoPoE, M3Care, OnlineMAE, CMVAE) and recovery-free methods (ModDrop, MTL, MAP, MUSE), DyMo reports 5.67% higher accuracy on PolyMNIST with 80% missing modalities and 1.97% higher AUC on Infarction with 70% missing tabular features.
• The dilemma is empirically visible. MUSE drops 61.18% in accuracy on MST with missing {M,T} versus {S,T}, illustrating the cost of discarding highly task-relevant modalities. OnlineMAE's accuracy on PolyMNIST decreases by 9.91% at η = 0.8 versus η = 0, illustrating unreliable recovery.
• Prior dynamic fusion methods struggle where recovery is semantically wrong. They outperform static fusion on DVM but achieve limited gains on the three simulated benchmarks, which the authors attribute to VAE-based reconstruction producing visually plausible but class-misaligned recoveries.
• Distance metric choice is not critical. Both DyMo_c (cosine distance) and DyMo_e (squared Euclidean distance) consistently outperform prior SOTAs on most datasets.
• Components all help, with one exception. Ablations show that fusing all recovered modalities without selection degrades performance; adding positive-reward modalities simultaneously (S), selecting iteratively (I), and using the calibrated reward (I+C) each help. For CAD and Infarction, however, I outperforms I+C, which the authors attribute to the calibration term (bounded between 0 and 1) making selection more conservative when the recovered table is consistently informative.
• Simulation training transfers. Integrating the proposed incomplete-modality simulation training improves models marked with † versus their unmarked counterparts across the reported settings.
• One near-tie. DyMo and CONCAT perform similarly on CAD, which the authors suggest is because the recovered table modality is consistently informative across samples, leaving little room for dynamic selection to help. DyMo also matches M3Care† on DVM and CAD at η = 1, which the authors link to TIP's limited full-table reconstruction.

Methodology in Plain English

The system has three parts.

First, a multimodal network that can classify from any subset of modalities. Each available modality goes through its own encoder to produce feature tokens, and the tokens are concatenated with a learnable [CLS] token. Dummy tokens occupy the slots of missing modalities so the sequence structure is preserved. A stack of transformer layers models cross-modal interactions using attention masks, and a linear softmax classifier reads out the [CLS] representation.

Second, a selection procedure at test time. Missing modalities are first recovered (using MoPoE for PolyMNIST, MST, and CelebA; TIP for DVM and UKBB). Then, for each recovered modality, the method computes a reward: it measures how the class-prediction loss changes when that recovered modality is added to the currently observed set. Theoretically, the authors show that lowering test-time cross-entropy raises a lower bound on the mutual information between representation and label, so loss decrease is a usable stand-in for information gain. Positive reward means useful information, zero reward means low-fidelity noise, negative reward means task-relevant but semantically inconsistent content. Because the true label is not available at test time, predicted labels substitute for it, and a calibration term based on how representative a representation is within its predicted class cluster (its intra-class similarity, modeled via a truncated normal distribution over distances to the class prototype) down-weights rewards when the new representation is less representative than the old one. The algorithm then repeatedly adds the highest-reward recovered modality, discards all candidates with non-positive rewards, and stops when none remain.

Third, training. Each minibatch randomly samples A non-empty modality subsets from the 2^M − 1 available subsets, rather than enumerating every missing pattern, which the authors note reduces computational cost. A classification loss is combined with an auxiliary missing-agnostic contrastive loss that pulls same-class representations toward their class prototype and pushes apart different classes, regardless of which modalities are present. Squared Euclidean and cosine distance are both tested in this loss.

Why This Matters

Impact on research. The paper reframes incomplete multimodal learning from a binary choice (impute or ignore) into a per-sample selection problem, and provides a label-free criterion for doing the selection at inference. It also connects information theory to a practical reward signal, which may be reusable in other settings where reliability of auxiliary or synthetic inputs must be judged without ground truth.

Real-world applications.

• Clinical decision support: the UK Biobank experiments address coronary artery disease and myocardial infarction classification from cardiac MR images plus tabular features, where patients routinely lack one data source.
• Multi-center medical studies: heterogeneous collection protocols across centers are explicitly named as a cause of missing modalities.
• Marketing and multimodal product data: the DVM (Data Visual Marketing) dataset covers natural images paired with tabular data, evaluated under partial or complete table loss.
• Sensor and streaming systems: sensor malfunctions and transmission errors are cited as practical causes of missing modalities, relevant to robotics and embodied intelligence.

Industry relevance. The method reportedly requires no additional architecture overhead for its dynamic algorithm, avoids the extra modality-specific branches and multi-stage training that prior dynamic fusion methods use, and is compatible with any modality recovery method — properties that matter for deployment in systems where models must degrade gracefully as inputs disappear.

Future Directions

• Stronger recovery methods. The authors observe that DyMo matches M3Care† on DVM and CAD at η = 1, likely because of TIP's limited full-table reconstruction, and state that stronger recovery could further enhance results.
• Tuning the calibration conservatism. Since the calibration term hurt CAD and Infarction, the authors propose tuning a dataset-specific scaling hyper-parameter before applying it, and leave this for future work.
• Handling incomplete data during training. The authors note that DyMo can also handle incomplete modality training because the network supports arbitrary modality combinations, but the reported experiments train on complete datasets and evaluate under missing scenarios.
• Extending the reward beyond classification. The reward is derived from classification cross-entropy, so whether the same information-gain proxy works for other task types is left open.

Target Audience

Researchers and practitioners in multimodal machine learning, particularly those working on robustness to missing modalities, dynamic or adaptive neural networks, and multimodal fusion. It is also relevant to applied researchers in medical imaging and clinical machine learning who deal with incomplete patient records, and to engineers deploying multimodal systems where inputs cannot be guaranteed at inference time. Readers will need a background in information theory and transformer architectures to follow the derivations in Sections 3.2 and 3.3, though the experimental results in Section 4 are accessible without it.

Authors’ abstract

Multimodal deep learning (MDL) has achieved remarkable success across various domains, yet its practical deployment is often hindered by incomplete multimodal data. Existing incomplete MDL methods either discard missing modalities, risking the loss of valuable task-relevant information, or recover them, potentially introducing irrelevant noise, leading to the discarding-imputation dilemma. To address this dilemma, in this paper, we propose DyMo, a new inference-time dynamic modality selection framework that adaptively identifies and fuses reliable recovered modalities, fully exploring task-relevant information beyond the conventional discard-or-impute paradigm. Central to DyMo is a novel selection algorithm that maximizes multimodal task-relevant information for each test sample. Since direct estimation of such information at test time is intractable due to the unknown data distribution, we theoretically establish a connection between information and the task loss, which we compute at inference time as a tractable proxy. Building on this, a novel principled reward function is proposed to guide modality selection. In addition, we design a flexible multimodal network architecture compatible with arbitrary modality combinations, alongside a tailored training strategy for robust representation learning. Extensive experiments on diverse natural and medical image datasets show that DyMo significantly outperforms state-of-the-art incomplete/dynamic MDL methods across various missing-data scenarios. Our code is available at https://github.com//siyi-wind/DyMo.