Distilling Cross-Modal Knowledge via Feature Disentanglement

AI summary

Overview

• Research area: Cross-modal knowledge distillation (CMKD) in computer vision, combining frequency-domain signal analysis with feature-level distillation for audio-visual and image-text models.
• Technical level: Advanced. The paper assumes familiarity with knowledge distillation, Fourier transforms, and domain-adaptation concepts such as feature standardization and classifier alignment.
• Scope: Presents a frequency-decoupled cross-modal distillation framework ("FD-CMKD") that splits features into low- and high-frequency components, applies different losses to each, and adds scale and feature-space alignment, evaluated on audio-visual classification, image-text classification, and semantic segmentation benchmarks.

What This Paper Is About

Traditional knowledge distillation transfers knowledge between models working on the same type of data, but it breaks down when the teacher and student operate on different modalities (for example, a visual teacher and an audio student), because their features encode both shared semantics and modality-specific detail in ways that do not line up. This paper's goal is to separate those two kinds of information so that the shared part can be aligned strongly while the modality-specific part is only loosely constrained, thereby improving the student without destroying its native representation. The authors do this by decomposing features in the frequency domain and adding alignment mechanisms for feature scale and feature space.

Key Contributions

1. A frequency-domain observation about cross-modal features. The authors measured inter-modal cosine similarity and found low-frequency features are more similar across modalities than raw features (0.91 vs. 0.84 on CREMA-D; 0.85 vs. 0.74 on AVE), while high-frequency features are nearly uncorrelated (−0.02 on CREMA-D; −0.01 on AVE).
2. A frequency-decoupled CMKD framework. Features from each modality are converted with the Discrete Fourier Transform, split by fixed binary low-pass and high-pass masks (the first half of frequency components for low-pass, the second half for high-pass), and reconstructed with the Inverse Discrete Fourier Transform. Strong alignment (MSE) is applied to low-frequency features and weak alignment (logMSE) to high-frequency features.
3. Distribution-alignment components. A scale consistency loss built on feature standardization (mean subtraction plus L2 normalization, with mean subtraction implemented as a DC filter in the frequency domain) addresses inter-modal scale differences, and a shared classifier aligns teacher and student features in a common decision space.
4. Broad empirical validation. Experiments span audio-visual classification (CREMA-D, AVE, VGGSound), image-text classification (CrisisMMD), and semantic segmentation (NYU-Depth V2), across ResNet-18, BERT-base with MobileNetV2, and DeepLab V3+ backbones, with code released at a public GitHub repository.

Main Findings

• Low-frequency features carry cross-modal semantics; high-frequency features do not. Cosine similarity for low-frequency features (0.91 on CREMA-D, 0.85 on AVE) exceeded that of raw features (0.84 and 0.74), whereas high-frequency similarity was essentially zero (−0.02 and −0.01), supporting the "strong consistency for low frequency, weak consistency for high frequency" design.
• Modalities differ in feature scale. Visualizing feature means across 512 dimensions showed the audio modality's feature values are noticeably higher than the visual modality's, motivating standardization before alignment rather than direct MSE.
• Best accuracy across every tested dataset and modality. On CREMA-D, 64.1% for audio and 71.0% for visual, versus unimodal baselines of 62.4% and 66.8%. On AVE, 64.9% for audio and 47.8% for visual, versus 63.7% and 38.8% — a 9-point gain on the visual modality. On VGGSound, 70.0% for audio and 48.1% for visual, versus 68.9% and 44.9%. On CrisisMMD, 79.1% for text and 72.7% for visual, versus 77.4% and 70.2%.
• Competitive baselines frequently underperform the no-distillation baseline. On CREMA-D's visual modality, AVE's audio modality, and VGGSound's audio modality, most compared logit-based, feature-based, and cross-modal methods scored below the unimodal baseline, while the proposed method consistently improved on it. The reported cross-modal baseline C2KD scored 57.5% on CREMA-D audio, below the 62.4% unimodal result.
• Stable bidirectional transfer. On CrisisMMD, DKD worked for text but not visual, and AFD worked for visual but failed for text; the proposed method reached 79.1% (text) and 72.7% (visual) consistently.
• Segmentation gains. On NYU-Depth V2, 33.2% mIoU for Depth and 36.9% mIoU for RGB, surpassing the next best methods (DIST for Depth, DKD for RGB) by 0.9 and 1.6 points respectively, and beating the unimodal baselines of 30.9% and 34.1%.
• Comparison against segmentation-specific methods. Against MMANet (29.6 Depth / 32.6 RGB), RDP (29.7 / 27.2), and CIRKDv2 (33.1 / 36.4), the proposed method scored 33.2 / 36.9, and combining CIRKDv2 with the proposed method reached 35.1 / 37.9.
• Ablation confirms each component contributes. Starting from an original feature-distillation baseline (60.9 / 64.3 / 58.7 / 39.6 on CREMA-D audio, CREMA-D visual, AVE audio, AVE visual), adding frequency decomposition gave 60.8 / 68.7 / 61.0 / 43.3 — a 0.1-point drop on CREMA-D audio despite gains elsewhere. Adding alignment and standardization modules improved results substantially, and the full combination of all components produced 64.1 / 71.0 / 64.9 / 47.8.
• Loss choice matters and is asymmetric. MSE for low frequency with logMSE for high frequency gave 64.1 / 71.0 on CREMA-D. Using MSE for both gave 62.2 / 70.5, logMSE for both gave 62.6 / 68.0, and logMSE for low with MSE for high gave 61.7 / 67.6.
• t-SNE shows separated modality clusters. Without distillation, modality feature distributions were distinguishable; with conventional feature distillation, the teacher visual and student audio features overlapped heavily, indicating loss of modality-specific information; with the proposed method, the two modalities formed two distinct clusters.

Methodology in Plain English

The approach starts from a measurement, not a guess: the authors took features from trained unimodal models and checked how similar they are across modalities before and after splitting them by frequency. Low frequencies — the smooth, slow-changing part of the feature vector — turned out to be highly similar across modalities, while high frequencies were essentially unrelated. They interpret this as low frequency carrying the shared "what" of the content and high frequency carrying the modality-specific "how."

To exploit this, the method transforms each modality's feature vector with the DFT, multiplies it by a fixed binary mask that keeps the first half of the components (low-pass) or the second half (high-pass), and transforms back with the IDFT. The two resulting feature sets get different treatment during distillation. Low-frequency features from teacher and student are forced to match tightly with a standard MSE loss, since they should represent the same underlying content. High-frequency features use a logMSE-style loss defined by applying a signed logarithmic function to each value before computing squared error, which makes the gradient gentler when values differ greatly — appropriate because high-frequency components carry fine detail plus noise, and full alignment would distort the student's own modality-specific behavior. The mean subtraction step in that standardization is implemented as a DC filter in the frequency domain.

Two additional mechanisms handle distribution mismatch. First, both modalities' features are standardized by subtracting the mean and dividing by the L2 norm, so they are compared on the same numerical scale regardless of how large their raw values are. Second, teacher and student features are passed through shared classifiers for the high-frequency and low-frequency branches, with cross-entropy supervision, so both modalities are judged within the same decision space. Cross-entropy is also applied to the student's raw, low-frequency, and high-frequency features to keep them discriminative. The total objective combines the task loss, the alignment loss, and the low- and high-frequency distillation losses weighted by λ1 and λ2.

Why This Matters

• Impact on research: The paper reframes cross-modal distillation as a problem of separating shared from modality-specific information rather than forcing a single strong alignment across heterogeneous features. Its similarity measurements give a concrete, quantifiable justification for frequency decoupling, and the finding that many existing distillation methods fall below the no-distillation baseline on certain modality pairs — such as C2KD at 57.5% versus 62.4% unimodal on CREMA-D audio — is a useful caution for the field.
• Real-world applications:
  • Audio-visual event localization and sound classification, where a strong visual model can improve a weaker audio model (and vice versa) on datasets like AVE and VGGSound.
  • Emotion recognition from speech and facial video, as on CREMA-D, where one modality may be far more informative at deployment time.
  • Disaster-response analysis of social media, as on CrisisMMD, where both tweet text and attached images must be classified across categories such as rescue, infrastructure damage, and affected individuals.
  • Efficient multimodal deployment: indoor scene understanding for robotics or AR, as on NYU-Depth V2, where an RGB-trained model can be distilled into a depth-based student for a cheaper sensor, or a depth model into an RGB one.
• Industry relevance: Model compression is a practical constraint on edge devices and embedded sensors. This work targets settings where the deployable modality is not the one with abundant labels or a large pretrained model, and its segmentation experiments cover RGB-to-depth and depth-to-RGB transfer on DeepLab V3+ with a ResNet-18 backbone. A public code repository (github.com/Johumliu/FD-CMKD) lowers the barrier to adoption.

Future Directions

• Choosing frequency split points adaptively. The masks are fixed binary filters that keep the first half or second half of frequency components; whether the split point should depend on modality, dataset, or layer is not resolved by the reported results.
• Tuning the loss weights. The total loss uses weighting parameters λ1 and λ2, and a sensitivity study of the high- and low-frequency loss weights is referenced (Figure 5), but the specific values and their sensitivity behavior are not reported in the available content.
• Applying the shared classifier more broadly. The shared classifier is used for classification but not for segmentation, since segmentation produces pixel-level rather than global classification outputs; whether feature-space alignment can be reformulated for dense prediction beyond the scale-alignment component is left open.
• Extending beyond the tested modality pairs. The experiments cover audio-visual, image-text, and RGB-depth combinations with ResNet-18, BERT-base/MobileNetV2, and DeepLab V3+; whether the frequency-similarity pattern holds for other modality combinations or architectures is not established.

Target Audience

Researchers and practitioners working on knowledge distillation, multimodal learning, and model compression, particularly those transferring knowledge between heterogeneous sensor or data types. It is also relevant to engineers deploying compact multimodal models on devices where only one modality is cheap to acquire or label. Readers need a working understanding of Fourier transforms, distillation losses, and standard vision backbones to follow the methodology; the similarity measurements and t-SNE results, however, are accessible to a broader audience.

Authors’ abstract

Knowledge distillation (KD) has proven highly effective for compressing large models and enhancing the performance of smaller ones. However, its effectiveness diminishes in cross-modal scenarios, such as vision-to-language distillation, where inconsistencies in representation across modalities lead to difficult knowledge transfer. To address this challenge, we propose frequency-decoupled cross-modal knowledge distillation, a method designed to decouple and balance knowledge transfer across modalities by leveraging frequency-domain features. We observed that low-frequency features exhibit high consistency across different modalities, whereas high-frequency features demonstrate extremely low cross-modal similarity. Accordingly, we apply distinct losses to these features: enforcing strong alignment in the low-frequency domain and introducing relaxed alignment for high-frequency features. We also propose a scale consistency loss to address distributional shifts between modalities, and employ a shared classifier to unify feature spaces. Extensive experiments across multiple benchmark datasets show our method substantially outperforms traditional KD and state-of-the-art cross-modal KD approaches. Code is available at https://github.com/Johumliu/FD-CMKD.