X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation

AI summary

Overview

Research area: Multimodal machine learning, specifically dataset distillation (dataset condensation) for paired audio-visual data, evaluated on audio-visual event recognition.

Technical level: Advanced. The paper assumes familiarity with dataset distillation paradigms (distribution matching, gradient matching, trajectory matching), attention mechanisms, and multimodal representation learning.

Scope: The paper introduces X-AVDD, a cross-attention-based, memory-decoupled method for distilling small synthetic audio-visual datasets from four benchmark benchmarks (AVE, MUSIC-21, VGGSound, VGGSound-10K), and evaluates it against DM, AVDD, and a multimodal adaptation of SRe²L across ConvNet, ResNet-18, ViT, VGG-11, and AudioCLIP architectures.

What This Paper Is About

Training multimodal models on large paired audio-visual datasets is expensive, so dataset distillation tries to synthesize a much smaller dataset that preserves most of the training utility. The prior audio-visual method, AVDD, has two problems: it must hold all synthetic samples in GPU memory at once (so it runs out of memory at higher samples-per-class and on modern architectures), and it produces synthetic audio-visual pairs whose audio and visual features drift far apart, losing the cross-modal correspondence that makes multimodal learning valuable. X-AVDD's goal is to distill audio-visual data that stays semantically aligned across modalities while using far less memory and time.

Key Contributions

1. A cross-attention-based distillation method (X-AVDD) that explicitly models inter-modal relationships during synthesis using bi-directional, multi-head cross-attention between audio and visual streams, rather than distilling each modality independently.
2. A decoupled cross-modal alignment strategy: the alignment phase is separated from the synthesis loop, using a pre-trained teacher model with frozen cross-attentive guidance. This enables class-wise, batch-wise optimization instead of holding the entire synthetic set in memory.
3. Demonstrated memory and compute efficiency: X-AVDD reduces total synthesis time on a Tesla V100 GPU from 10 hours to 1.3 hours and reduces end-to-end synthesis cost by roughly 8x GPU-hours, enabling scaling to higher SPC settings and larger architectures.
4. Empirical gains across benchmarks and architectures, including lower modality gap (class-wise ℓ₂ distance between audio and visual class centers), better diversity metrics, and consistent improvements over AVDD and a multimodal SRe²L baseline on four datasets and five backbone families.

Main Findings

• Large accuracy gains on VGGSound: X-AVDD improves top-1 test accuracy from 8.2% to 24.0% at SPC=10, and from 9.8% to 26.5% at SPC=20, compared to AVDD. On VGGSound-10K with ConvNet at SPC=20, X-AVDD reaches 62.7% ± 0.5 versus AVDD's 58.0% ± 1.7 and DM's 49.0% ± 2.4 (full dataset reference: 74.7%).
• Reduced modality gap: The paper reports that X-AVDD achieves a consistently lower class-wise ℓ₂ distance between audio and visual class centers than AVDD on the AVE dataset using the AudioCLIP network, measured against the original dataset.
• Cross-architecture generalization: With a ViT target on the AVE dataset at SPC=50, X-AVDD improves accuracy from 25.5% to 35.3% over the multimodal SRe²L adaptation. Synthetic data generated by ResNet-18 transfers to VGG-11 and ViT targets across VGGSound-10K, AVE, and MUSIC-21.
• Scaling to AudioCLIP: On VGGSound-10K, X-AVDD reaches 62.3% ± 0.8 accuracy using only 5.65% of the full dataset, against a full-dataset accuracy of 72.4% ± 0.7, and completes in 2.74 hours versus 25.30 hours for the full dataset (described as a 9x lower training time). On AVE, X-AVDD reaches 81.3% ± 0.9 at 5.0% ratio versus 85.6% ± 1.1 on the full dataset (6.43 hours versus 72.12 hours).
• Transfer to frozen foundation-model backbones: With ImageBind frozen audio/visual backbones on VGGSound-10K at SPC=50, ResNet-18-distilled X-AVDD data yields 67.1% ± 0.3 versus 56.7% ± 0.6 for SRe²L. With an AST audio encoder plus SigLIP2 visual encoder, X-AVDD reaches 73.5% ± 0.3 at SPC=50 versus 56.5% ± 0.7 for SRe²L. The full-dataset reference figures for ImageBind are reported as 79.8 and 81.3.
• Memory and GPU-hour savings: In a single 32 GB Tesla V100 comparison at SPC=20 on ConvNet, GPU-hour consumption drops from 165 to 23 on VGGSound-10K, 471 to 51 on AVE, 363 to 45 on MUSIC-21, and 5342 to 628 on VGGSound.
• Class-wise batching hurts AVDD: The authors report that a class-wise batching workaround for AVDD's memory problem degrades performance from 58.0% to 49.5% at SPC=20, because it loses inter-class relationship information.
• Bi-directionality matters: Ablating either cross-modal direction underperforms the full bidirectional objective on VGGSound-10K. At SPC=50, the full method gives 66.7% ± 0.9, "Only V→A" gives 63.1% ± 0.1, and "Only A→V" gives 62.0% ± 0.9.
• Improvement Over Random (IOR): X-AVDD beats random subset selection at every reported SPC. IOR values are 9.4/6.2/7.2 (VGGSound-10K), 22.3/11.9/9.0 (MUSIC-21), and 11.9/14.5/10.1 (AVE) at SPC=1/10/20.
• Higher synthetic diversity: Using Vendi Score, intra-class cosine similarity, and Coverage, X-AVDD's synthetic data outperforms the multimodal SRe²L baseline on all three metrics across VGGSound-10K, MUSIC-21, and VGGSound.
• Soft labels help: Training the student network with a combination of hard and soft labels improves test accuracy for both the SRe²L adaptation and X-AVDD across SPC settings on VGGSound-10K and MUSIC-21.

Methodology in Plain English

The method has three stages. First, a teacher network containing a bi-directional cross-attention module is pre-trained on the full real audio-visual dataset. In this module, audio queries attend to visual keys and values, and visual queries attend to audio keys and values, so each modality refines its representation using the other. Second, the teacher computes and stores the "attended embeddings" for the real data — what the audio looks like when informed by the video, and what the video looks like when informed by the audio — for each class. Third, the only recurring step: a small synthetic dataset is initialized from randomly sampled real audio-frame pairs, and its samples are optimized so that their cross-attended embeddings (using real audio as key/value when synthesizing video, and vice versa) land close to the stored real embeddings, minimizing pairwise Euclidean distance. This is called Euclidean Distribution Matching (EDM), combined with a standard cross-entropy loss on the classification task via a weighted sum. Because the synthetic samples can be optimized class by class and do not need to be held in memory all at once — unlike AVDD's global distribution matching — peak VRAM use and total synthesis time drop sharply. The paper notes that AVDD's memory footprint forces it toward simple ConvNet architectures and low SPC settings.

Why This Matters

Impact on research: The paper casts the "modality collapse" problem — where synthetic multimodal pairs lose their semantic alignment — as something that can be addressed proactively through a decoupled, cross-attentive alignment step, rather than as an unavoidable side effect of joint optimization. It is described as the first work to use bi-directional cross-attention in a multimodal distillation context, distinguishing it from unimodal attention-based distillation such as DataDAM. It also makes audio-visual distillation feasible on architectures (ViT, AudioCLIP, ImageBind, AST + SigLIP2) on which AVDD runs out of memory.

Real-world applications:

• Audio-visual event recognition in video platforms, where labeled paired data is expensive to curate.
• Faster neural architecture search and hyperparameter tuning for multimodal models, using compact distilled datasets as a proxy.
• Privacy-conscious sharing of training signals, since distilled synthetic data can be distributed instead of the original corpus.
• Cheaper downstream adaptation of multimodal foundation models, demonstrated by training small heads on frozen ImageBind or AST + SigLIP2 features using only distilled data.

Industry relevance: The reported reduction from 10 hours to 1.3 hours of synthesis on a single V100, and 9x lower training time to reach comparable accuracy on AudioCLIP, points directly at cost reduction for teams training or repeatedly retraining multimodal models. The 5342 to 628 GPU-hour drop on VGGSound is the largest reported absolute saving.

Future Directions

• Extending X-AVDD beyond audio-visual classification to other downstream tasks such as temporal localization and audio-visual retrieval, which the authors explicitly name as future work.
• Testing whether the decoupled cross-attentive alignment transfers to other modality pairings, such as vision-language, where modality collapse has also been identified.
• Pushing to higher samples-per-class budgets and larger backbone families, since the reported experiments stop at SPC=50 in most tables.
• Investigating why cross-modal constraints over-regularize at very low SPC on ConvNet, where the extra constraints hurt at SPC=1 but help from SPC≥10 onward.

Target Audience

Researchers and graduate students working on dataset distillation, dataset condensation, or multimodal representation learning; practitioners who need to train audio-visual models on constrained GPU budgets; and engineers evaluating whether distilled synthetic data can substitute for large proprietary audio-visual corpora in transfer-learning and foundation-model fine-tuning pipelines. Readers without background in attention mechanisms or distribution-matching distillation will find the loss formulation and the SRe²L appendix material difficult, though the high-level three-step recipe is accessible.

Authors’ abstract

Audio-visual dataset distillation aims to synthesize a small dataset from a large paired audio-visual dataset, while nearly preserving the training utility for neural networks. Existing approaches such as DM and AVDD face two critical limitations: 1) they are memory-intensive, making them difficult to scale to complex architectures and higher samples-per-class (SPC) settings, and 2) they often fail to preserve strong cross-modal semantic correspondence. To tackle these issues, we propose X-AVDD, a cross-attention-based distillation method that explicitly captures inter-modal relationships during synthesis. By decoupling cross-modal alignment from joint optimization, X-AVDD is substantially more memory-efficient, reducing end-to-end synthesis cost by $\sim 8\times$ GPU-hours, while producing highly aligned synthetic data. Empirically, X-AVDD significantly outperforms other audio-visual distillation methods. On VGGSound, an audio-visual dataset of short YouTube clips, X-AVDD improves top-1 test accuracy from $8.2\%\rightarrow24.0\%$ at SPC=$10$ and $9.8\%\rightarrow26.5\%$ at SPC=$20$. Furthermore, X-AVDD improves cross-architecture generalization (e.g., $25.5\%\rightarrow35.3\%$ for a ViT target on AVE dataset at SPC $=50$) and successfully scales to complex architectures such as AudioCLIP, achieving $62.3\%$ accuracy using only $5.65\%$ of the full dataset, against the full dataset accuracy of $72.4\%$ while completing in $9\times$ lower training time compared to the full dataset.