ImagebindDC: Compressing Multi-modal Data with Imagebind-based Condensation

AI summary

Overview

Research area: Multi-modal dataset condensation / dataset distillation, sitting at the intersection of computer vision, audio-visual learning, and efficient machine learning. The work builds on ImageBind as a unified multi-modal embedding space and on distribution-matching theory (Maximum Mean Discrepancy, Characteristic Functions).

Technical level: Advanced. The paper assumes familiarity with dataset distillation, bi-level optimization, kernel methods, and Fourier-domain distribution distances.

Scope: A single paper-length study that proposes and empirically validates a framework for condensing paired multi-modal data (vision, audio, text, depth) into a handful of synthetic examples per class.

Venue/status: arXiv preprint arXiv:2511.08263v1 [cs.CV], dated 11 November 2025, released under a CC BY 4.0 license. Authors are affiliated with EPIC Lab, SJTU; Bosch Corporate Research Asia Pacific; and HKUST. Yue Min and Shaobo Wang are listed as equal contributors.

What This Paper Is About

Dataset condensation tries to compress a large dataset into a tiny synthetic one that trains models almost as well — a solved-enough problem for images alone, but one that breaks down when data comes in linked pairs, such as a video and its soundtrack. Existing multi-modal methods condense each modality in its own feature space or reduce cross-modal structure to a single similarity number, which loses the relationship that gives the data meaning. ImageBindDC instead condenses everything inside one shared ImageBind embedding space and uses a Characteristic Function loss to match the real and synthetic distributions at three levels at once: within each modality, between modalities, and jointly across the full multi-modal structure.

Key Contributions

1. ImageBindDC, a unified-space condensation framework. The authors state this is the first data condensation framework specifically designed to operate in a unified multi-modal feature space (the pretrained ImageBind embedding space), rather than condensing each modality separately.

2. A three-level distribution-matching objective. The total loss is L_total = L_uni + L_cross + L_joint (with hyperparameters λ_uni, λ_cross, λ_joint in the full formulation), covering uni-modal alignment (real vs. synthetic within each modality), cross-modal alignment (hybrid real-synthetic pairs), and joint-modal alignment (paired real data vs. paired synthetic data).

3. Use of Characteristic Function Discrepancy (CFD) instead of a kernel-based metric. Replacing MMD (used by the DM baseline) with CFD, which operates in the Fourier domain and implicitly matches all statistical moments, removes the dependence on a heuristically chosen kernel.

4. State-of-the-art results across four multi-modal settings with large efficiency gains. Reported improvements span audio-visual classification (VGGS-10K, AVE), depth-text classification (NYU-v2), and audio-text retrieval (Clotho), along with condensation times and GPU memory figures far below the baselines.

Main Findings

• NYU-v2 depth-text classification: The abstract reports that a model trained on just 5 condensed datapoints per class achieves lossless performance comparable to training on the full dataset, with a new state of the art and an 8.2% absolute improvement over the previous best method, plus more than 4× less condensation time. In Table 3, ImageBindDC reaches 97.30±1.17 at 5 DPC versus 95.92±1.63 for AVDD and 89.08±4.96 for DM, against 98.62±0.25 for the whole dataset. At 10 DPC, ImageBindDC reaches 98.73±1.04 versus 96.89±1.26 for DM and 98.62±0.45 for AVDD.

• Audio-visual event localization (AVE): The introduction reports that a model trained on just 20 synthesized image-audio pairs reached 98% of full-dataset performance, a 2.53% improvement. In Table 2 (ImageBind-guided), ImageBindDC reaches 70.10±1.24 at 1 DPC versus 67.32±1.55 for AVDD, 65.35±2.06 for DM, and 76.93±0.14 for the whole dataset.

• VGGS-10K: With 10 DPC and ImageBind-guided distillation, ImageBindDC achieves 55.23±0.13, compared with 48.08±0.92 for AVDD and 43.85±1.75 for DM (whole data: 60.57±0.05). With 1 DPC, ImageBindDC reaches 42.66±1.48 against 40.41±1.81 for AVDD (whole data: 68.24±0.75).

• ConvNet-guided settings (Table 1): On VGGS-10K at 1 DPC, ImageBindDC scores 37.25±1.62 versus 36.10±1.67 for AVDD; at 20 DPC it scores 56.11±0.98 versus 55.30±0.18 for AVDD. On AVE at 1 DPC it scores 18.08±0.52 versus 16.90±0.14 for AVDD.

• Cross-architecture generalization (Table 5, AVE): At 1 DPC, ImageBindDC reaches 12.69±1.23 on ConvNet and 70.10±1.24 on ImageBind, versus 4.85±1.11 and 67.32±1.55 for AVDD. At 10 DPC, the table reports 16.74±0.18 on ConvNet and 73.67±0.31 on ImageBind, versus 12.44±1.29 and 71.33±0.35 for AVDD. The body text cites 14.42% on ConvNet for the 10 DPC case, which does not match the 16.74 figure printed in Table 5.

• Audio-text retrieval (Clotho, 20 DPC, Table 4): ImageBindDC improves over AVDD and DM on all six recall metrics — for example A2T R@1 0.0362 (vs. 0.0316 AVDD, 0.0268 DM) and T2A R@10 0.2281 (vs. 0.2048 AVDD, 0.1684 DM), against whole-data 0.0526 and 0.2488 respectively.

• CFD beats MMD (Figure 3): For the uni-modal audio task at 1 DPC, ImageBindDC achieves 32.33% versus 27.87% for DM; at 10 DPC on the combined Video+Audio task, 70.34% versus 69.26%.

• All three alignment terms are needed (Figure 4): At 10 DPC, uni-modal loss alone gives 70.34%; adding joint-modal or cross-modal objectives individually improves on that baseline; combining all three reaches 73.67%, an absolute improvement of 3.33%. The authors describe the objectives as synergistic rather than merely additive.

• Efficiency (Table 6, VGGS-10K): At 1 DPC, ImageBindDC takes 57.46 s versus 140.3 s for DM (over 2.4× faster) and uses 5.6 GB versus 8.96 GB (37.5% less GPU memory). At 20 DPC it takes 123.74 s versus 419.9 s for a single full-data epoch (over 3.4× faster) and uses 13.39 GB versus 55.29 GB (over 75% reduction). The introduction summarizes this as reducing condensation time by over 4.6× for 20 DPC. DC and MTT ran out of memory in this table; DC and MTT also recorded OOM in parts of Table 2, where the note states a 2 × H100 GPU limit.

• Qualitative evidence: On NYU-v2, distilled images from ImageBindDC retain recognizable scene features (e.g., bathroom, bedroom) whereas AVDD samples degrade into distorted amalgamations. A UMAP projection on Clotho shows ImageBindDC's synthetic audio and text embeddings integrated with the real embedding clouds, while Random collapses into a small cluster and DM forms separate clusters.

Methodology in Plain English

The approach has three moving parts.

1. Condense in a shared space, not separate ones. Instead of comparing synthetic and real data in raw pixels and raw waveforms, the method pushes both through a frozen pretrained ImageBind encoder. ImageBind already maps images, audio, and other modalities into one common embedding space, so "does the synthetic data look like the real data" becomes a single question asked once, across modalities, rather than once per modality. The synthetic inputs themselves are the only things being optimized.

2. Measure the gap with characteristic functions rather than a kernel. The standard tool in this literature is Maximum Mean Discrepancy (MMD), which compares the mean embeddings of two distributions in a kernel space. Its accuracy depends on picking a good kernel, and the field typically defaults to a Gaussian kernel heuristically. The authors instead use the Characteristic Function — the Fourier transform of a probability density — and define the Characteristic Function Discrepancy (CFD) as the squared L2 distance between the characteristic functions of the real and synthetic feature distributions. Because a distribution is uniquely determined by its characteristic function (Lévy's Uniqueness Theorem), matching them amounts to matching all statistical moments at once, with no kernel to tune. In practice the expectation is approximated with sampled random frequency vectors; the appendix states a Gaussian distribution is used for these frequency vectors and that the number of evaluation steps is 3.

3. Enforce alignment at three levels. The final objective sums three losses. Uni-modal alignment matches real versus synthetic embeddings within audio and within vision separately. Cross-modal alignment forms element-wise products of real audio×image embeddings and of synthetic audio×image embeddings, computes their cosine similarity, and minimizes 1 minus that similarity. Joint-modal alignment takes mean embeddings per modality, reshapes them into row vectors, and multiplies them to form a joint cross-modal similarity score, again minimizing 1 minus the score. The three are combined with weighting hyperparameters.

Setup details. Experiments ran on 4 NVIDIA A100 GPUs. Audio-visual training used a learning rate of 0.2 with SGD and momentum 0.5; NYU-v2 used a learning rate of 0.001 for both backbone and classifier. Every experiment was run 3 times. Synthetic data was initialized with herding, trained for 30 iterations with evaluation every 10 steps, with separate learning rates for synthetic audio and images (0.5), classifier (0.001), frame modality (0.0001) and sound modality (0.001), using Adam with beta1 = 0.9 and weight decay 0.0001, batch sizes of 32 for synthetic and 128 for real data, and differentiable Siamese augmentation.

Why This Matters

Impact on research. The paper reframes multi-modal condensation as a distribution-matching problem in a single shared embedding space, and it imports CFD — previously used in uni-modal condensation — into that setting. If the reported numbers hold, this suggests that much of the difficulty in multi-modal condensation comes from condensing modalities separately and from using kernel-based metrics, not from the data itself. It also gives the field a three-part ablation structure (uni-modal / cross-modal / joint-modal) that future work can build on or challenge, and it reports cross-architecture transfer, which is a stricter test than same-architecture accuracy.

Real-world applications (as suggested by the paper's experimental settings):

• Training audio-visual event localization models on small, shareable synthetic pairs instead of large video corpora — the AVE setting.
• Depth-and-text scene understanding for robotics or AR, using condensed NYU-v2-style data (the paper's depth-text task draws text labels from scene names such as "bathroom").
• Audio captioning and audio-text retrieval pipelines, where condensed data can stand in for a large captioning corpus (the Clotho experiments).
• Resource-constrained or privacy-sensitive training, where the point is to avoid storing or redistributing the full original media collection.

Industry relevance. The condensation-time and GPU-memory results (57.46 s and 5.6 GB at 1 DPC versus 140.3 s and 8.96 GB for DM on VGGS-10K) point at the practical bottleneck for teams without large compute budgets. Several competing methods ran out of memory on the hardware used, which the paper highlights as a limitation of existing approaches. The presence of industrial co-authors from Bosch Corporate Research Asia Pacific suggests the authors see multi-modal condensation as relevant to applied perception systems rather than purely academic.

Future Directions

• Closing the residual accuracy gap. The condensed models still trail whole-dataset training in most tables (for example 73.67 versus 76.93 on AVE at 10 DPC with ImageBind, and 55.23 versus 60.57 on VGGS-10K). How much of that gap is fundamental to condensation, and how much is optimization, remains open.

• Extending beyond the two modalities demonstrated. The framework is written to be modality-agnostic, but the audio-visual experiments use audio and vision, NYU-v2 uses depth and text, and Clotho uses audio and text. Scaling to three or more simultaneously bound modalities, and to the other modalities ImageBind supports, is not demonstrated.

• Removing the dependence on one frozen backbone. All results rest on ImageBind's shared embedding space. How the method behaves with a different or weaker joint encoder, and whether the three losses stay well balanced without it, is not addressed.

• Settling the unreconciled reporting details. The body text's cross-architecture ConvNet number for 10 DPC (14.42%) differs from the value printed in Table 5 (16.74±0.18), and the abstract attributes the 8.2% NYU-v2 gain to the 5-DPC result while Table 3 shows the largest gap at 1 DPC. Independent replication and clearer reporting would help.

Target Audience

Researchers working on dataset distillation and condensation, particularly those moving from uni-modal image benchmarks toward paired multi-modal data. It is also relevant to practitioners in audio-visual learning, audio captioning, and depth-based scene understanding who need to shrink training sets under compute or storage constraints, and to anyone interested in distribution-matching theory (kernel methods versus characteristic functions) applied at scale. Readers without a background in dataset distillation or Fourier-domain statistics will find the method section dense; the introduction, figures, and results tables are more accessible.

Authors’ abstract

Data condensation techniques aim to synthesize a compact dataset from a larger one to enable efficient model training, yet while successful in unimodal settings, they often fail in multimodal scenarios where preserving intricate inter-modal dependencies is crucial. To address this, we introduce ImageBindDC, a novel data condensation framework operating within the unified feature space of ImageBind. Our approach moves beyond conventional distribution-matching by employing a powerful Characteristic Function (CF) loss, which operates in the Fourier domain to facilitate a more precise statistical alignment via exact infinite moment matching. We design our objective to enforce three critical levels of distributional consistency: (i) uni-modal alignment, which matches the statistical properties of synthetic and real data within each modality; (ii) cross-modal alignment, which preserves pairwise semantics by matching the distributions of hybrid real-synthetic data pairs; and (iii) joint-modal alignment, which captures the complete multivariate data structure by aligning the joint distribution of real data pairs with their synthetic counterparts. Extensive experiments highlight the effectiveness of ImageBindDC: on the NYU-v2 dataset, a model trained on just 5 condensed datapoints per class achieves lossless performance comparable to one trained on the full dataset, achieving a new state-of-the-art with an 8.2\% absolute improvement over the previous best method and more than 4$\times$ less condensation time.