Connecting Giants: Synergistic Knowledge Transfer of Large Multimodal Models for Few-Shot Learning

AI summary

Overview

• Research area: Computer vision — few-shot learning (FSL), knowledge distillation, and transfer of knowledge from large multimodal models (LMMs).
• Technical level: Advanced. The paper assumes familiarity with few-shot episode training (N-way K-shot), cosine classifiers, CLIP-style vision-language alignment, autoencoders, and knowledge distillation.
• Scope in one sentence: The paper proposes SynTrans, a three-stage framework that distills visual knowledge from CLIP and mines semantic knowledge from a large language model plus a vision-language model, then fuses both into classifier weights for a frozen few-shot vision encoder.

What This Paper Is About

Few-shot learning tries to classify new categories when only a handful of labeled examples exist, and existing methods usually lean on visual priors or on thin semantics such as word vectors or attribute annotations, which are noisy or expensive to collect. This paper asks whether the rich implicit knowledge inside large multimodal models can be extracted and converted into usable classifier weights that compensate for the missing visual data. The goal is to boost an off-the-shelf few-shot learner without ever fine-tuning its vision encoder.

Key Contributions

1. A three-stage framework (SynTrans) that transfers complementary knowledge from large multimodal models into a small few-shot learner: visual knowledge distillation, semantic knowledge transfer, and multi-modal knowledge fusion.
2. A train-free Synergistic Knowledge Mining (SynMine) module that uses chain-of-thought prompting with GPT-3.5-turbo and refining by Res50x4 CLIP to turn class names and WordNet definitions into high-quality semantic descriptors.
3. A Visual-Semantic Bridging (VSBird) module, built as a dual autoencoder with self- and cross-reconstruction objectives, that maps visual embeddings and semantic descriptors into category-specific classifier weights in both directions.
4. A visual weight generator and semantic weight reconstructor that act as meta-learners, adaptively combining visual- and semantic-derived weights into a multimodal FSL classifier — with the claim that SynTrans is the first framework to systematically integrate large multimodal model knowledge to empower small few-shot learners.

Main Findings

• Strong overall accuracy: On MiniImageNet, SynTrans reaches 81.30 ± 0.61 (5-way 1-shot) and 89.96 ± 0.42 (5-way 5-shot) with a ViT-S backbone, and 76.20 ± 0.69 / 86.12 ± 0.54 with ResNet-12. On TieredImageNet it reaches 84.31 ± 0.54 / 91.73 ± 0.44 with ViT-S and 79.69 ± 0.81 / 87.78 ± 0.60 with ResNet-12.
• CIFAR-FS and FC100 results: 84.64 ± 0.65 (1-shot) and 90.81 ± 0.41 (5-shot) on CIFAR-FS with ViT-S; 56.38 ± 0.69 and 69.45 ± 0.54 on FC100 with ViT-S. ResNet-12 gives 82.58 ± 0.75 / 89.42 ± 0.56 on CIFAR-FS and 52.30 ± 0.75 / 64.91 ± 0.59 on FC100.
• Margin over the closest semantic-based baseline: The paper reports that in the 5-way 1-shot setting SynTrans outperforms SemFew by 2.98% (the text does not state which of the four datasets this margin corresponds to).
• Bigger gains when data is scarcer: The paper states improvements are larger in the 1-shot setting than the 5-shot setting, while SynTrans still holds an advantage in 5-shot, suggesting high-quality semantics help even with more visual data.
• Knowledge quality matters (MiniImageNet ablation, ResNet-12): Rich LLM-generated descriptions encoded by a VLM give the best result (76.20 ± 0.69 / 86.12 ± 0.54), ahead of short WordNet definitions with a VLM (73.52 ± 0.73 / 85.26 ± 0.51), plain class names with a VLM (72.87 ± 0.71 / 84.67 ± 0.50), and plain class names with Word2vec (72.66 ± 0.70 / 84.68 ± 0.50). Notably, short WordNet definitions with Word2vec score lowest (72.03 ± 0.73 / 84.18 ± 0.52), below plain class names.
• VLM encoders beat Word2vec: For every knowledge source tested, encoding with a vision-language model outperforms Word2vec in both 1-shot and 5-shot settings.
• Balance hyperparameter α: Accuracy on the visual-dominated classifier peaks at α = 0.7 for the 1-shot setting and α = 0.8 for the 5-shot setting; larger or smaller values reduce accuracy because self-reconstruction and cross-reconstruction losses trade off against each other.
• Fusion produces tighter class clusters: t-SNE visualization of classifier weights for all novel MiniImageNet categories shows loosely defined, overlapping clusters for visual-only weights in the 1-shot case and more compact, distinct clusters after multimodal fusion, with the same pattern repeated in the 5-shot case.

Methodology in Plain English

The framework works in three stages and never fine-tunes the vision encoder.

Stage 1 — Visual knowledge distillation. A large CLIP model (Res50x4 CLIP vision and text encoders) acts as a strong teacher. The few-shot vision encoder acts as a weak student, with a small learnable linear projector and a cosine classifier attached. Teacher and student produce logits over the same images, and the student is trained to match the teacher's output distribution using a Kullback-Leibler divergence loss with a temperature term. Separately, the cosine classifier is trained with cross-entropy on the disjoint base dataset, so its weight vectors carry transferable visual knowledge.

Stage 2 — Semantic knowledge mining and bridging. The train-free SynMine module asks GPT-3.5-turbo, through chain-of-thought prompting, to describe the visual features of each class; the first prompt supplies a WordNet definition to remove ambiguity, and the second asks for at most five sentences focused on discriminative visual features and scientific accuracy. Those descriptions are passed through a pretrained vision-language model's text encoder to become semantic descriptors. VSBird then uses two encoder-decoder subnetworks sharing a latent space: one for the visual space and one for the semantic space. Self-reconstruction losses keep each modality's structure intact, cross-reconstruction losses align the two latent spaces, and a weight α balances the two groups of terms. At test time, the semantic encoder plus visual decoder generate a weight from a semantic descriptor, and the visual encoder plus semantic decoder generate a weight from a visual prototype.

Stage 3 — Multi-modal knowledge fusion. N-way K-shot meta-tasks are sampled from the base training set to mimic testing. Visual prototypes are computed from support samples, and VSBird produces the complementary derived weights. A small fully connected layer with a sigmoid output (the visual weight generator) produces a coefficient β in [0, 1] that blends the semantic-derived weight with the visual prototype; a reconstructor of two fully connected layers with a LeakyReLU and a 2048-dimensional hidden layer blends the semantic weight with the visual-derived weight. At inference, predictions combine both classifiers, with a balancing coefficient λ empirically set to 1/K.

Setup. ResNet-12 (640-dimensional features, pretrained per IER) and ViT-Small (384-dimensional features, pretrained per SMKD) serve as the few-shot encoders. Image sizes are 84×84 for ResNet-12, 320×320 for ViT-S on MiniImageNet and TieredImageNet, and 224×224 for CIFAR-FS and FC100. Training uses Adam with learning rate 0.0001 and weight decay 5×10⁻⁴; VSBird trains for 50 epochs with α = 0.7, and the weight generator/reconstructor train for 10 epochs. Evaluation uses 5-way 1-shot and 5-shot tasks, 600 randomly sampled tasks, 15 query samples per class, and 95% confidence intervals.

Why This Matters

• Research impact: The paper reframes FSL as a knowledge-transfer problem rather than a visual-only or semantics-only one, and shows that a frozen, lightweight backbone can gain substantially from external multimodal knowledge. It also argues against the common practice of relying on word vectors or sparse attribute annotations, showing that LLM-generated descriptions encoded by a VLM outperform both.
• Real-world applications (implied by the method, not demonstrated in the paper — no deployment or field study is reported):
  • Medical and health image classification where labeled cases are scarce, consistent with the authors' affiliation with a Centre for Smart Health.
  • Rare species or rare-object recognition in biodiversity and conservation monitoring.
  • Industrial quality inspection, where defect categories are rare and annotated examples are expensive.
  • Personalization and long-tail product or content recognition in consumer applications.
• Industry relevance: Because the vision encoder is never fine-tuned, the approach is designed to sit on top of existing few-shot learners such as IER and SMKD, which lowers the cost of adopting it. It also offers a practical recipe for using LLM-generated descriptions plus CLIP-style models as a reusable knowledge source instead of paying for attribute annotation.

Future Directions

• Reducing dependence on external model APIs: SynMine relies on GPT-3.5-turbo and Res50x4 CLIP; the paper does not report results with other or smaller LLMs/VLMs, nor the cost of generating descriptions.
• Closing the accuracy gap on certain benchmarks: SynTrans with ResNet-12 trails SemFew with Swin-T on MiniImageNet 1-shot (76.20 versus 78.94), so the benefit of using a heavier or stronger few-shot backbone remains open.
• Better balancing mechanisms: α and λ are set by hand or empirically (λ = 1/K), and accuracy falls off beyond α = 0.7/0.8, suggesting learned or task-adaptive balancing as a next step.
• Generalization beyond the four benchmarks: Results are limited to MiniImageNet, TieredImageNet, CIFAR-FS, and FC100, all in the standard 5-way 1/5-shot protocol; cross-domain, fine-grained, or non-image FSL settings are not explored. The paper reports no limitations section and no code or dataset release.

Target Audience

Researchers and graduate students working on few-shot learning, vision-language models, or knowledge distillation, and practitioners who need strong classification from very few labeled examples without retraining a vision backbone. Readers need prior exposure to episode-based FSL training and multimodal representation learning to follow the loss formulations and the dual autoencoder design.

Authors’ abstract

Few-shot learning (FSL) addresses the challenge of classifying novel classes with limited training samples. While some methods leverage semantic knowledge from smaller-scale models to mitigate data scarcity, these approaches often introduce noise and bias due to the data's inherent simplicity. In this paper, we propose a novel framework, Synergistic Knowledge Transfer (SynTrans), which effectively transfers diverse and complementary knowledge from large multimodal models to empower the off-the-shelf few-shot learner. Specifically, SynTrans employs CLIP as a robust teacher and uses a few-shot vision encoder as a weak student, distilling semantic-aligned visual knowledge via an unsupervised proxy task. Subsequently, a training-free synergistic knowledge mining module facilitates collaboration among large multimodal models to extract high-quality semantic knowledge. Building upon this, a visual-semantic bridging module enables bi-directional knowledge transfer between visual and semantic spaces, transforming explicit visual and implicit semantic knowledge into category-specific classifier weights. Finally, SynTrans introduces a visual weight generator and a semantic weight reconstructor to adaptively construct optimal multimodal FSL classifiers. Experimental results on four FSL datasets demonstrate that SynTrans, even when paired with a simple few-shot vision encoder, significantly outperforms current state-of-the-art methods.