Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective

AI summary

Overview

Research area: Computer vision / machine learning — specifically semi-supervised few-shot learning (SSFSL), Vision-Language Model (VLM) finetuning, and retrieval augmentation. Technical level: Intermediate (assumes some familiarity with CLIP-style VLMs, pseudo-labeling, and semi-supervised learning). Scope: The paper diagnoses why standard semi-supervised learning methods fail when used to finetune a VLM for few-shot tasks, proposes a temperature-based fix, and introduces a stage-wise method (SWIFT) that exploits retrieved open data to reach state-of-the-art accuracy across five fine-grained recognition benchmarks.

What This Paper Is About

Semi-supervised few-shot learning aims to train a model from a small labeled set plus a large unlabeled set — the same situation as "auto-annotation," where you want a model to label new data for you. The authors test whether modern Vision-Language Models (like OpenCLIP) can be finetuned with established semi-supervised methods (e.g., FixMatch) and find that these methods actually perform worse than simply finetuning on the labeled data alone. The paper's goal is to explain that failure and deliver a simple method that makes VLM finetuning with unlabeled data work.

Key Contributions

1. A realistic SSFSL setup. The authors reframe SSFSL from an auto-annotation perspective, exploiting open-source VLMs (OpenCLIP ViT-B/32) and open-world data retrieved from LAION-400M — resources that prior SSFSL and SSL work largely ignored.
2. A failure diagnosis. They identify that VLMs produce "flat" distributions of softmax probabilities, which causes zero utilization of unlabeled data (pseudo-labels fall below the default 0.8 confidence threshold) and weak training supervision that blocks effective finetuning.
3. A temperature-based remedy. They show that a confidence temperature (T_conf) sharpens pseudo-label confidences and a loss temperature (T_loss) strengthens the cross-entropy supervision, and that both are compatible with existing FixMatch-style SSL methods.
4. The SWIFT method. They present Stage-Wise Finetuning with Temperatures (SWIFT), a three-stage pipeline that combines VLM finetuning, temperature control, and retrieved open data, outperforming prior FSL and SSL methods by more than 5 accuracy points across five benchmarks.

Main Findings

• Standard SSL methods fail to finetune VLMs. Averaged over five datasets with OpenCLIP ViT-B/32, FixMatch reaches 39.3 / 49.9 / 57.2 at 4-, 8-, and 16-shot, and DebiasPL reaches 39.6 / 49.8 / 57.1 — all well below the FSL baseline FS-FT, which uses only labeled data and scores 61.0 / 65.8 / 69.1. These are degradations of 11.9 to 21.7 accuracy points.
• The root cause is "flat" softmax probabilities. The authors show that VLM softmax outputs are low-confidence, which produces low-confidence pseudo-labels (zero utilization under the default threshold σ = 0.8) and weak learning signals that prevent effective finetuning.
• Temperatures fix the problem. Simply adding temperatures raises FixMatch to 57.7 (+18.4) / 65.7 (+15.8) / 71.2 (+14.0), and raises DebiasPL to 60.3 (+20.7) / 67.6 (+17.8) / 73.2 (+16.1). Both improved baselines surpass the SOTA SSL method FineSSL (57.6 / 64.6 / 68.9).
• SWIFT reaches state of the art. SWIFT scores 71.5 / 76.3 / 79.7 at 4-, 8-, and 16-shot, beating the SOTA FSL method SWAT (which also uses retrieval augmentation) by 4.1, 5.3, and 5.7 points respectively.
• SWIFT rivals fully supervised learning. A fully supervised reference that finetunes the VLM with ground-truth labels on the unlabeled data scores 74.8 / 75.4 / 76.0 (and 76.0 / 76.8 / 77.2 with retrieval augmentation), which SWIFT comes close to at 16-shot (79.7) and stays competitive with at lower shots.
• The stage-wise design matters. Ablation shows incremental gains over directly applying FixMatch: stage-1 classifier initialization gives +17.0 / +9.9 / +5.0, stage-2 semi-supervised finetuning gives +11.1 / +7.6 / +6.1, and stage-3 few-shot finetuning gives +2.7 / +3.0 / +2.4 (OpenCLIP, 4-/8-/16-shot).
• SWIFT generalizes beyond CLIP. On DINOv2 ViT-B/14, SWIFT reaches 78.2 / 84.4 / 87.8, improving over the FS-FT baseline on that backbone (56.5 / 71.1 / 79.6) and over DINOv2 FixMatch (50.2 / 66.3 / 77.1).
• Learning the loss temperature beats fixing it. The authors report that dynamically learning T_loss outperforms a fixed T_loss, and that confidence temperatures within a broad range of [0.001, 0.05] consistently yield strong gains — making temperature more robust than hand-tuning the confidence threshold σ.
• Better classifier initialization helps. In stage-1, initializing from text embeddings with T_loss gives 57.0 / 61.2 / 64.7 versus 54.9 / 57.6 / 59.9 with plain text text embeddings and 22.5 / 36.4 / 47.3 with random initialization. In stage-2, the stage-1 learned classifier gives 68.8 / 73.3 / 77.3 versus 66.0 / 71.4 / 77.0 (text) and 65.3 / 60.9 / 66.8 (random).

Methodology in Plain English

A downstream task provides C classes, with only K labeled images per class (K is 4, 8, or 16) plus a much larger unlabeled set. The authors start from OpenCLIP ViT-B/32, whose visual encoder and text encoder can already match images to class names. Following the FSL method FS-FT, they initialize a classifier using the text embeddings of class names, then encode images and compute logits and softmax probabilities.

The problem they uncover is that VLM softmax outputs are too "flat" — no class stands out — so a FixMatch-style pipeline that keeps only high-confidence pseudo-labels ends up keeping almost nothing, and the training signal is too weak. Their fix is to divide the logits by a temperature: a small confidence temperature (T_conf) before the softmax makes pseudo-labels confident enough to pass the 0.8 threshold, and a small loss temperature (T_loss) in the cross-entropy loss makes the gradients stronger. They initialize T_loss to 0.07 and learn it during training, and set T_conf to 0.01, choosing these values on the semi-Aves dataset and applying them unchanged everywhere else under a "validation-free" protocol.

SWIFT then trains in three stages: (1) learn a classifier on the few-shot labeled data with T_loss, (2) finetune the visual encoder and classifier semi-supervised on labeled, unlabeled, and retrieved data using a FixMatch-style loss with both temperatures, and (3) finetune again on the few-shot labeled data to counteract the noise, imbalance, and distribution shift of the retrieved open data. The retrieved data (500 images per class) comes from LAION-400M. Experiments run on a single NVIDIA RTX 4090 (24GB) GPU with 50 GB of disk space.

Why This Matters

This work closes a gap: FSL research had already embraced VLMs and open data, while SSL and SSFSL research had not. By showing why naive SSL finetuning of VLMs collapses and how a one-line temperature change repairs it, the paper makes a large, previously unusable class of resources available to semi-supervised practitioners.

Real-world applications, grounded in the datasets and framing the paper uses:

• Automated image annotation for fine-grained categories such as bird species (semi-Aves) and aircraft or car models (FGVC-Aircraft, Stanford Cars).
• Remote sensing and land-use mapping, using the EuroSAT satellite imagery benchmark, where labeled expert annotations are scarce.
• Material and texture recognition (DTD) for industrial inspection or product quality tasks.
• Low-label scientific or cataloging workflows generally, where a domain expert labels a handful of examples and the system must propagate labels to the rest.

Industry relevance: many deployed vision systems face exactly the "few labeled, many unlabeled" regime. The paper's claim that SWIFT, which uses no ground-truth labels for the unlabeled data, rivals a fully supervised reference is directly relevant to teams that want to reduce annotation cost.

Future Directions

• Validation without labels. The paper adopts a validation-free protocol and notes that future work could use the unlabeled and retrieved data themselves to aid validation, enabling per-task hyperparameter tuning.
• More realistic dataset diversity. The authors note that their benchmarks may not capture the extreme natural class imbalances found in real applications and suggest constructing more diverse datasets for SSFSL research.
• Generalizing the temperature insight. The paper shows temperatures help FixMatch and DebiasPL, but whether they transfer to other SSL families, adaptive-threshold methods, and other architectures is left as an open question.
• Mitigating side effects of retrieval. The authors flag that retrieved open data may cause overgeneralization to open-set or anomalous inputs, and that the finetuned model may inherit biases from the pretrained VLM — both noted as risks rather than solved problems.

Target Audience

Researchers and practitioners in computer vision and machine learning working on semi-supervised learning, few-shot learning, or VLM adaptation. It is especially useful for engineers building annotation-efficient pipelines who want to finetune open-source VLMs with a small labeled set and a larger unlabeled pool, and for students who want a clear case study in diagnosing why a standard baseline fails before proposing a fix.

Authors’ abstract

Semi-supervised few-shot learning (SSFSL) resembles real-world applications such as auto-annotation, as it aims to learn a model from a few labeled and abundant unlabeled task-specific examples to annotate the unlabeled ones. Despite the availability of powerful open-source Vision-Language Models (VLMs) and open-world data, existing SSFSL literature largely neglects these resources. In contrast, the related area few-shot learning (FSL) has already exploited them to boost performance. Arguably, to solve real-world auto-annotation, SSFSL should leverage such open resources. To bridge this gap, we explore established SSL methods to finetune a VLM. Unexpectedly, they significantly underperform FSL baselines that do not use unlabeled data. Our in-depth analysis reveals the root cause of failure: VLMs produce flat distributions of softmax probabilities, resulting in zero utilization of unlabeled data and weak supervision signals. To address this challenge, we propose an embarrassingly simple solution that uses temperatures to sharpen the softmax output, which not only increases the confidence scores of pseudo-labels to improve the utilization of unlabeled data, but also strengthens training supervision for effective finetuning. Furthermore, we exploit task-relevant open data, e.g., those retrieved from VLMs' publicly available pretraining set. To mitigate the imbalance and domain gaps in retrieved data, we employ a stage-wise training strategy. Building on the successful finetuning of VLMs and the exploitation of open data, we present a simple yet effective SSFSL method, Stage-Wise Finetuning with Temperatures (SWIFT). Across five benchmarks, SWIFT outperforms recent FSL and SSL methods by $\sim$5 accuracy points. SWIFT even rivals supervised learning, which finetunes a VLM assuming unlabeled data having ground-truth labels!