EfficientFSL: Enhancing Few-Shot Classification via Query-Only Tuning in Vision Transformers

AI summary

Overview

• Research area: Computer vision — few-shot learning (FSL) with Vision Transformers, specifically parameter-efficient transfer learning (PETL) for low-data classification.
• Technical level: Advanced. The paper assumes familiarity with Vision Transformers, attention mechanisms, Prototypical Networks, and parameter-efficient fine-tuning.
• Scope: The paper proposes a query-only fine-tuning framework, EfficientFSL, that freezes a pre-trained ViT entirely and trains a small set of modules to adapt it to few-shot classification tasks.

What This Paper Is About

Few-shot classification asks a model to recognize new categories from only a handful of labeled examples per class, so the quality of the model's learned features matters a great deal. Vision Transformers provide strong features, but adapting them by full fine-tuning demands large GPU memory, long training times, and a separate copy of all parameters per task. EfficientFSL addresses this by freezing the pre-trained ViT and learning only a lightweight querying module that pulls task-relevant information out of the model's intermediate layers, using a very small number of trainable parameters.

Key Contributions

1. A parameter-efficient fine-tuning framework for few-shot classification. EfficientFSL freezes the pre-trained ViT backbone and integrates its representations with task-specific knowledge using a Forward Block, a Combine Block, and Bottleneck-Structured Fully Connected Layers to keep the trainable parameter count low. It uses 1.25M trainable parameters with ViT-S and 2.48M with ViT-B.
2. A Support-Query Attention Block (SQ Attention Block). This component addresses the distribution shift between the support set and the query set by adjusting prototype positions so that they align with the center of the corresponding query distribution, rather than remaining at the plain mean of support features.
3. A query-only design that leaves the backbone untouched. Rather than inserting modules that modify the feature flow inside the frozen model, EfficientFSL introduces a lightweight Forward Block containing a trainable Active Block and a frozen sub-block, where the Active Block's output is used as queries against the pre-trained ViT's intermediate features as keys and values.
4. Extensive evaluation across ten benchmarks. The method is validated on 4 in-domain few-shot datasets and 6 cross-domain datasets, together with ablations of its individual components.

Main Findings

• In-domain results with ViT-S (1.25M parameters): miniImageNet 97.40 ± 0.47 (5-way 1-shot) and 99.05 ± 0.14 (5-way 5-shot); tieredImageNet 89.72 ± 0.96 and 95.41 ± 0.48; CIFAR 88.82 ± 0.86 and 94.60 ± 0.42; FC100 69.94 ± 1.16 and 81.68 ± 0.95. The authors state this surpasses the fully fine-tuned ViT-S (21.7M parameters), which scores 89.41 ± 0.88 / 95.59 ± 0.36 on miniImageNet, 77.87 ± 1.35 / 89.13 ± 0.74 on tieredImageNet, 84.86 ± 1.11 / 92.30 ± 0.62 on CIFAR, and 52.93 ± 1.29 / 66.70 ± 1.01 on FC100.

• In-domain results with ViT-B (2.48M parameters): Pre-trained on ImageNet-1K, Ours(1K) reaches miniImageNet 97.57 ± 0.48 / 98.96 ± 0.14, tieredImageNet 87.63 ± 1.08 / 93.13 ± 0.62, CIFAR 85.25 ± 1.21 / 92.64 ± 0.54, and FC100 72.60 ± 1.48 / 80.74 ± 1.04. Pre-trained on ImageNet-21K, Ours(21K) reaches miniImageNet 98.34 ± 0.30 / 99.12 ± 0.13, tieredImageNet 93.27 ± 0.75 / 96.78 ± 0.33, CIFAR 93.25 ± 0.69 / 97.28 ± 0.31, and FC100 80.13 ± 1.26 / 88.81 ± 0.81.

• Comparison with other PETL methods on FC100: With ViT-S, EfficientFSL scores 69.94 accuracy, versus LoRA 69.04, AdaptFormer 64.13, and Adapter 62.80. With ViT-B, EfficientFSL scores 80.13, versus LoRA 74.19, AdaptFormer 68.83, and Adapter 67.48.

• Efficiency in the PETL comparison (FC100, ViT-S): EfficientFSL reports 23.61 s/epoch train time, 0.49 GB peak memory, and 391.57 images/s inference speed, compared with Adapter (26.68, 0.50, 380.46), AdaptFormer (38.22, 0.57, 274.53), and LoRA (34.02, 0.49, 298.14). With ViT-B, EfficientFSL reports 59.62 s/epoch, 1.08 GB, and 134.29 images/s, against Adapter (74.19, 1.09, 134.21), AdaptFormer (76.87, 1.19, 129.98), and LoRA (76.35, 1.10, 131.84).

• Cross-domain evaluation: Using a ViT-S/16 pre-trained on ImageNet-1K and meta-learning on miniImageNet, the authors evaluate 5-way 1-shot and 5-way 5-shot performance on CUB, Stanford Cars, Places, Plantae, EuroSAT, and CropDiseases. The paper states that the approach outperforms existing state-of-the-art methods across all these benchmarks, with the results shown in Figure 2 rather than reported as numbers in the text.

• Component ablation (Table 3, FC100): The complete model (2.48M parameters) achieves 80.13 (1-shot) and 88.81 (5-shot). Removing the projection layer drops to 1.05M parameters and 72.15 / 87.81. Removing the attention and MLP modules drops to 1.58M parameters and 51.15 / 68.95. Removing the Combine Block gives 2.37M parameters and 75.55 / 88.60.

• Internal component ablation (Table 4): Removing the prompt P_i drops performance to 39.36 / 60.99 at 2.44M parameters. Removing Att gives 75.98 / 87.83 (1.95M), and removing MLP gives 55.09 / 69.99 (1.56M). Within the Combine Block, removing F_i^att gives 78.31 / 89.71, removing F_i^mlp gives 73.26 / 87.52, and removing H_i gives 77.77 / 89.20, all at 2.47M parameters.

• SQ Attention Block ablation (Table 5): Removing it (2.48M) yields miniImageNet 95.95 / 98.97, tieredImageNet 89.78 / 96.61, CIFAR 90.15 / 96.99, and FC100 75.42 / 87.82. Removing the projection on q (2.48M) yields 98.34 / 99.12, 93.27 / 96.78, 93.25 / 97.28, and 80.13 / 88.81. The full model with the SQ Attention Block (2.55M) yields 98.49 / 99.23, 93.85 / 96.89, 93.34 / 97.28, and 80.20 / 89.53.

• Feature aggregation ablation (Table 6, FC100): Simple averaging gives 69.58 ± 1.22 / 85.38 ± 0.83 at 2.45M parameters; fixed weights give 69.97 ± 1.22 / 85.49 ± 0.84 at 2.45M; conditional weights give 80.13 ± 1.26 / 88.81 ± 0.81 at 2.48M.

• Qualitative evidence: t-SNE visualizations show that incrementally adding the prompt P_i, the projection, the Active Block, the Combine Block, and the SQ Attention Block progressively produces tighter, better-separated clusters. A further t-SNE figure shows the SQ Attention Block moving prototypes closer to the distribution center of the corresponding query samples.

Methodology in Plain English

The authors start from a Vision Transformer pre-trained on a large dataset and never update its weights — the backbone stays frozen throughout.

On top of it they attach a set of small modules, one per layer of the backbone. Each module has two parts:

• An Active Block, trained from scratch, which adds a learnable prompt to the previous layer's output, projects it through a narrow bottleneck layer, then runs it through a self-attention layer and a small MLP. This is where task-specific knowledge is learned. The authors use narrow bottleneck fully connected layers liberally to keep the parameter count down.
• A Frozen Block, which takes the Active Block's output as queries and uses the frozen ViT's intermediate layer features as keys and values. Nothing here is trained; it reuses the pre-trained model's existing knowledge. The outputs from the attention module, the fully connected layer, and their sum are all kept for later use.

A Combine Block then aligns all these hierarchical features through a shared bottleneck MLP and merges them with adaptive weights. The weights are generated from the final layer's hidden state through a small weighting MLP and a softmax, so the fusion is conditioned on the input rather than fixed.

Classification uses a Prototypical Network head: class prototypes are the mean of support features. To correct for the fact that support and query images differ slightly in background, lighting, or shooting angle, the Support-Query Attention Block adjusts each prototype by attending it to the projected query features, blending the original prototype with the adjusted one via a mixing factor α. Cosine similarity between each query and the updated prototypes gives the prediction.

Training details: the scale factors ξ, ζ, and α are tuned within the range {0.1, 1}; hidden size of the bottleneck projections is 48 and of the query/key/value attention bottlenecks in the Active Block is 8; AdamW with a cosine learning rate scheduler, batch size 64, learning rate 0.0001 for 5 epochs; evaluation on 320 batches with 95% confidence intervals; images resized to 256 × 256 and center-cropped to 224 × 224; all experiments on a single NVIDIA V100 GPU. Backbones tested are ViT-S/16 and ViT-B/16 pre-trained on ImageNet-1K, and ViT-B/16 pre-trained on ImageNet-21K.

Why This Matters

Impact on research: The work sits at the intersection of two active lines of work — few-shot learning and parameter-efficient transfer learning — and argues that a PETL method can beat full fine-tuning of the same backbone while using a small fraction of the parameters. It also offers a specific mechanism (prototype adjustment through query attention) for the widely observed support-query distribution shift in prototype-based methods.

Real-world applications:

• Rare species or plant identification (reflected in the Plantae and CUB benchmarks used), where only a few labeled images exist per category.
• Agricultural disease diagnosis (reflected in the CropDiseases benchmark), where labeled examples are scarce and field images vary in lighting and angle.
• Remote sensing and land-use classification (reflected in the EuroSAT benchmark), where new categories appear with limited annotation.
• Deploying vision models on low-resource or edge hardware, where per-task storage of full model copies and long training runs are impractical.

Industry relevance: The combination of low trainable parameter counts (1.25M for ViT-S and 2.48M for ViT-B), fast training, competitive memory, and high inference speed makes the approach attractive for teams that must adapt one pre-trained model to many narrow tasks without provisioning large GPU budgets. The storage argument is also direct: full fine-tuning requires storing a complete parameter set per task.

Future Directions

• Extending the query-only paradigm to other backbones and modalities. The paper tests ViT-S/16 and ViT-B/16; whether the same decoupled design transfers to other architectures or to non-vision data is not explored.
• Deeper analysis of the SQ Attention Block. The ablation shows a gain, but the reported full-model improvement over the no-projection variant is small on several datasets; a more systematic study of when prototype adjustment helps most would clarify its role.
• Scaling the number of shots and classes. All evaluations use 5-way 1-shot and 5-shot settings; behavior under larger N-way or higher-shot regimes is not reported.
• Broader domain-shift evaluation. Cross-domain results are presented in Figure 2 rather than broken out numerically in the text, and the paper does not quantify how much of the cross-domain gain comes from each module.

Target Audience

Researchers and practitioners working on few-shot learning, parameter-efficient fine-tuning, and Vision Transformer adaptation. It is most useful to those who already understand prototype-based classification and attention, and who need a concrete, low-parameter recipe for adapting a frozen pre-trained ViT to new classification tasks with limited labeled data.

Authors’ abstract

Large models such as Vision Transformers (ViTs) have demonstrated remarkable superiority over smaller architectures like ResNet in few-shot classification, owing to their powerful representational capacity. However, fine-tuning such large models demands extensive GPU memory and prolonged training time, making them impractical for many real-world low-resource scenarios. To bridge this gap, we propose EfficientFSL, a query-only fine-tuning framework tailored specifically for few-shot classification with ViT, which achieves competitive performance while significantly reducing computational overhead. EfficientFSL fully leverages the knowledge embedded in the pre-trained model and its strong comprehension ability, achieving high classification accuracy with an extremely small number of tunable parameters. Specifically, we introduce a lightweight trainable Forward Block to synthesize task-specific queries that extract informative features from the intermediate representations of the pre-trained model in a query-only manner. We further propose a Combine Block to fuse multi-layer outputs, enhancing the depth and robustness of feature representations. Finally, a Support-Query Attention Block mitigates distribution shift by adjusting prototypes to align with the query set distribution. With minimal trainable parameters, EfficientFSL achieves state-of-the-art performance on four in-domain few-shot datasets and six cross-domain datasets, demonstrating its effectiveness in real-world applications.