PAND: Prompt-Aware Neighborhood Distillation for Lightweight Fine-Grained Visual Classification

AI summary

Overview

• Research area: Computer vision, specifically knowledge distillation from large vision-language models (VLMs) into lightweight image classifiers for Fine-Grained Visual Classification (FGVC).
• Technical level: Advanced. The paper assumes familiarity with knowledge distillation, CLIP-style contrastive vision-language models, prompt learning (CoOp), and divergence-based relational losses.
• Scope: The paper proposes PAND, a two-stage framework that (1) learns task-adaptive prompt-based "semantic anchors" from a frozen CLIP teacher and (2) distills local sample-pair logit relations from that teacher into ResNet-18 and MobileNet-V2 students, evaluated on four FGVC benchmarks.

What This Paper Is About

Distilling a large vision-language model such as CLIP into a small network is a practical way to get strong recognition at low cost, but existing methods (e.g., VL2Lite) rely on fixed hand-crafted prompts like "a photo of a [CLASS]" and only align global features or logits. The authors argue this leaves two gaps: a semantic gap (generic prompts cannot capture subtle differences between fine-grained categories) and a structural gap (global alignment ignores the local relationships between similar samples that actually determine fine-grained decision boundaries). PAND's goal is to close both gaps by calibrating the teacher's semantic space with learned prompts and then forcing the student to reproduce the teacher's local neighborhood relations.

Key Contributions

1. PAND, a two-stage framework that decouples semantic calibration from structural transfer: Stage-PSC (Prompt-Aware Semantic Calibration) builds task-adaptive semantic anchors from a frozen VLM, and Stage-NSD (Neighborhood-Aware Structural Distillation) trains the lightweight student against them.
2. Prompt-Aware Semantic Calibration, which applies CoOp-style learnable context tokens on a frozen image and text encoder to produce calibrated, task-specific text features used as fixed semantic anchors for distillation.
3. A sample-level neighborhood-aware structural distillation module that selects each sample's Top-K nearest neighbors in the teacher's logit space and aligns the teacher and student distributions of sample-pair logit differences using Jensen-Shannon divergence — without modifying the student architecture.
4. Empirical validation on four FGVC benchmarks, reporting consistent gains over KD, RKD, RISE, and VL2Lite, with the ResNet-18 student reaching 76.09% on CUB-200 (3.42 percentage points above VL2Lite) and the MobileNet-V2 student improving on FGVC-Aircraft by 5.7 points over VL2Lite.

Main Findings

• ResNet-18 on CUB-200: PAND reaches 76.09% Top-1 accuracy, beating the no-KD baseline (64.48%) by 11.61 points and VL2Lite (72.67%) by 3.42 points.
• ResNet-18 across all four datasets: PAND scores 88.97% on Oxford Pets (VL2Lite: 88.56%), 63.25% on FGVC-Aircraft (VL2Lite: 60.82%), and 74.98% on Stanford Dogs (VL2Lite: 73.14%). The authors note the Oxford Pets improvement over VL2Lite is about 0.4 points, on a nearly saturated dataset.
• MobileNet-V2 results: PAND achieves 76.52% on CUB-200 (VL2Lite: 72.19%, a 4.33-point gain), 64.75% on FGVC-Aircraft (VL2Lite: 58.99%, a 5.7-point gain), 88.28% on Oxford Pets (VL2Lite: 87.55%), and 74.52% on Stanford Dogs (VL2Lite: 73.28%).
• PAND beats standard KD and RKD: On CUB-200 with ResNet-18, KD scores 70.95% and RKD 68.31%; with MobileNet-V2, KD scores 68.00%, RKD 67.95%, and RISE 67.51% — all below PAND's 76.52%.
• Ablation shows both stages help, and NSD helps more: Starting from the VL2Lite baseline at 72.67%, adding only PSC gives 73.52%, adding only NSD gives 75.91%, and adding both (full PAND) gives 76.09%.
• PSC and NSD are complementary: The paper attributes NSD's larger standalone gain to the value of modeling local sample-pair logit relations, while PSC provides an additional gain, which the authors interpret as calibrated anchors further benefiting neighborhood-aware relation transfer.
• Sensitivity to the structural loss weight: Accuracy improves from λ_NSD = 0 up to 76.41% at λ_NSD = 0.5, and further increases do not consistently improve performance. Note that the main experiments report λ_NSD = 1.0; the paper does not reconcile the main-result setting with the sensitivity-analysis optimum of 0.5.
• Qualitative evidence via t-SNE: On MobileNet-V2/FGVC-Aircraft and ResNet-18/CUB-200, the no-KD models show scattered, overlapping features; VL2Lite produces clearer clusters but still entangles visually similar classes; PAND yields more compact clusters with visible margins between classes.

Methodology in Plain English

PAND trains in two separate passes rather than end-to-end.

Stage 1 — fixing the teacher's words. Instead of hand-writing a prompt for every class, the authors use CoOp to learn 16 continuous context vectors that are prepended to the fixed class-name embedding. Both CLIP encoders (image and text) stay frozen; only these context tokens are optimized, using an image-to-text cross-entropy loss that pulls each image toward its correct class text feature. Training uses SGD with learning rate 0.002, momentum 0.9, weight decay 0, batch size 128, for 200 epochs. The resulting text features become fixed "semantic anchors."

Stage 2 — teaching a small student the local structure. The teacher is now the frozen CLIP image encoder plus the frozen calibrated anchors; its logits are cosine similarities between the image feature and those anchors. The student (ResNet-18 or MobileNet-V2, ImageNet-1k initialized) is trained with the VL2Lite base loss — classification plus visual and textual alignment terms, weighted λ_cls = 0.01 and λ_vis = λ_txt = 0.495 — plus the new NSD loss.

For the NSD loss, each mini-batch sample finds its Top-K (K = 3) most cosine-similar peers under the teacher's logits, excluding itself. For each query-neighbor pair, the relation is represented not by one similarity number but by the full vector difference between their logits, passed through a softmax with temperature τ_r = 1.0. The teacher's relation distribution and the student's relation distribution are then compared with Jensen-Shannon divergence and averaged over the batch. The total objective is the base loss plus λ_NSD times this NSD loss, with λ_NSD = 1.0 and τ = 2.0 in the main experiments. Stage 2 trains for 300 epochs with AdamW (learning rate 1×10⁻⁴, weight decay 1×10⁻⁴) and cosine annealing down to a minimum learning rate of 1×10⁻⁵. Implementation is in PyTorch on four NVIDIA V100 GPUs.

The authors keep the stages separate deliberately: joint training would feed noisy gradients from a randomly initialized student back into prompt learning, destabilizing it. Freezing the anchors after Stage 1 gives the student a stationary target.

Why This Matters

Impact on research. The paper reframes VLM-to-small-model distillation as two distinct problems — calibrating the teacher's semantic space and transferring its local relational structure — rather than one global alignment problem. It also shows that prompt learning, usually used to improve a VLM's own inference, can instead be repurposed as a teacher-preparation step, and it brings neighborhood-based relational distillation (previously studied for unimodal teachers) into the vision-language setting.

Real-world applications:

• Mobile or embedded fine-grained recognition, such as identifying bird species (CUB-200), cat and dog breeds (Oxford-IIIT Pet), or dog breeds (Stanford Dogs) on-device.
• Aircraft variant identification (FGVC-Aircraft), relevant to aviation inspection, maintenance, and remote sensing analysis.
• Any deployment where a large VLM cannot run but its accuracy is still needed — for example, retail product or produce grading, or field biology tools.
• Robotics and edge vision systems where inference latency and memory budgets rule out CLIP-scale models.

Industry relevance. The method requires no architectural change to the student and only adds a training-time loss, which makes it a drop-in addition to existing distillation pipelines. Because ResNet-18 and MobileNet-V2 are standard deployment backbones, the reported gains translate directly into accuracy improvements at unchanged inference cost. Code is released at https://github.com/LLLVTA/PAND.

Future Directions

• Reconcile the loss-weight discrepancy. The sensitivity analysis finds the best accuracy (76.41%) at λ_NSD = 0.5, while the main results use λ_NSD = 1.0 (76.09%). Reporting main results at the sensitivity optimum, or explaining why 1.0 was kept, would clarify the design guidance.
• Test other teachers and students. Only CLIP ConvNeXt-XXL (laion2b_s34b_b82k_augreg_soup) as teacher and ResNet-18 / MobileNet-V2 as students are reported; generalization to other VLMs, larger student backbones, or transformer-based students is not explored.
• Examine K and τ_r more thoroughly. The paper fixes K = 3 and τ_r = 1.0 and reports a sensitivity study only for λ_NSD; how neighborhood size and relation temperature interact with dataset granularity remains an open question.
• Address the two-stage cost. Decoupling adds an extra training stage with its own 200-epoch prompt-learning phase. Whether the calibration can be made cheaper, or stabilized enough for single-stage joint training, is not investigated.

Target Audience

Researchers and practitioners working on knowledge distillation, vision-language model compression, or efficient fine-grained visual classification will benefit most. It is also relevant to engineers deploying compact classifiers on edge hardware who want improved accuracy without changing inference-time models. Readers should already be comfortable with CLIP-style contrastive training, prompt learning, and distillation losses; the paper is not an introductory treatment of any of these topics.

Authors’ abstract

Distilling knowledge from large Vision-Language Models (VLMs) into lightweight networks is crucial yet challenging in Fine-Grained Visual Classification (FGVC), due to the reliance on fixed prompts and global alignment. To address this, we propose PAND (Prompt-Aware Neighborhood Distillation), a two-stage framework that decouples semantic calibration from structural transfer. First, we incorporate Prompt-Aware Semantic Calibration to generate adaptive semantic anchors. Second, we introduce a neighborhood-aware structural distillation strategy to constrain the student's local decision structure. PAND consistently outperforms state-of-the-art methods on four FGVC benchmarks. Notably, our ResNet-18 student achieves 76.09% accuracy on CUB-200, surpassing the strong baseline VL2Lite by 3.4%. Code is available at https://github.com/LLLVTA/PAND.