Logit-Based Losses Limit the Effectiveness of Feature Knowledge Distillation

AI summary

Overview

• Research area: Computer Vision, specifically knowledge distillation (KD) for model compression, with connections to the geometry of neural network latent representations.
• Technical level: Intermediate. The core idea is intuitive, but the paper introduces a geometric quality metric built from cosine similarities, SVD entropy, and hyperspherical packing arguments, plus a theoretical appendix.
• Scope: The paper (arXiv:2511.14981v1, cs.CV, 18 Nov 2025, by Nicholas Cooper, Lijun Chen, Sailesh Dwivedy, and Danna Gurari of the University of Colorado Boulder) proposes training a student model's backbone using feature-based losses exclusively, selecting which teacher layers to distill from with a new "knowledge quality" metric, and evaluates this on three image classification datasets with four student-teacher pairs spanning CNNs and vision transformers.

What This Paper Is About

Feature knowledge distillation (FKD) normally combines two kinds of supervision: logit-based losses (cross entropy and KL divergence between the softened class scores of teacher and student) and feature-based losses that pull the student's intermediate representations toward the teacher's. The authors hypothesize that logit-based losses are actively holding FKD back, because logits live in a low-dimensional space (the number of classes) while intermediate features live in a very high-dimensional space and can therefore carry far richer information. The goal is to show that training the student backbone with feature losses alone, paired with a smarter way of choosing which teacher layers to distill from, produces better students than the standard recipe.

Key Contributions

1. A loss recipe with no logit losses in the backbone. The KL divergence term is dropped entirely, and cross entropy is applied only to the student's classifier, not back-propagated through the backbone. The backbone is trained purely to minimize feature-based losses.
2. A geometry-aware knowledge quality (KQ) metric for teacher layer selection. Denoted Q, it combines three properties of a layer's representations: separation (S), information (I), and efficiency (E), which the authors describe as the first automated metric for teacher layer selection.
3. Experiments across datasets and architectures. Three image classification datasets and four student-teacher pairs covering VGG, ResNet, MobileNet, and ViT families, showing gains of up to 15% top-1 accuracy over standard approaches.
4. Fine-grained analyses isolating each design choice. Ablations over the loss recipe, the layer selection method, and each term of the KQ metric, plus validation with longer training, a different temperature, and data augmentation.

Main Findings

• Removing logit losses from the backbone helps. Training the student backbone with feature-based losses exclusively achieved state-of-the-art performance against seven baselines: Vanilla KD, Logit Standardization, Base FKD (a modernization of FitNets), Base FKD + FC, Similarity Preserving, SemCKD, and Reused Teacher Classifier (SimKD). Reported gains reach up to 15% top-1 accuracy.
• Harder datasets give bigger relative gains. Absolute relative improvement (ARI) scores were highest on Tiny ImageNet, the most difficult of the three datasets, and a positive correlation was observed between dataset difficulty and the benefit of the method. The authors suggest harder tasks yield teacher representations with higher intrinsic dimensionality.
• Transformers benefit less than CNNs. The method produced similar absolute gains but smaller ARIs on the ViT pair. Logit-loss-based methods performed better by roughly 2x on that pair, which the authors connect to ViT_B's knowledge quality increasing almost monotonically across layers (so its final layers still hold good knowledge), unlike CNNs, whose knowledge quality degrades in final layers.
• Layer selection is critical without logit losses. When comparing loss recipes and layer selection methods, the student failed to converge in 6 of 12 cases using standard layer selection. With logit losses present, the layer selection method mattered less, suggesting those losses mask poor-quality teacher layers.
• Two exceptions suggest room for improvement. Standard layer selection with logit losses excluded outperformed the proposed approach on the VGG model pair on CIFAR10 and CIFAR100.
• All three metric components are necessary. Ablating the KQ metric, the only strategy that always converged was the full Q metric. Using only S, I, or the sqrt(I·E) term failed in 4 of 12 cases, and using only E failed in 5 cases. VGG19 was the most sensitive teacher, with 3 of 4 modifications failing to converge; ResNet34 and ViT_B tolerated a wider range of selections. Even when ablated variants converged, they trailed the full method by 0.5–10%.
• Peak knowledge quality occurs at different depths per model. Analysis on CIFAR100 showed S, I, and E exhibiting clear extraction and compression phases in both ResNet34 and ViT_B, with separation rising in final layers at the cost of information and efficiency. Q peaked at different relative depths, and only ResNet34 showed knowledge quality decay in its final layers.
• Training converges quickly. The method often surpassed the non-distillation student's accuracy after only 10–30 epochs, behavior the authors say is distinctly different from all baselines.
• The advantage persists under other protocols. With a "standard" 240-epoch SGD schedule (about 5x the training time of the paper's 50-epoch protocol), no baseline surpassed the proposed method. A temperature of T = 4 was superior to T = 2. With data augmentation, the same conclusions held, and the appendices report a new best recorded result for MobileNetV2 on CIFAR100 at 73.36% accuracy.

Methodology in Plain English

A teacher network is a large, accurate model; a student is a small one that should learn to imitate it. Normally the student is pushed to match both the teacher's final class scores (logits) and its intermediate feature maps. The authors instead split responsibility: the student's classifier is trained only with cross entropy on the labels, while everything before the classifier (the backbone) is trained only to match the teacher's intermediate features. This means logit information never flows backward through the backbone.

The second piece is deciding which teacher layers to learn from. Most prior work simply takes the last layer of each "stage" (the layers before a pooling operation). The authors instead score every candidate layer using a knowledge quality metric Q, combining three geometric measurements computed on the training set's representations:

• Separation (S): average within-class cosine similarity minus average between-class cosine similarity, measuring how well the layer distinguishes classes.
• Information (I): one minus the minimum within-class cosine similarity, multiplied by the average class-wise normalized SVD entropy, capturing the richness and effective dimensionality of the representation.
• Efficiency (E): a ratio derived from a hyperspherical packing argument, comparing the empirical average norm to the smallest norm estimated to be needed to keep classes separated, penalizing unnecessarily inflated representations that destabilize training.

The metric is the sum S + sqrt(I·E). The top-scoring teacher layers are selected, matched one-to-one to four student layers (|L^T| = |L^S| = 4), and connected through a single-layer projector (spatial pooling plus a bias-free convolution, a 2D version for CNNs and a 1D version for transformers). Models were trained for 50 epochs with Adam and a one-cycle learning rate schedule, without data augmentation in the main experiments so that knowledge quality could be measured reproducibly.

Why This Matters

Impact on research. The paper challenges a long-standing assumption in knowledge distillation, namely that feature-based losses should always be augmented with logit-based ones. If a student backbone learns better from features alone, then a large body of FKD work is leaving performance on the table, and the "what knowledge should be transferred" question gains a new, geometry-grounded answer. The knowledge quality metric also gives the layer-selection step, previously the least studied of the three FKD steps, a concrete, automatic solution.

Real-world applications (extrapolations from the method's purpose, since the paper evaluates only image classification):

• Deploying compact image classifiers on phones, cameras, drones, and other devices with tight memory and compute budgets.
• Compressing large vision models for embedded or edge inference where latency and power matter.
• Building efficient backbones for downstream vision pipelines that need faster training and smaller footprints.
• Any classification setting the authors identify as analogous, such as text or audio, if the method transfers.

Industry relevance. Model compression directly affects deployment cost. A recipe that produces more accurate small models, converges in as few as 10–30 epochs to beat an undistilled student, and matches or beats baselines even when those baselines are given roughly 5x the training time, is relevant to teams that pay for training compute and inference hardware. The result that logit losses can mask poor teacher-layer choices also gives practitioners a diagnostic for why a distillation run may underperform.

Future Directions

• Scaling to larger data. The authors did not experiment with ImageNet because of the computational cost of their geometric analysis, and they name investigating the scaling properties of the method as valuable future work.
• Beyond classification. Image segmentation has no notion of logits, so the loss-recipe question there is different; the authors suggest exploring analogous questions in other computer vision tasks and in text and audio classification.
• A principled formulation of the metric. Q was assembled from S, I, and E based on empirical evidence, and the authors suspect a theoretical analysis could improve it. They also want to study how S, I, and E relate to model generalization outside of KD.
• Student-side layer selection and further layer-selection gains. The paper studies teacher layer selection; how student layer selection influences distillation is listed as an open question, and the two VGG cases where a simpler recipe beat the proposed method indicate the layer selection algorithm can still be improved.

Target Audience

Researchers and graduate students working on knowledge distillation, model compression, or the geometry of neural representations will get the most from this paper, along with practitioners who train small vision models for deployment and want a concrete alternative to the standard logit-plus-feature loss recipe. Readers should be comfortable with standard deep learning concepts such as logits, softmax, cross entropy, KL divergence, and distillation; the geometric metric and the hyperspherical packing proof in the appendix suit a more mathematically inclined reader.

Authors’ abstract

Knowledge distillation (KD) methods can transfer knowledge of a parameter-heavy teacher model to a light-weight student model. The status quo for feature KD methods is to utilize loss functions based on logits (i.e., pre-softmax class scores) and intermediate layer features (i.e., latent representations). Unlike previous approaches, we propose a feature KD framework for training the student's backbone using feature-based losses exclusively (i.e., without logit-based losses such as cross entropy). Leveraging recent discoveries about the geometry of latent representations, we introduce a knowledge quality metric for identifying which teacher layers provide the most effective knowledge for distillation. Experiments on three image classification datasets with four diverse student-teacher pairs, spanning convolutional neural networks and vision transformers, demonstrate our KD method achieves state-of-the-art performance, delivering top-1 accuracy boosts of up to 15% over standard approaches. We publically share our code to facilitate future work at https://github.com/Thegolfingocto/KD_wo_CE.