You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces

AI summary

Overview

Research area: Computer vision / knowledge distillation, specifically cross-architecture knowledge distillation (transferring knowledge between structurally different networks such as CNNs, Vision Transformers, and MLP-based models).

Technical level: Intermediate. The paper assumes familiarity with standard knowledge distillation terminology (logits, soft targets, feature maps) but its central idea—comparing two networks through a shared, randomly initialized reference space—is conceptually accessible.

Scope: The paper proposes RPKD, a distillation framework that replaces learned teacher–student feature alignment with similarity matching in a frozen random prototype space, and evaluates it across 10 CIFAR-100 pairs, 12 ImageNet-100 pairs, 4 ImageNet-1K cross-architecture pairs, 2 homogeneous CNN pairs, and ImageNet subsets of varying class count.

What This Paper Is About

Most knowledge distillation methods assume the teacher and student share a compatible feature space, which breaks down when, for example, a large pretrained Vision Transformer must teach a small CNN. Existing solutions insert learned alignment modules that are architecture-specific, pair-dependent, and—when anchored to the class logit space—lose supervisory richness as the number of task classes shrinks. This paper asks whether the shared alignment space needs to be learned at all, and shows that a bank of randomly initialized, frozen prototype vectors, fixed before training begins, provides an effective and reusable reference frame for transferring knowledge between heterogeneous networks.

Key Contributions

1. RPKD framework: A cross-architecture distillation method built on a frozen random prototype space. The shared prototype bank and all teacher-side projections are fixed at initialization, and no design choice depends on the number of task classes.

2. Dual-path objective: A logit branch that aligns global prototype similarity distributions (using temperature-scaled KL divergence with per-sample logit standardization borrowed from LSKD) and a feature branch that matches spatially resolved prototype activation maps via cosine similarity, both operating inside the same fixed prototype space.

3. Low-category robustness: An empirical and theoretical argument for why a prototype vocabulary decoupled from the label space offers richer, more stable supervision than class-logit-based transfer when class counts are small, supported by ablations on ImageNet subsets of 10, 20, 50, and 100 classes.

4. State-of-the-art comparisons: Reported gains of up to +7.98% over OFA on CIFAR-100, +12.44% on ImageNet-100, and +1.28% on ImageNet-1K, with results across CNN, ViT, and MLP architectural families and few-class subsets.

Main Findings

• Large CIFAR-100 gains: Across 10 teacher–student pairs, RPKD achieves a maximum gain of +7.98% over OFA. On Swin-T → ResNet18 it reaches 85.15 ± 0.18 (versus OFA 81.61, RSD 83.92, FBT 80.54), and on Swin-T → MobileNetV2 it reaches 84.90 ± 0.04, corresponding to gains of +4.61% and +3.92% over OFA respectively.

• Strongest where the structural gap is largest: On ConvNeXt-T → DeiT-T, RPKD reaches 83.74 ± 0.25, which is +7.98 pp over OFA and +1.28 pp over RSD. The authors read this as evidence that prototype-based transfer succeeds by avoiding model-specific structural assumptions.

• Direct feature matching can collapse: FitNet drops to 24.06% on ConvNeXt-T → Swin-P when feature spaces are incompatible, illustrating the fragility the paper targets.

• ImageNet-100 breadth: Over 12 pairs, RPKD ranks first on 8 of 12, exceeds OFA on 11 of 12, and improves over the student baseline on every pair. It reaches 81.00 ± 0.12 on ResNet50 → DeiT-T (+12.44% over OFA) and 79.14 ± 0.26 on ConvNeXt-T → DeiT-T (+11.76% over OFA).

• Specific failure modes: The four ImageNet-100 pairs RPKD does not win are all won by RSD, and all involve an MLP-Mixer teacher or the Swin-P / Swin-N students. The authors attribute this to token-mixing weakening the localized positional correspondence that the feature branch relies on.

• ImageNet-1K: RPKD exceeds OFA on all four cross-architecture pairs and leads overall on DeiT-T → MobileNetV2 at 72.67 ± 0.11 (+1.28% over OFA and +0.49% over RSD). On the other three pairs it stays within 0.32 pp of the better of FBT and RSD, which the authors describe as broadly comparable rather than a clear lead.

• Works in the homogeneous setting too: On ResNet34 → ResNet18, RPKD reaches 72.23 ± 0.13, with OFA at 72.10 and RSD at 72.18, and FBT ahead at 72.29—a spread the authors explicitly call a tie given their own ±0.13 variation. On ResNet50 → MobileNetV1, RPKD reaches 74.20 ± 0.09, surpassing FBT (73.45) by +0.75 pp.

• The low-category advantage is the distinctive result: On the 10-class ImageNet subset, both CRD (88.60) and DIST (89.80) fall below the from-scratch student (90.80), while RPKD reaches 95.40—a +4.60% gain over the student and +2.80% over OFA. As class count drops from 100 to 10, KD's gain over the student shrinks from +2.74 to +0.60 pp and OFA's from +3.88 to +1.80 pp, while RPKD's stays between +6.29 and +4.60 pp, widening its relative advantage from roughly 1.5–1.6× at 50–100 classes to 2.3–2.6× at 10–20.

• The two branches are complementary: Each branch alone is competitive—RPKD_feat ranks second at IN-10 and IN-20, RPKD_logit at IN-50 and IN-100—but the combined method outperforms both at every subset size.

• Final-stage features are best: Restricting distillation to deeper stages improves monotonically on IN-100 (DeiT-T → ResNet18): all four stages 86.30%, stages {2,3,4} 86.80%, stages {3,4} 87.65%, final stage alone 87.87%.

• Hyperparameters are not fragile: A 16× sweep of prototype count M and embedding dimension R spans only 0.22 and 0.27 pp for the CNN student (within its ±0.24 pp training noise) and 0.84 and 1.27 pp for the ViT student, whose only notable preference is R = 512. RPKD retains its margin over OFA at every setting (≥ +2.14 pp and ≥ +11.80 pp respectively), and the same defaults were reused across all 28 pairs.

• Ablation on frozen versus learned prototypes is not available in the supplied text: The paper's Table 8 begins this comparison but the content is truncated before any numbers appear, so its outcome is not reported here.

Methodology in Plain English

The core move is to stop trying to make the teacher's features and the student's features speak the same language. Instead, both networks are asked to describe their inputs using the same fixed set of reference directions.

That reference set is a matrix P with M rows (M = 500 by default), each a randomly drawn, L2-normalized vector of dimension R = 256, frozen at initialization. It has nothing to do with class labels.

The teacher is pushed into this space by two fixed random linear projections—one for its global feature, one for its spatial features—with weight standard deviation 0.02. These are never trained. The student uses two small learnable two-layer MLPs with GELU activations to project its own features into the same space. This asymmetry is deliberate: the teacher's side of the comparison never moves, so the student chases a stable target instead of a co-adapting partner.

Two losses operate in that space. The logit branch takes the global feature, projects it, L2-normalizes, computes cosine similarities to all M prototypes, standardizes those scores per sample, and matches the student's distribution to the teacher's with temperature-scaled KL divergence (T = 4.0). The feature branch does the same per spatial token, reshapes each prototype's activations into a 7×7 map (49 positions), standardizes each map over its own spatial extent, and aligns teacher and student maps with a cosine loss—which, after centring, behaves as a Pearson correlation between activation patterns. For Swin students the standardization is dropped because it destabilizes training.

Feature routing is handled per architecture: global average pooling for CNNs, the CLS token for ViT-style models, and mean pooling over output tokens for Swin, MLP-Mixer, and ResMLP. Spatial features are taken from the final stage only. The total loss adds both distillation terms to standard cross-entropy, and gradients flow only into the student encoder and its projection heads.

The theoretical justification rests on the Johnson–Lindenstrauss lemma: a random linear map approximately preserves pairwise inner products between unit-norm vectors with high probability, so the teacher's inter-sample similarity structure survives the projection, and the student can learn that geometry without ever seeing the teacher's native feature space.

Why This Matters

Impact on research: The paper challenges a working assumption that runs through most of the feature-based distillation literature—that teacher and student must occupy an aligned space, and that alignment must be learned. If a fixed random space suffices, then alignment modules are not where the value lies, and the field's default of designing pair-specific projectors becomes optional rather than necessary. The class-count analysis also identifies a concrete weakness in logit-anchored cross-architecture methods that prior benchmarks largely hid.

Real-world applications:

• Deployment of pretrained ViTs onto edge hardware: An organization with a large pretrained transformer but CNN-only deployment targets can distill without a same-family teacher or a bespoke alignment design.
• Domain-specific datasets with few categories: Medical imaging, industrial defect inspection, and other settings where the label space is naturally small are exactly where logit-anchored methods lose their supervisory signal and RPKD's advantage grows.
• Model compression pipelines: RPKD slots alongside pruning, quantization, and low-rank decomposition as a compression step, and works in standard CNN-to-CNN settings as well.
• Rapid model portfolio maintenance: Because the objective and prototype space need no redesign per pair and the teacher side needs no training, adding a new teacher–student combination is close to a drop-in operation.

Industry relevance: The practical appeal is engineering cost. A distillation setup that is configured once and reused across heterogeneous model pairs—with a teacher side that requires zero training—removes a recurring design and tuning burden from model-compression workflows.

Future Directions

• Addressing the MLP-Mixer and Swin-P/Swin-N weakness: RPKD loses on the pairs involving MLP-Mixer teachers or these compact Swin students. Whether a feature branch less dependent on localized spatial correspondence could close this gap is an open question the paper raises but does not resolve.

• Why R = 512 helps ViT students: The ViT student's only notable hyperparameter preference is a larger embedding dimension, and the authors do not explain the mechanism. Understanding it could inform per-student defaults without per-pair tuning.

• Reconciling the ImageNet-1K results: At full scale the three leading cross-architecture methods land within 0.32 pp of each other on three of four pairs. Determining whether prototype-space transfer and learned alignment are genuinely converging at scale, or whether the benchmark saturates, is unresolved.

• Completing the frozen-versus-learned prototype ablation: The central design claim—that the prototype bank should be frozen rather than learned jointly with the student heads—is tested in a table whose results are cut off in the supplied content. Its outcome, and the conditions under which learning P might help, remain to be examined.

Target Audience

Graduate students and researchers working on knowledge distillation, model compression, or cross-architecture transfer will get the most from this paper, particularly those who have hit the dimensionality and geometry mismatches that make heterogeneous distillation awkward. Practitioners building compression pipelines for deployment—especially on small-category, domain-specific datasets—will find the low-category analysis and the minimal teacher-side setup directly actionable. Readers new to distillation will need some background in soft targets and feature-based transfer to follow the method section, but the central idea and the low-category results are legible without it.

Authors’ abstract

A widely held assumption in knowledge distillation is that teacher and student representations must occupy a compatible or explicitly aligned feature space. We propose RPKD (Random Prototype Knowledge Distillation), which discards this assumption by comparing both networks against a shared bank of randomly initialized, frozen prototype vectors. Alignment is therefore carried out in an external reference space rather than in the teacher's native feature coordinates: the teacher side is projected by fixed random maps that carry no trainable parameters, and only the student adapts. Two branches handle the transfer: a logit branch aligning global prototype similarity distributions and a feature branch matching spatially-resolved prototype activation maps. Both operate within the same fixed prototype space, which can be interpreted as a random feature embedding that approximately preserves inter-sample similarity structure, so the prototype bank and the distillation objective stay unchanged across teacher-student pairs while only the input projections are instantiated per architecture. In low-category settings, decoupling the prototype vocabulary from the task label space yields richer supervision than the label space alone, an advantage absent from logit-based methods, whose supervisory signal collapses as class count shrinks. This advantage is especially relevant in real-world scenarios where the label space is typically constrained. Extensive experiments demonstrate the effectiveness of RPKD, achieving a maximum gain of +7.98% over OFA on CIFAR-100, +12.44% on ImageNet-100, and +1.28% on ImageNet-1K. Source code is available at https://github.com/HamzaEtcibasi/RPKD