MobileFetalCLIP: Selective Repulsive Knowledge Distillation for Mobile Fetal Ultrasound Analysis

AI summary

Overview

• Research area: Computer vision and multimodal (vision-language) model compression, applied to medical imaging, specifically fetal ultrasound.
• Technical level: Advanced. The method is built on contrastive image-text pretraining and knowledge distillation mathematics, though the core intuition is accessible.
• Scope: The paper proposes a knowledge-distillation framework (DARK) that lets a small mobile vision-language model selectively imitate and then deliberately diverge from a much larger teacher, demonstrated by compressing a fetal-ultrasound model at a 26x visual-encoder parameter gap.

Note on naming: the arXiv listing title is "MobileFetalCLIP: Selective Repulsive Knowledge Distillation for Mobile Fetal Ultrasound Analysis," while the paper content itself is titled "DARK: Diagonal-Anchored Repulsive Knowledge Distillation for Vision-Language Models under Extreme Compression." The student model is named MobileFetalCLIP; the method is named DARK.

What This Paper Is About

Domain-specialized vision-language models can perform zero-shot medical imaging tasks well, but models with hundreds of millions of parameters cannot run on the handheld ultrasound devices that would most benefit from them. Standard knowledge distillation, the usual tool for shrinking such models, degrades when the teacher is an order of magnitude larger than the student. This paper asks what a compact student should actually try to inherit under that extreme capacity gap, and proposes treating one part of the teacher's knowledge as something to imitate and another part as something to push away from.

Key Contributions

1. A contrastive distillation framework (DARK) for extreme compression. The symmetric cross-entropy distillation loss is decomposed into a diagonal term (matched image-caption alignment) and an off-diagonal term (non-target inter-class similarities). The diagonal term is anchored at a fixed weight throughout training, while the off-diagonal term is scaled by a coefficient that decays linearly from positive to negative, flipping the objective from imitation to repulsion.
2. An empirical instantiation in fetal ultrasound. FetalCLIP (427M parameters, ViT-L/14 visual encoder with 304M visual parameters) is distilled into MobileFetalCLIP (75M parameters, 11.4M-parameter FastViT visual encoder, 4-layer Transformer text encoder, 512-dimensional embeddings) — a 26x reduction in visual-encoder parameters. The model and training framework are released.
3. Analysis of why repulsion helps. The paper reports embedding-geometry and logit analyses supporting an effect the authors call "structured decorrelation": the student retains teacher-aligned per-image confidence while diverging from the teacher's inter-class similarity structure.
4. On-device efficiency results. The student visual encoder requires 32x fewer multiply-accumulate operations (MACs) and runs in 1.6 ms on an iPhone 16 Pro.

Main Findings

• The student matches or exceeds the teacher on two of three benchmarks. MobileFetalCLIP reaches HC18 biometry validity of 88.6% versus the teacher's 83.5%, and brain sub-plane macro-F1 of 0.784 versus 0.702. On five-plane classification it stays competitive at 0.946 versus the teacher's 0.973. Its composite score is 0.886 against the teacher's 0.853.
• Static distillation helps but plateaus below the teacher. Static logit KD (CLIP-KD objective, lambda = 1.0) lifts HC18 validity from 0.713 to 0.794 and five-plane F1 from 0.889 to 0.946, but remains below the teacher's 0.835 validity and barely moves brain sub-plane F1 (0.712 to 0.715).
• Weakening or removing distillation is ineffective. Decaying the KD weight from 1 to 0.1 gives HC18 validity of 0.746; decaying it fully from 1 to 0 gives 0.731 — both worse than static KD.
• Feature-level distillation is harmful at this gap. Adding feature KD degrades HC18 validity from 0.794 to 0.759 and brain sub-plane F1 from 0.715 to 0.664.
• Undirected confidence regularization is insufficient. A confidence penalty (epsilon = 0.1) yields HC18 validity of 0.749 and brain sub-plane F1 of 0.680, showing that a uniform entropy penalty cannot replace a teacher-derived, directional repulsive signal.
• Repelling the whole KD loss helps, but less than DARK. Coupled Repulsive KD (annealing the full loss from beta_start = 2 to beta_stop = -0.8) reaches HC18 validity of 0.844 and brain sub-plane F1 of 0.763. DARK's diagonal anchoring adds +4.2% HC18 validity and +2.1% brain sub-plane F1 on top of that.
• Both decomposition and repulsion strength matter. Stronger amplification (beta_start = 4 or 8) and weaker repulsion (beta_stop = -0.4 or -0.5) both degrade performance relative to beta_start = 2, beta_stop = -0.8.
• Inference is far cheaper than the teacher. The student encoder uses 1.2 GMACs versus 38.9 GMACs, and runs in 1.6 ms on an iPhone 16 Pro (versus the teacher's 37.6 ms) and 1.4 ms on an iPhone 17 Pro (versus 31.9 ms). This corresponds to over 600 frames per second.
• Cluster geometry improves substantially. On the Planes DB 5-plane set, DARK raises the silhouette score to 0.525 versus 0.375 for static KD (reported as +40%), collapses inter-class cosine similarity from 0.445 to 0.076, achieves the highest effective dimensionality (10.0 versus 8.0) and best uniformity (-2.308). Coupled Repulsive KD instead concentrates features into fewer dimensions (d_eff 6.4).
• Predictions become sharper but remain consistent with the teacher. Logit analysis reports entropy dropping from 0.666 to 0.044 with high rank correlation (rho = 0.822), which the authors interpret as teacher-aligned per-image confidence.
• Repulsion fails when mistimed or too aggressive. An exploratory coupled run with beta_stop = -1.6 collapsed after roughly epoch 14, and overly strong off-diagonal amplification (beta_start = 8) degraded HC18 validity to 0.786. Successful schedules required an initial attractive phase.
• Teacher-specific confusion structure is the culprit. The teacher's ViT-L/14 distributes capacity across all inter-class relationships via global self-attention, including confusable brain sub-planes; the paper argues a compact FastViT cannot represent this faithfully and that DARK instead uses those confusion patterns as a signal for where to build sharper boundaries.

Methodology in Plain English

The setup is standard zero-shot CLIP-style training with a frozen teacher. A batch of N image-text pairs passes through both the teacher and the student, and each produces an N x N matrix of image-text similarity scores. The distillation loss makes the student's row-wise similarity distribution match the teacher's.

The paper's key move is to split that loss by position in the matrix. Entries on the diagonal are matched pairs — image i with its own caption i. Entries off the diagonal are non-target similarities — image i with some other caption j. The diagonal term captures the basic alignment every vision-language model must get right. The off-diagonal term captures the teacher's particular pattern of which classes look alike to it, which the authors argue reflects the teacher's architecture as much as the data.

DARK weights these terms differently. The diagonal term keeps a fixed weight of 1 for the entire run. The off-diagonal term is multiplied by a coefficient beta(t) that decays linearly over training from beta_start = 2 to beta_stop = -0.8. Early on, beta(t) is positive, so the student imitates the teacher's relational structure. When beta(t) crosses zero, the off-diagonal gradient inverts: instead of minimizing divergence from the teacher's non-target distributions, the objective maximizes it, actively pushing the student away. The paper calls this the repulsive regime and describes training in three phases: attractive, transition, repulsive. With these settings, roughly 70% of training is attractive and 30% repulsive.

Training details: 20 epochs, effective batch size 1,024, KD temperature 5.0. The corpus is 246,349 fetal ultrasound image-caption pairs curated from a tertiary hospital plus expert-annotated textbook pairs. Student and teacher share affine and color-jitter augmentation parameters within each forward pass so their similarity matrices correspond to the same augmented input. Evaluation is zero-shot on Planes DB (12,400 images from 1,792 patients across two hospitals; 8,187 images used for 5-plane classification, 2,949 brain images for 3-class sub-plane classification) and HC18 (999 head-circumference images, 814 retained with plausible head circumference of 100–342 mm and 14–40 weeks gestational age; a prediction is valid if the true HC falls within the 2.5th–97.5th percentile of WHO growth charts for the predicted gestational age).

Why This Matters

The paper reframes distillation under extreme compression: rather than asking what a compact student can absorb, it asks which parts of the teacher's knowledge are worth transferring at all. If the reported effect generalizes, it suggests a cheap and broadly applicable modification to existing contrastive distillation pipelines — one that needs no architectural compatibility between teacher and student, unlike weight-inheritance approaches such as TinyCLIP.

Real-world applications suggested by the work:

• Point-of-care ultrasound in low-resource settings. The authors explicitly target real-time assistive feedback on handheld POCUS devices, where a 1.6 ms encoder leaves headroom relative to the typical 30–60 fps diagnostic ultrasound frame rates.
• Standard-plane identification during scanning. A model that can flag whether a fetal head, abdomen, femur, thorax, or cervix plane is being acquired could support less-experienced operators.
• Fetal biometry and gestational-age estimation. The HC18 validity task measures whether predicted gestational age is consistent with measured head circumference.
• Brain sub-plane classification. Distinguishing transthalamic, transcerebellum, and transventricular planes was the task with the largest reported gain over the teacher (+8.2% F1).

Industry relevance is centered on mobile medical AI: on-device inference avoids network dependency and keeps patient data local, and the reported latency on two iPhone generations (16 Pro and 17 Pro) suggests generational consistency in deployment performance. The authors state that the framework is architecture- and domain-agnostic, positioning it as a general compression recipe rather than a fetal-ultrasound-specific trick.

Future Directions

• Prospective clinical validation. The paper states that the reported benchmarks are retrospective, and that clinical translation requires prospective validation for robustness against diverse ultrasound hardware and operator variability.
• Real-time evaluation on POCUS devices. Given the 1.6 ms latency, the authors' immediate focus is live assistive feedback in low-resource settings rather than retrospective benchmark scores.
• Extension to other clinical domains. Because DARK is described as architecture- and domain-agnostic, the authors plan to apply it to echocardiography and cross-modal retrieval in general radiology.
• Understanding the boundaries of repulsion. The paper documents failure modes (beta_stop = -1.6 collapsed after roughly epoch 14; beta_start = 8 degraded HC18 validity to 0.786) but the underlying stability conditions and how to schedule them automatically remain open.

Target Audience

Researchers working on model compression and knowledge distillation, particularly those interested in contrastive or vision-language architectures rather than classifiers. Also relevant to applied medical-imaging and mobile-health engineers who need to deploy large models on-device, and to readers interested in the "students surpassing teachers" question in born-again-network-style settings. Some familiarity with CLIP-style contrastive objectives and softmax-based distillation losses is assumed; the supplementary material contains implementation, spectral-analysis, linear-probing, and seed-robustness details that the truncated main text references but does not fully reproduce.

Authors’ abstract

Fetal ultrasound AI could transform prenatal care in low-resource settings, yet current foundation models exceed 300M visual parameters, precluding deployment on point-of-care devices. Standard knowledge distillation fails under such extreme capacity gaps (~26x), as compact students waste capacity mimicking architectural artifacts of oversized teachers. We introduce Selective Repulsive Knowledge Distillation, which decomposes contrastive KD into diagonal and off-diagonal components: matched pair alignment is preserved while the off-diagonal weight decays into negative values, repelling the student from the teacher's inter-class confusions and forcing discovery of architecturally native features. Our 11.4M parameter student surpasses the 304M-parameter FetalCLIP teacher on zero-shot HC18 biometry validity (88.6% vs. 83.5%) and brain sub-plane F1 (0.784 vs. 0.702), while running at 1.6 ms on iPhone 16 Pro, enabling real-time assistive AI on handheld ultrasound devices. Our code, models, and app are publicly available at https://github.com/numanai/MobileFetalCLIP.