vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMs

AI summary

Overview

• Research area: Biomedical vision-language models; prompt learning / context optimization (CoOp) for CLIP-style VLMs; geometric representation learning on Riemannian manifolds.
• Technical level: Advanced. The paper assumes familiarity with CLIP contrastive pretraining, few-shot prompt tuning, and directional statistics (von Mises–Fisher distributions) on unit hyperspheres.
• Scope: The paper proposes vMFCoOp, a few-shot prompt-learning framework that models LLM-derived medical prompts and CLIP text embeddings as von Mises–Fisher distributions on a shared hyperspherical manifold, aligns them via Unified Semantic Anchors, and optimizes prompts under three geometric constraints.

What This Paper Is About

Adapting biomedical CLIP models to new clinical tasks usually means either writing prompts by hand or fully fine-tuning, both of which are costly or brittle when labeled data is scarce. Recent work (BiomedCoOp) instead has large language models generate medical prompts to guide CLIP's learnable context tokens, but it implicitly assumes LLM language priors and CLIP embedding spaces are directly compatible, and it aligns modalities with conventional pairwise matching in flat Euclidean space. vMFCoOp's goal is to reconcile the semantic biases between arbitrary LLMs and arbitrary biomedical CLIP backbones by placing both on one hyperspherical manifold and estimating their distributions, so that few-shot prompting stays stable, generalizable, and model-agnostic.

Key Contributions

1. The first framework to introduce inverse-estimation based on hyperspherical manifold probabilistic modeling, explicitly capturing and reconciling the semantic biases of diverse foundation models so the method can scale across evolving model families.
2. A reformulation of few-shot prompt learning for biomedical VLMs on a Unified Hyperspherical Manifold, where Unified Semantic Anchors bridge cross-modal gaps and improve generalization beyond Euclidean-space approaches.
3. Extensive validation on 14 realistic clinical few-shot scenarios, including challenging datasets from UK Biobank, showing consistent gains in biomedical prompt-based classification that support potential clinical deployment.
4. Public release of resources at https://github.com/VinyehShaw/UniEqui.

Main Findings

• Few-shot accuracy: On 14 biomedical datasets (K = 1, 2, 4, 8, 16, 32, 64 shots), vMFCoOp reports 57.25 ± 4.75, 58.88 ± 3.95, 68.29 ± 2.07, 72.07 ± 1.98, 75.45 ± 1.48, 77.08 ± 1.36, and 77.49 ± 1.05 respectively, and is best at every shot count.
• Margin over the strongest baseline: The paper reports a relative gain of 7.29% at 4-shot and 5.22% at 64-shot over the second-best method, BiomedCoOp (which scores 63.65 ± 3.27 at 4-shot and 73.65 ± 3.98 at 64-shot).
• Stability at higher shot counts: Competing methods plateau or reverse at higher shot counts (CoOp is cited as degrading due to overfitting), whereas vMFCoOp sustains both accuracy improvement and low variance across all K-shot regimes.
• Base-to-novel generalization: Averaged over 13 datasets (BUSI excluded for limited class diversity), vMFCoOp reaches HM 77.35% with base 78.02 and novel 76.70, versus BiomedCoOp's HM 72.58 (base 73.26, novel 71.91), CoCoOp's HM 66.73 (base 67.94, novel 65.57), and BiomedCLIP's HM 53.51 (base 45.64, novel 64.66). The paper states this surpasses BiomedCoOp by approximately 4.8%.
• Largest per-dataset gains: Improvements are most pronounced in clinically nuanced cases such as BTMRI, CTKIDNEY, and cardiac MRI — for example CardiacMRI base rises from 52.59 (BiomedCoOp) to 64.59 and novel from 47.36 to 70.85; LiverMRI base moves from 48.29 to 66.39.
• Zero-shot and adapter/linear-probe baselines are weaker: BiomedCLIP zero-shot is 46.32, with + Ensemble at 54.46 and + Selective Ensemble at 56.98; CLIP-Adapter ranges roughly from 43.52 ± 1.25 (K = 1) to 50.85 ± 1.40 (K = 64), and LP++ from 49.39 ± 7.28 to 71.89 ± 2.87.
• Interpretability: Using gScoreCAM saliency maps on five representative samples across modalities, vMFCoOp consistently highlights lesion-centric regions aligned with ground-truth contours. On a rare cardiac cine MRI case with a posterior mediastinal tumor, BiomedCoOp's attention stays on the cardiac region and misses the tumor, while vMFCoOp localizes the approximate lesion region.
• Constraint ablation (partially reported): With none of the three constraints enabled, base-to-novel Base is 68.43, Novel 42.11, HM 52.14, and few-shot accuracy is 43.22 (1-shot), 47.81 (4), 55.27 (8), 60.90 (16), 62.34 (32). Adding only the Semantic Anchor Loss raises this to Base 72.65, Novel 69.38, HM 70.98. The remaining rows of this ablation table are not included in the provided content.
• Geometric behavior: The temperature in the Spherical Contrastive Loss is annealed with a cosine schedule from τ0 to τ_max, evolving decision boundaries from initial equal-angle partitions to large-margin separations and refining within-class representations from broad angular spread to compact clusters.

Methodology in Plain English

CLIP's text and image encoders output vectors that are normalized to unit length, so they already sit on the surface of a sphere. The authors exploit this by treating all embeddings as directions on one shared hypersphere rather than as points in flat space.

1. Collect two kinds of priors. From the CLIP text encoder's vocabulary embedding matrix, they fit a single von Mises–Fisher (vMF) distribution, producing a CLIP Semantic Anchor Field characterized by a mean direction and a concentration parameter estimated by maximum likelihood. Separately, for each medical class, an LLM generates prompt templates (default: GPT-4 with 50 prompts per class; Qwen2.5-72B-Instruct, Claude 3.5–1022, and DeepSeek R1 are also used), which are encoded by the CLIP text encoder and fit with a class-conditional vMF distribution, yielding an LLM Semantic Prototype Field.
2. Fuse into Unified Semantic Anchors. The CLIP field and each class's LLM prototype field are scaled by their concentration parameters, summed, and re-normalized to the sphere. Each resulting vector is one anchor per class, blending CLIP's global directional prior with class-specific LLM semantics.
3. Optimize prompts under three constraints. The prompt context (initialized from "a photo of a") is learned so that (i) the Semantic Anchor Loss pulls each class prompt toward its anchor, with a learnable offset and scaling factor allowing the target direction to navigate dynamically; (ii) the Spherical Contrastive Loss applies a supervised angular margin, using a prototype affinity matrix of temperature-scaled cosine similarities between prompt embeddings and anchors and a row-wise softmax cross-entropy, so each prompt moves toward its own anchor and away from distractors; (iii) the Symmetric Cross-Entropy Loss jointly minimizes forward and reverse divergences between image-based and prompt-based class distributions.
4. Train cheaply. Learning rate 0.003, batch size 4, SGD with cosine learning-rate scheduling, all on a single NVIDIA A100 GPU (80GB). Default backbone is BiomedCLIP with a ViT-B/16 vision transformer; PubMedCLIP, MedCLIP, and PMC-CLIP are also evaluated for compatibility.

Why This Matters

• Research impact: The work argues that pairwise multimodal alignment in Euclidean space cannot model unified representations or apply localized geometric constraints, and shows that distributional, non-Euclidean modeling gives more stable few-shot adaptation. It also positions the method as model-agnostic, addressing the practical problem that foundation-model families keep changing.
• Real-world applications:
  • Few-shot diagnostic support where a hospital has only a handful of labeled cases for a rare disease class.
  • Cross-modality deployment across the 12 imaging modalities covered, from X-ray and CT to histopathology, dermoscopy, fundus photography, OCT, ultrasound, and endoscopy.
  • Population-scale cardiac, liver, and pancreas MRI analysis derived from UK Biobank with ICD-10 coded labels.
  • Explainable triage, where saliency maps localize the lesion region and can be overlaid with ground-truth contours for clinician review.
• Industry relevance: Prompt learning avoids full fine-tuning, which the paper describes as often impractical under computational and data constraints, and avoids the instability and hallucination risk the authors attribute to multimodal LLMs in structural understanding and high-level semantics. That makes the approach attractive for clinical deployment where lightweight, interpretable architectures are preferred.

Future Directions

• The authors state that the work aims to continuously expand to encompass more downstream applications, and that corresponding resources will be shared via the project GitHub page.
• Appendix-only material that remains open to exploration includes results for 32- and 64-shot base-to-novel settings and for 150 prompts per class, which the main text does not report.
• Extending compatibility to further LLM and CLIP families is framed as a design goal; the current evaluation covers three non-default LLMs (Qwen2.5-72B-Instruct, Claude 3.5–1022, DeepSeek R1) and three additional biomedical CLIP variants (PubMedCLIP, MedCLIP, PMC-CLIP) beyond the default BiomedCLIP.
• The ablation analysis of the three constraints is only partially reported in the provided content, leaving the individual contribution of the Spherical Contrastive and Symmetric Cross-Entropy losses as an open question in this summary.

Target Audience

Researchers and practitioners working on vision-language models, prompt/context optimization, and few-shot medical image classification. It will be most useful to readers comfortable with CLIP-style contrastive embeddings, Riemannian or hyperspherical geometry, and directional statistics, as well as clinical AI engineers looking for a lightweight alternative to full fine-tuning and to LLM-generated prompt pipelines.

Authors’ abstract

Recent advances in context optimization (CoOp) guided by large language model (LLM)-distilled medical semantic priors offer a scalable alternative to manual prompt engineering and full fine-tuning for adapting biomedical CLIP-based vision-language models (VLMs). However, prompt learning in this context is challenged by semantic misalignment between LLMs and CLIP variants due to divergent training corpora and model architectures; it further lacks scalability across continuously evolving families of foundation models. More critically, pairwise multimodal alignment via conventional Euclidean-space optimization lacks the capacity to model unified representations or apply localized geometric constraints, which tends to amplify modality gaps in complex biomedical imaging and destabilize few-shot adaptation. In this work, we propose vMFCoOp, a framework that inversely estimates von Mises-Fisher (vMF) distributions on a shared Hyperspherical Manifold, aligning semantic biases between arbitrary LLMs and CLIP backbones via Unified Semantic Anchors to achieve robust biomedical prompting and superior few-shot classification. Grounded in three complementary constraints, vMFCoOp demonstrates consistent improvements across 14 medical datasets, 12 medical imaging modalities, and 13 anatomical regions, outperforming state-of-the-art methods in accuracy, generalization, and clinical applicability. This work aims to continuously expand to encompass more downstream applications, and the corresponding resources are intended to be shared through https://github.com/VinyehShaw/UniEqui.