Explainable Melanoma Diagnosis with Contrastive Learning and LLM-based Report Generation

AI summary

Overview

Research area: Computer vision for medical imaging — specifically explainable melanoma classification from dermoscopic images, combining contrastive learning, clinical rule-based feature extraction, segmentation, and LLM-based report generation.

Technical level: Advanced. The paper assumes familiarity with Vision Transformers, contrastive learning objectives (NT-Xent), promptable segmentation models, CLIP-style vision-language alignment, and large language model prompting.

Scope: The paper proposes CEFM (Cross-modal Explainable Framework for Melanoma), which aligns ViT image embeddings with quantitatively computed Asymmetry, Border, and Color (ABC) clinical features via contrastive learning, and translates the result into structured diagnostic reports using CLIP-retrieved descriptors and a domain-adapted DeepSeek model.

What This Paper Is About

Deep learning models can classify melanoma at or above clinician-level accuracy, but they behave as "black boxes," which makes dermatologists reluctant to trust them. Existing explanation tools such as Grad-CAM highlight image regions without connecting them to the established ABCD diagnostic rule clinicians actually use. CEFM's goal is to make melanoma classification explainable by grounding the model's internal image representations in explicit, computable ABC clinical criteria and then generating a written diagnostic report from them.

Key Contributions

1. Clinically-grounded feature extraction pipeline: A coarse-to-fine segmentation module combining UltraLight VM-UNet and SAM2 to extract three dermoscopic features — Asymmetry, Border irregularity, and Color variation — derived from the ABCD rule, with explicit quantitative formulas for each (Table 1).

2. Cross-modal contrastive alignment: A contrastive learning architecture with dual projection heads that maps visual representations from a ViT and clinical ABC feature vectors into a shared latent space, so the classifier learns embeddings reflecting dermatological reasoning.

3. Hybrid diagnostic report generation: An explanation module integrating CLIP-based medical concept activation (CLIP-ViT-B/16) with a domain-adapted language model (DeepSeek) to produce structured, clinically accurate diagnostic reports.

4. Validation with domain experts and ablation: A structured user study with three board-certified dermatologists and an ablation study isolating the contribution of the CLIP module, the clinical explanation module, and the DeepSeek module.

Main Findings

• Classification accuracy: After contrastive feature alignment, the ViT-based model achieved 92.79% ± 0.57% accuracy and an AUC of 0.961 ± 0.004, averaged over 10 repeated experiments.

• Backbone comparison on ISIC2020: ViT and EfficientNet-B2 achieved the highest accuracy (94.26% each); ViT additionally showed the best precision (88.19%), AUC (0.972), and specificity (97.15%), and was selected as the baseline. Backbones evaluated included ResNet50, DenseNet121, Inception-V3, EfficientNet-B2, VGG16, and ViT.

• Segmentation quality: UltraLight VM-UNet reached a Dice Similarity Coefficient (DSC) of 0.8909, accuracy of 95.56%, and specificity of 0.9746 on ISIC2018, outperforming most baselines in overall performance.

• Cross-modal alignment works: After contrastive learning, positive image-clinical pairs clustered in the high-similarity region (greater than 0.75, concentrated near 0.8), while negative pairs were distributed near zero.

• Expert-rated interpretability: Averaged across three board-certified dermatologists, interpretability of AI decision-making scored 4.60/5, report clarity and readability 4.50, usefulness of ABC feature analysis 4.40, consistency with clinical judgment 4.18, and clinical applicability and decision support 4.00.

• LLM selection: Reports from DeepSeek were compared against Anthropic Claude and Google Gemini Base using BERTScore (RoBERTa-Large, layer 9), reporting Precision, Recall, and F1. The paper states semantic performance was comparable across models and DeepSeek was selected for its superior cost-effectiveness. Specific BERTScore values are not reported in the provided content.

• Ablation confirms complementary modules: Removing CLIP strips contextual visual cues, leaving only ABC numbers; removing the clinical explanation module removes ABC quantification entirely, leaving only visual descriptions; removing DeepSeek makes the report fragmented due to loss of structured synthesis.

Methodology in Plain English

The framework runs four connected pipelines.

First, a classification pipeline feeds a dermoscopic image into a ViT, which encodes it into a latent vector. A two-layer MLP projection head with non-linear activations and L2 normalization maps that vector into a shared space.

Second, a clinical explanation pipeline isolates the lesion. A pretrained UltraLight VM-UNet (trained on ISIC 2018) produces a rough mask, which serves as a pseudo-label. Foreground and background points sampled from that coarse mask prompt SAM2 to generate candidate masks; the candidate with the highest IoU against the pseudo-label is kept. From the refined mask, the system computes three numbers: Asymmetry as the ratio of differential pixels between the image and its mirror to total lesion pixels; Border as the mean curvature along the lesion contour; and Color as the standard deviations of the hue, saturation, and value channels. The "D" (differential structure) criterion is excluded because fine-grained expert annotations were unavailable.

Third, a contrastive module aligns the two. Image features and clinical feature vectors each pass through their own projection head into a common space, are L2-normalized, and are trained with a bidirectional normalized temperature-scaled cross-entropy (NT-Xent) loss, averaging the image-to-clinical and clinical-to-image directions over the batch. The pretrained image encoder stays frozen and only the projection heads are optimized; afterward the projection head is frozen and a lightweight classification head is trained on the embeddings.

Fourth, a report module discretizes the ABC values into five severity levels and uses CLIP-ViT-B/16 to retrieve top-ranked melanoma-related descriptors from a curated attribute list (for example "asymmetric shape," "blue-gray areas"). The numbers and descriptors are combined into a prompt and passed to DeepSeek to produce a structured report covering quantitative feature analysis, visual feature description, consistency and complementarity evaluation, and risk assessment.

Training and evaluation used ISIC data: ISIC 2018 for the coarse segmentation model, and for classification and fine segmentation 3,130 high-quality melanoma and nevus images from ISIC 2020 supplemented with 448 unique melanoma images from ISIC 2018 — totaling 3,578 images (646 melanoma, 2,932 nevi) for training — with evaluation on the official ISIC 2020 validation split (n = 573) and test split (n = 714).

Why This Matters

Impact on research: The paper offers a route past post-hoc saliency explanations for medical imaging. Rather than drawing a heatmap over a black box, it forces the model's embedding space to align with clinically defined, numerically computable criteria, and then surfaces those criteria in natural language. This connects explainability research to the diagnostic lexicon clinicians already use, and provides a template for other specialties where rule-based clinical criteria exist.

Real-world applications:

• Early triage of suspicious lesions in dermatology clinics, where the generated report can flag high-risk features before specialist review.
• Decision support for junior clinicians or non-specialists who need explicit ABC reasoning rather than a bare probability score.
• Longitudinal lesion tracking, which experts in the study endorsed as a use case, comparing quantitative ABC values across visits.
• Structured documentation, since the system produces a formatted report rather than an unstructured prediction.

Industry relevance: Regulatory and clinical adoption of AI diagnostics increasingly hinges on transparency. A pipeline that emits auditable numbers (asymmetry 11.69, border curvature 2.71) alongside prose is more defensible in review than a saliency map. The use of a cost-effective LLM (DeepSeek) after benchmarking against Claude and Gemini also signals that clinical report generation can be run economically rather than requiring frontier-model pricing.

Future Directions

• Integrate the differential structure criterion: The "D" of the ABCD rule was omitted due to lack of expert-level annotated data. Adding it would make the assessment complete but requires acquiring region-specific specialist labeling.

• Handle degraded and rare inputs: The current system assumes high-quality dermoscopic images and may underperform on poor imaging conditions or rare lesion subtypes absent from training data.

• Multi-temporal lesion tracking: Extending the framework to compare lesions across time points, using the quantitative ABC features as longitudinal measurements.

• Prospective clinical validation: Moving from retrospective ISIC benchmarks and a three-dermatologist user study to prospective clinical studies to test robustness across diverse lesion types and imaging conditions.

Target Audience

This paper is most valuable to researchers working at the intersection of medical computer vision and explainable AI, particularly those interested in contrastive alignment between imaging features and structured clinical knowledge. It is also relevant to dermatology informatics teams and clinical AI developers evaluating report-generation systems built on LLMs. Clinicians and regulatory reviewers assessing the trustworthiness of AI diagnostic tools will find the expert user study and ablation analysis directly useful, though the technical sections on NT-Xent loss and projection-head design require a machine learning background.

Authors’ abstract

Deep learning has demonstrated expert-level performance in melanoma classification, positioning it as a powerful tool in clinical dermatology. However, model opacity and the lack of interpretability remain critical barriers to clinical adoption, as clinicians often struggle to trust the decision-making processes of black-box models. To address this gap, we present a Cross-modal Explainable Framework for Melanoma (CEFM) that leverages contrastive learning as the core mechanism for achieving interpretability. Specifically, CEFM maps clinical criteria for melanoma diagnosis-namely Asymmetry, Border, and Color (ABC)-into the Vision Transformer embedding space using dual projection heads, thereby aligning clinical semantics with visual features. The aligned representations are subsequently translated into structured textual explanations via natural language generation, creating a transparent link between raw image data and clinical interpretation. Experiments on public datasets demonstrate 92.79% accuracy and an AUC of 0.961, along with significant improvements across multiple interpretability metrics. Qualitative analyses further show that the spatial arrangement of the learned embeddings aligns with clinicians' application of the ABC rule, effectively bridging the gap between high-performance classification and clinical trust.