Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models

AI summary

Overview

Research area: Medical AI, specifically vision-language models (VLMs) for dermatological diagnosis, combining textbook-derived reasoning supervision with reinforcement learning.

Technical level: Advanced. The paper assumes familiarity with vision-language model architectures, supervised fine-tuning (SFT), Group Relative Policy Optimization (GRPO), reward shaping, and dermatology taxonomy concepts such as differential diagnosis (DDx).

Scope: The paper introduces Skin-R1, a dermatology-oriented VLM built on Qwen2.5-VL-7B-Instruct that is trained in three stages (textbook-grounded reasoning-trajectory construction, supervised fine-tuning, and hierarchical-reward reinforcement learning) and evaluates it against general, medical, and dermatology-specific VLMs on in-distribution and out-of-distribution dermatology benchmarks.

What This Paper Is About

Dermatological diagnosis is a visual reasoning task in which clinicians compare subtle skin-image patterns against hierarchical disease categories and weigh visually similar conditions against each other through differential diagnosis. Existing medical VLMs struggle here because dermatology datasets use inconsistent labels and concept annotations, rarely provide supervision for grounded diagnostic reasoning, and do not transfer well from small, densely annotated collections to large, sparsely labeled ones. Skin-R1 addresses this by first learning diagnostic reasoning from textbook-derived trajectories and then generalizing that reasoning to large, sparsely annotated datasets through reinforcement learning.

Key Contributions

1. Skin-R1 training paradigm. A dermatology-oriented VLM training framework that integrates textbook-grounded clinical reasoning with reinforcement learning to improve diagnostic accuracy and robustness across heterogeneous datasets.

2. Textbook-grounded trajectory generation. A framework that synthesizes hierarchy-aware and differential-diagnosis (DDx)-informed diagnostic reasoning traces, producing the SkinRationale dataset of 2,020 diagnostic trajectories for scalable, clinically consistent reasoning supervision.

3. Hierarchical reward design. A reward function that incorporates disease taxonomy structure into RL, allowing grounded reasoning patterns learned from densely annotated data to generalize to large-scale sparsely annotated datasets.

4. Empirical validation. Experiments across multiple dermatology benchmarks showing Skin-R1 outperforms state-of-the-art baselines in diagnostic accuracy and robustness, with ablations isolating the contribution of grounded reasoning supervision during SFT.

Main Findings

• Best in-distribution and out-of-distribution disease diagnosis. Skin-R1 achieves the highest average accuracy on in-distribution datasets (0.6385, which is +0.1955 over the next-best model) and on out-of-distribution datasets (0.7171, +0.0284).

• Balanced lesion condition classification. On classifying cases as benign, malignant, or precancerous in situ, Skin-R1 reaches the highest average accuracy (0.6928, +0.0297) and the highest average Macro-F1 (0.4287, +0.0633).

• Baseline biases are severe. MedGemma-4B shows a strong bias toward the rare class "C: precancerous in situ" (7,860 of 8,390 predictions), while LLaVA-Med-7B is biased toward the majority class "A: benign" (5,563 of 8,390 predictions), yielding low F1 scores. SkinVL-PubMM shows a mild bias toward "A: benign" and rarely produces meaningful diagnostic responses on in-distribution and out-of-distribution tasks, so its Table 2 metrics are reported as "−".

• Trajectory SFT improves both diagnosis and format compliance. The SFT model outperforms the base Qwen2.5-VL-7B-Instruct model. Under strict answer extraction, the base model fails to produce valid answers in approximately 30% of in-distribution diagnosis cases, versus only 2% for the SFT model.

• RL built on SFT is essential. The full Skin-R1 pipeline consistently outperforms the "RL without SFT" variant. RL without SFT improves on the base model, but its best performance remains significantly lower (Figure 2), indicating that RL-only approaches lack the grounded knowledge needed for robust reasoning. The paper notes that "RL without SFT" can be viewed as adapting Med-R1 and MedVLM-R1 to dermatological diagnosis.

• Hierarchy-aware reward improves generalization. Replacing the granularity-aware reward with a standard binary reward leads to weaker out-of-distribution performance. In the ablation table, RL with standard reward reaches an OOD average of 0.6386 versus 0.7171 for Skin-R1 at 1,500 steps.

• Checkpoint selection. The checkpoint at 1,500 RL steps is selected as the representative model; performance at 500 and 1,000 steps is lower on the in-distribution and out-of-distribution averages (0.5875 and 0.6156 in-distribution; 0.6742 and 0.6870 out-of-distribution).

• Stronger DDx and hierarchical reasoning. On two targeted evaluations that replace distractors with taxonomy ancestors (hierarchical diagnosis) or confusable neighbors in the DDx graph (DDx diagnosis), Skin-R1 achieves the highest accuracy in both settings. These evaluations use held-out images never seen during training, and the taxonomy and DDx graph are used only to generate distractors, not provided to the model at inference.

• More focused qualitative outputs. The base model tends to produce longer descriptive outputs while sacrificing diagnostic precision; Skin-R1 produces more concise responses consistent with expert reasoning, for example correctly concluding that observed features "indicate a benign pigmented nevus rather than melanoma."

Methodology in Plain English

The pipeline has three stages.

Stage 1: Build a reasoning dataset from a textbook. From an authoritative dermatology textbook, the authors extract three things: 220 diagnostic exemplars (each an image, a textual rationale, and a diagnostic label), a differential diagnosis graph with 211 nodes and 245 edges covering 61.36% of the collected exemplars, and a disease taxonomy with 458 nodes and 473 edges. Extraction follows a five-stage automated pipeline: image and text extraction, image clustering and filtering, text pairing and filtering, diagnostic rule extraction, and DDx and taxonomy confirmation and refinement. Human involvement is limited to selecting representative image clusters.

Stage 2: Turn textbook knowledge into training trajectories. Rather than using large reasoning models such as QwQ-32B to generate reasoning traces (which can contain errors and propagate hallucination), the authors build trajectories directly from textbook-derived rationales. Two mechanisms handle structural problems: hierarchical diagnosis completion, which augments each label with its ancestral and sibling nodes in the diagnosis tree to address label granularity inconsistency, and differential diagnosis reasoning enrichment, which uses the DDx graph to describe how visually similar conditions are distinguished. This yields K = 2,020 trajectories, each comprising a reasoning trace, a diagnosis label, and hierarchical diagnostic metadata.

Stage 3: Train, then reinforce. The model is formulated as a policy that maps a clinical image and instruction prompt to a structured output of a chain-of-thought rationale, a predicted disease label, and a predicted lesion category (benign, malignant, or precancerous in situ). First, supervised fine-tuning on the trajectories imitates expert diagnostic reasoning. Then GRPO is applied: the model generates K candidate responses, each scored by a reward and normalized within its group as A_j = (r_j − μ)/σ, and optimized with a clipped GRPO objective plus a KL penalty against a reference policy. The total reward is the sum of three terms: a format compliance reward (1 if required output tags are present, else 0), a granularity-wise reward (0.75 times the normalized taxonomy depth w_i = i/L if the predicted label lies on the ground-truth diagnostic path, else 0), and a malignancy discrimination reward (0.25 if the predicted malignancy category matches the ground truth, else 0).

Implementation. The backbone is Qwen2.5-VL-7B-Instruct, with both SFT and RL trained using LoRA at lora_r=64, lora_alpha=32, and lora_dropout=0.1. Training and evaluation use six dermatology datasets: PAD-UFES-20 (2,298 images across six lesion types, with 22 clinical and demographic attributes), DermNet (approximately 19,500 images spanning 23 coarse and more than 600 fine-grained categories, filtered by removing roughly 80% of samples to drop rare long-tail diseases), BCN20000 (18,946 images from 8 diagnostic classes), DERM12345 (12,345 dermoscopic and clinical images with multi-granular categories), Derm7pt, and HAM10000. Cross-dataset generalization is measured on the dermoscopy subset of OmniMedVQA. All tasks are posed as multiple-choice visual question answering, scored mainly by accuracy with Macro-F1 added for lesion condition classification, and an answer extraction protocol is applied to baselines that lack the required response format.

Why This Matters

Impact on research. The paper argues that RL-only medical VLM training produces heuristic or hallucinated rationales because it lacks clinically grounded supervision, and shows empirically that trajectory-based SFT raises the performance ceiling for later RL. It also demonstrates that structuring reward around a disease taxonomy improves out-of-distribution generalization, offering a template for other hierarchical medical domains.

Real-world applications:

• Triage and screening support for skin conditions such as melanoma, eczema, and psoriasis, where the paper notes millions of people experience delayed or incorrect diagnosis due to limited access to dermatologists.
• Differential diagnosis assistance, since the model is trained to justify excluding visually similar alternatives rather than only naming a label.
• Malignancy triage through lesion condition classification into benign, malignant, or precancerous in situ categories.
• Cross-dataset deployment, since Skin-R1 is designed to transfer reasoning learned from densely annotated data to large, sparsely annotated collections such as DermNet.

Industry relevance. Healthcare AI developers get a reproducible recipe (code and model weights are available at https://github.com/l593191569/Skin-R1) that combines a small set of expert-derived trajectories with RL, potentially reducing the annotation burden that limits most medical VLMs. The paper also documents failure modes of competing models—class bias and refusal to produce valid answers—that matter for anyone deploying or regulating clinical AI.

Future Directions

• Rationale quality and safety behavior. Scoring currently relies on multiple-choice answer-level accuracy, while the correctness and faithfulness of generated reasoning traces are examined only qualitatively. The authors leave quantification of rationale quality, confidence calibration, abstention or referral under high uncertainty, and severity-aware error analysis for malignant conditions to future work.

• Moving beyond a fixed knowledge structure. The taxonomy and DDx graph are extracted offline and held fixed during training; because these structures also generate the distractors for the in-distribution and targeted evaluations, those results partially reflect consistency with this fixed clinical knowledge, though the out-of-distribution OmniMedVQA evaluation is not subject to this coupling. Adapting to evolving or dataset-specific ontologies is an open problem.

• Testing other backbones. Skin-R1 is instantiated only on Qwen2.5-VL-7B; whether the SFT+RL recipe transfers to other backbones, including medical ones, and how it compares with additional dermatology-specific foundation models, remains unstudied.

• Fairness and process-level supervision. The authors observe modest degradation in certain Fitzpatrick skin types, likely reflecting training-data bias, and note that their outcome-level objective does not explicitly reward intermediate reasoning steps. Fairness-aware training, domain adaptation, and process-level supervision are suggested directions, though defining reliable reward signals for clinical reasoning remains challenging.

Target Audience

This paper is most useful to medical AI and multimodal machine learning researchers working on vision-language models and reinforcement learning from verifiable rewards; dermatology informatics groups seeking AI-assisted triage and differential diagnosis tools; and practitioners or regulators evaluating the trustworthiness, bias, and generalization limits of clinical VLMs. Readers without background in RL optimization or vision-language architectures will find the training details demanding, though the clinical motivation and results are accessible.

Authors’ abstract

Vision--language models (VLMs) have recently shown promise for assisting clinical reasoning in dermatological diagnosis. However, their trustworthiness and clinical utility remain limited by three key challenges: heterogeneous datasets with inconsistent diagnostic labels and concept annotations, the lack of grounded diagnostic rationales for reliable reasoning supervision, and limited scalability when transferring knowledge from small, densely annotated datasets to large collections with sparse labels. To address these challenges, we propose Skin-R1, a dermatology-oriented VLM that integrates textbook-grounded clinical reasoning supervision with reinforcement learning (RL) to improve the accuracy and robustness of diagnostic prediction. First, we construct a textbook-based reasoning generator that synthesizes hierarchy-aware and differential-diagnosis (DDx) diagnostic trajectories derived from authoritative dermatology knowledge. Second, these trajectories are used for supervised fine-tuning (SFT), establishing a clinically grounded reasoning foundation for the model. Finally, we introduce an RL training framework that incorporates the hierarchical structure of dermatological diseases into the reward design, enabling the model to generalize grounded diagnostic reasoning to large-scale datasets with sparse annotations. Extensive experiments across multiple dermatology benchmarks demonstrate that Skin-R1 consistently improves diagnostic accuracy and robustness compared to state-of-the-art Med-VLM baselines. Ablation studies further highlight the critical role of grounded reasoning supervision introduced during the SFT stage.