Multi-Dataset Diagnostic Utility of Clinical Visual Concepts in AI Systems for Dermatology

AI summary

Overview

• Research area: Explainable AI (XAI) and concept-based computer vision for digital dermatology, with a focus on feature selection over clinically interpretable visual concepts.
• Technical level: Intermediate. The paper assumes familiarity with classification metrics (ROC-AUC, average precision), concept bottleneck models, and bootstrapping, but its central question is conceptual rather than architectural.
• Scope in one sentence: The authors harmonize four public dermatology datasets into a single 20,411-record resource with 48 clinical morphological concepts, then measure how useful those concepts actually are for coarse nine-way skin disease classification.

What This Paper Is About

AI decision-support tools in dermatology need to be trustworthy, and clinically interpretable visual concepts (such as erythema, scale, plaque) are a promising intermediate layer because doctors can audit them. Progress has been blocked because concept annotations are scattered across datasets with different vocabularies, formats, and patient populations. This paper builds a harmonized dataset, SkinLex, and uses it to ask how much of the 48-concept SkinCon vocabulary is genuinely needed for coarse diagnostic classification.

Key Contributions

1. SkinLex, a harmonized multi-source concept dataset. The authors unify SkinCon, DermaCon-IN, MM-Skin, and PASSION into 20,411 records carrying 48 clinical morphological attributes in the SkinCon concept space, describing it as five times larger than SkinCon. Concept information comes either from existing structured clinician annotations (SkinCon, DermaCon-IN) or from expert-authored text processed by an LLM (MM-Skin, PASSION).
2. A consolidated diagnostic label space. More than 6,000 noisy, fine-grained condition identifiers across the sources are mapped into the nine mutually-exclusive categories established with Fitzpatrick17k, using human captions and an LLM with the Fitzpatrick17k granular-to-nine-partition dictionary as in-context reference.
3. A utility analysis of the 48 concepts for nine-partition diagnosis. The authors establish lower and upper performance bounds, compare concept-based classification against frozen deep visual features, evaluate clinically defined concept groups in isolation, and run leave-one-out sensitivity over individual concepts.
4. A redundancy estimate via bootstrapped backward elimination. Using 100 bootstrapped training/validation sets and a one-sided paired Wilcoxon signed-rank test at a 5% threshold, they estimate how many concepts can be dropped without significantly degrading performance. Code and prompt templates are released publicly.

Main Findings

• Grouped concepts cannot diagnose alone. Restricting LightGBM to a single clinically defined group drops performance well below the full model (median ROC-AUC / AP with 95% CI from 100 validation bootstraps): full 48-concept model 0.829 [0.812, 0.845] / 0.309 [0.288, 0.330]; Shapes (16 features) 0.711 [0.695, 0.727] / 0.174 [0.163, 0.183]; Colors (10 features) 0.690 [0.671, 0.713] / 0.173 [0.164, 0.185]; Textures (15 features) 0.661 [0.644, 0.675] / 0.153 [0.145, 0.163]; Other (7 features) 0.530 [0.524, 0.538] / 0.124 [0.119, 0.129]. The Other group essentially approaches random guessing.
• Shapes and colors carry the strongest standalone signal. Among the isolated groups, Shapes and Colors retain the most predictive capacity, while the seven-feature Other group is weakest.
• The full concept set contains redundancy. Backward elimination shows that removing 8 concepts does not significantly lower AP relative to the full set, while up to 18 concepts can be omitted without decreasing ROC-AUC. The outcome depends on which metric is used.
• No individual concept is indispensable in the leave-one-out analysis. Omitting a single concept produces only small performance shifts. The color Brown (Hyperpigmentation) causes the largest degradation, followed by structural attributes Plaque, Papule, Nodule, and Scale.
• Concept models underperform frozen pretrained visual features. The primary model LightGBM reaches 0.8555 ROC-AUC and 0.3236 AP on validation, capturing 90.9% of the ROC-AUC ceiling and 58.1% of the AP ceiling. Logistic regression (one-versus-rest) reaches 0.8384 ROC-AUC and 0.2947 AP. For reference, a LightGBM classifier on frozen DINOv3 features obtains 0.8907 ROC-AUC and 0.3756 AP, outperforming supervised ImageNet features extracted by ViT-base models.
• Performance bounds are reported explicitly. The metric baselines are 0.5000 for ROC-AUC and 0.1111 for AP; an overfit-on-validation ceiling of 0.9404 ROC-AUC and 0.5568 AP bounds achievable performance.
• Aggregation fills coverage gaps between sources. Sclerosis, Blue, and Warty/Papillomatous morphologies are entirely absent (0.0%) from DermaCon-IN but are heavily supplied by MM-Skin, which accounts for 87.1%, 96.7%, and 79.9% of those concept instances respectively. PASSION provides 92.4% of all Excoriation instances.
• The dataset is severely imbalanced. Inflammatory conditions account for 15,091 samples (73.94%), while Malignant Dermal accounts for 128 samples (0.63%). Other categories range from Malignant Epidermal at 964 (4.72%) down to Malignant Melanoma at 465 (2.28%).
• One source was deliberately excluded. Derm1M was not used because 378,678 of its 423,016 captions are repeated identically at least twice, with the most frequent caption appearing in 2,945 copies, which would create identical concept-set/diagnosis entries and unrealistic reweighting with no variance.

Methodology in Plain English

The authors start from four public skin image datasets that each carry some human-expert signal about what a lesion looks like. Two of them already label images with binary concept tags in the SkinCon vocabulary, so those transfer directly. The other two use free text: MM-Skin pairs images with long clinical captions, and PASSION has two expert captions per sample. For those, the authors use an LLM (QuantTrio/Qwen3.6-35B-A3B-AWQ at zero temperature) to pull out both the 48 SkinCon concepts and the diagnosis.

Diagnostic labels are the harder problem because the sources use over 6,000 different granular condition names. The authors use the same LLM, with the Fitzpatrick17k mapping dictionary supplied as in-context reference to keep the output medically grounded, to collapse everything into nine standard categories. After filtering (quality lists, removing unspecified or unusable entries, and discarding MM-Skin cases where the model extracted multiple or no diagnoses), the result is SkinLex: 20,411 records, split by stratified random sampling into 64% training, 16% validation, and 20% test.

The experiments deliberately treat concepts as the model's input, not as something inferred from pixels, so that concept usefulness is measured separately from the ability to detect concepts in images. The authors screen twelve classic classifiers with PyCaret, settle on LightGBM with logistic regression as a linear comparison, and then run three analyses: restricting inputs to one concept group at a time, removing one concept at a time and retraining 48 models, and a bootstrapped backward elimination over 100 resampled training/validation pools that repeatedly drops whichever concept can be removed with the best median score.

Why This Matters

Impact on research. The work separates two questions that are usually entangled: whether a clinical concept is informative, and whether a model can detect it from images. By using concept labels as direct input, the authors isolate concept utility. The finding that a relatively small but varied combination of concepts suffices for coarse diagnosis argues that current concept taxonomies need refinement, and it gives concept-bottleneck researchers a way to reason about which concepts earn their place.

Real-world applications.

• Reducing the annotation burden for clinicians, since fewer concepts need to be labeled per image if redundancy is confirmed.
• Lowering input dimensionality and computational cost for concept-based diagnostic models in deployed settings.
• Making explanations more concise and stable by presenting a compact core of high-signal indicators rather than a long, noisy term list.
• Guiding future dataset design in digital dermatology, including where cross-dataset harmonization is needed because certain morphologies are missing entirely from some sources.

Industry relevance. Developers of dermatology decision-support software depend on clinician trust for adoption. Demonstrating which concepts are redundant and which carry the strongest standalone signal gives product teams a defensible basis for trimming annotation workflows, simplifying model inputs, and auditing model behavior. Public release of the harmonized code and prompt templates lowers the barrier for others to build on this pipeline.

Future Directions

• Refine the concept taxonomy. The authors state that their flat 48-concept set and nine-diagnosis partition is a simplified setup that omits disease hierarchies and context-dependent clinical vocabulary. Finer concept sets such as those in Derm1M, or more granular disease targets, may capture more diagnostic nuance while increasing task complexity.
• Verify the extraction pipeline. Automated concept and diagnosis extraction with the LLM requires further cross-model and human verification, particularly because original sources emphasize concept presence over absence, so caption-based extraction can conflate unmentioned features with negative findings.
• Test generalization and alternative concept users. Combining diverse sources likely introduces hidden subgroups and demographic constraints, and humans or other models may leverage concepts differently, producing different importance rankings. A concept's lack of utility for coarse algorithmic classification cannot be read as general clinical irrelevance, since redundancy is only defined relative to the surrounding concept set.
• Distinguish metric-dependent conclusions. Because the number of removable concepts differs between macro AP (8) and ROC-AUC (18), the authors note that the identity of removable concepts cannot be deduced from their result at all: a missing concept can be compensated for by others, and which one gets removed may be an artifact of the procedure.

Target Audience

Researchers and practitioners working on explainable AI and concept-based models for medical imaging, especially those interested in concept bottleneck architectures. Clinical informaticists and dermatologists involved in dataset curation or annotation standards will find the dataset harmonization and coverage analysis directly useful. Machine learning engineers building or auditing diagnostic decision-support systems in healthcare will benefit from the redundancy and sensitivity results, as will dataset builders thinking about how to design concept vocabularies that survive cross-source harmonization.

Authors’ abstract

The clinical integration of AI systems in digital dermatology relies heavily on human trust. Clinically interpretable visual concepts can act as intermediate representations enhancing trust and reliability. However, research in this domain is currently limited by scattered, heterogeneous dataset annotations. In this work, we introduce SkinLex, a harmonized dataset of 48 clinical morphological attributes across four public datasets (SkinCon, DermaCon-IN, MM-Skin, and PASSION) for a total of 20,411 records. Supervised nine-partition classification of skin conditions shows that limiting features to specific visual groups, like shapes or colors alone, reduces diagnostic accuracy. Bootstrapped backward elimination reveals that the set of 48 visual concepts has some degree of redundancy for algorithmic nine-partition diagnosis on the examined dataset. This demonstrates that coarse diagnosis on the selected dataset requires a relatively small but varied combination of clinical concepts, and motivates further research to improve concept taxonomy. Results can be translated into clinical benefits by reducing inputs for concept-based models, improving efficiency for annotation and modeling, and further enhancing interpretability. Code and prompt templates are available at https://github.com/Digital-Dermatology/SkinLex.