IMA++: ISIC Archive Multi-Annotator Dermoscopic Skin Lesion Segmentation Dataset

AI summary

Overview

Research area: Medical image analysis / computer vision — specifically multi-annotator segmentation of dermoscopic skin lesion images, dataset curation, and inter-annotator agreement analysis.

Technical level: Intermediate. The paper is a dataset paper with accompanying validation analysis; it assumes familiarity with segmentation metrics (Dice, Hausdorff distance), consensus algorithms (STAPLE, majority voting), and standard machine learning evaluation conventions, but the core ideas are accessible.

Scope: The paper introduces and characterizes IMA++, a large public dataset of multi-annotator skin lesion segmentation masks drawn from the ISIC Archive, with annotator, tool, and skill-level metadata.

What This Paper Is About

Multi-annotator medical image segmentation research needs datasets where several experts label the same image, because lesions are genuinely ambiguous at the boundary and different experts draw different borders. The problem is that such datasets are expensive to collect, and skin lesion segmentation in particular had no large public multi-annotator resource — the only prior one, ISIC 2019-Seg, contains just 100 images with 3 segmentations each. This paper collects, filters, processes, and releases IMA++: 17,684 segmentation masks over 14,967 dermoscopic images, with 2,394 images carrying 2–5 segmentations each, plus metadata about who segmented, with what tool, and at what skill level.

Key Contributions

1. A large public multi-annotator skin lesion segmentation dataset. IMA++ contains 17,684 segmentation masks spanning 14,967 dermoscopic images produced by 16 annotators, with 2,394 images having 2–5 segmentations each. The authors state it is the largest publicly available skin lesion segmentation dataset, multi-annotator or otherwise, and 11,081 of its 14,967 images (~74%) are unique from past ISIC challenge datasets.

2. Rich annotation and image metadata enabling factor-based analysis. Each segmentation carries an annotator ID (A00–A15), a tool ID (T1 manual polygon tracing, T2 semi-automated flood-fill, T3 fully-automated segmentation reviewed by a human expert), and a skill level (S1 expert, S2 novice), plus an object ID and MD5 hash. Image metadata includes patient and clinical fields such as age, sex, anatomical site, malignancy status, diagnosis labels, and melanoma thickness.

3. Consensus masks and standardized data partitions. For the 2,394 multi-annotator images, the authors compute consensus segmentations using majority voting and STAPLE, bringing the total released mask count to 22,472. They also define training/validation/test splits (1,675 / 240 / 479 images, a 70:10:20 ratio) stratified by both the number of segmentations per image and by inter-annotator agreement level.

4. A quantitative characterization of inter-annotator variability. The paper reports inter-annotator agreement metrics across the multi-annotator subset, agreement broken down by annotator, tool, and skill level, an UpSet analysis of annotator overlap, and a comparison of IMA++ against eight other public skin lesion segmentation datasets.

Main Findings

• Dataset composition: Of the 14,967 images, 12,573 have one segmentation mask, 2,130 have two, 209 have three, 51 have four, and 4 have five, giving (12,573 × 1) + (2,130 × 2) + (209 × 3) + (51 × 4) + (4 × 5) = 17,684 segmentations.

• Quality filtering removed faulty masks: The quality assurance pipeline flagged 59 empty masks, 3 full-image masks, 0 masks with disconnected regions, and 1,129 masks touching the image border. Only empty masks were removed as affecting utility; 59 images lacking any associated metadata were also removed, leaving 14,967 images.

• Annotation is skewed across annotators: Annotator-segmentation counts follow a long-tailed distribution. Six annotators (A00–A05) contribute approximately 78% of the segmentations (13,748 of 17,684); Figure 3's caption reports the top 6 annotators (~37% of annotators) contributing ~91% of the segmentations. Tools T1 and T3 account for ~81% of segmentations (14,247), and ~70% (12,454) were reviewed by an S1 (expert) reviewer.

• The annotator pool is predominantly clinical: 12 annotators held a medical degree (MD or MD-PhD), of whom 9 were board-certified dermatologists; the remaining non-clinical annotators held MS or PhD degrees. These statistics are reported in aggregate to preserve anonymity.

• An incomplete bipartite annotation structure: Unlike most multi-annotator medical segmentation datasets where every annotator labels every image, IMA++ has an incomplete bipartite graph. With 16 annotators, 2^16 − 1 unique non-empty intersections are theoretically possible; IMA++ contains 57 unique annotator intersections.

• Pairwise agreement on the multi-annotator subset (2,394 images): Dice coefficient 0.7813 ± 0.2183 (median 0.8572, IQR 0.1879); Jaccard index 0.6830 ± 0.2381 (median 0.7502, IQR 0.2765); Hausdorff distance 382.5328 ± 496.5885 px (median 159.5243, IQR 404.3275); HD95 306.2117 ± 431.7573 px (median 114.0000, IQR 313.3568); ASSD 135.8883 ± 219.2955 px (median 46.0849, IQR 118.3817). Normalized variants: nHD 0.0970 ± 0.1046, nHD95 0.0783 ± 0.0947, nASSD 0.0354 ± 0.0513.

• Consensus algorithms largely agree with each other: The mean Dice coefficient between the STAPLE and majority voting consensus masks is 0.8613 ± 0.1724. However, the authors note that neither consensus method exhibits a very high degree of agreement with the underlying annotations.

• Agreement is high on average but with a long low-agreement tail: Across the entire dataset, most images showed reasonably high agreement (Dice 0.866 ± 0.187, inter-quartile range 0.161). Yet 236 images had a mean Dice below 0.5, 43 below 0.1, and 23 images had segmentations with zero overlap between annotators.

• Tool-specific and annotator-specific patterns: Tools T1 and T3 show much higher agreement with each other than with themselves or with T2, whereas T2 shows the opposite pattern — high intra-tool agreement but low inter-tool agreement. Annotators A01 and A03 have lower intra-annotator agreement; A08 generally has low agreement with all other annotators and with the consensus masks; and both surprisingly low (e.g., {A00, A06}, {A03, A12}, {A04, A11}) and high (e.g., {A03, A05}, {A06, A07}) pairwise agreements emerge.

• Metadata completeness varies by field: 99.98% of images (14,964) have a recorded diagnosis at any level; 96.08% (14,380) have diagnosis confirmation type; 99.96% (14,961) have malignancy status; 95.41% (14,280) have sex information; 94.94% (14,209) have age; and 32.06% (4,799) have anatomical site information.

• Scale comparison with prior datasets: Prior public skin lesion segmentation datasets range from 200 images (PH2, 2013) to 10,015 images (HAM10000, 2020), and all except ISIC 2019-Seg (100 images, 300 segmentations) are single-annotator. ISIC 2016, 2017, and 2018 contain 1,279, 2,750, and 3,694 images respectively, with 706 images common across the three.

Methodology in Plain English

The authors did not ask annotators to label images from scratch. Instead, they gathered existing segmentation masks and their associated metadata from the ISIC Archive, combining previously downloaded segmentations with metadata obtained by contacting sources. This was necessary because the old ISIC API v1 (now deprecated) exposed segmentation endpoints, while the newer ISIC API v2, its command-line interface, and the web interface do not.

They then ran masks through an automated quality check that looks for four problems: masks with no lesion pixels, masks covering the entire image, masks made of disconnected pieces, and masks touching the image edge. Each result was recorded so users can apply their own filters. Only empty masks were removed outright, along with images that had no metadata at all.

Next, they assigned anonymized identifiers to the three factors that drive segmentation variability — annotator (A00–A15, ordered by how many masks that person produced), tool (T1, T2, T3), and skill level of the human reviewer (S1 expert, S2 novice) — and encoded all of this into the mask filenames as well as metadata CSV files. MD5 hashes were computed for integrity checking.

For images with two or more segmentations, they computed consensus masks with two standard algorithms, majority voting and STAPLE, using SimpleITK filters. They then built evaluation splits of the multi-annotator subset, stratifying by how many segmentations each image has and by how much annotators agreed on it, where agreement was measured as the average pairwise Dice per image and bucketed into low Dice in [0, 0.5), medium in [0.5, 0.8], and high in (0.8, 1.0].

Finally, they quantified agreement using Dice and HD95 (the 95th percentile Hausdorff distance), and because images differ in size, they also computed normalized boundary metrics (nHD, nHD95, nASSD) by dividing raw distances by the image diagonal length, producing unitless values in [0, 1]. Annotator overlap was visualized with UpSet plots, which handle many sets better than Venn diagrams or pairwise co-occurrence matrices.

Why This Matters

Impact on research. The paper addresses a concrete bottleneck: multi-annotator segmentation research requires multi-annotator data, and the skin lesion domain previously had essentially none at scale. Prior work in this space had been limited to ISIC 2019-Seg's 100 images. A dataset with 2,394 multi-annotator images and documented annotator, tool, and skill metadata opens up research directions that were previously impractical, including annotator-specific preference modeling, consensus modeling, learning distributions over segmentations, discovering underlying segmentation styles, and predicting inter-annotator agreement directly from images. Two recent papers already used subsets of IMA++ for the latter two tasks.

Real-world applications:

• Clinical decision support. Widely used diagnostic rules such as ABCD (Asymmetry, Border, Color, Differential structure) depend on accurate lesion boundary delineation. Understanding how segmentation variability propagates into these rules is directly relevant to whether automated or semi-automated dermatology tools produce reliable outputs.

• Melanoma screening and triage. Melanoma accounted for 330,000 new diagnoses and 60,000 deaths in 2022, and incidence is projected to double every 10–20 years, while the dermatologist-to-population ratio is projected to decline. Robust segmentation is a component of automated skin analysis pipelines intended to help absorb that load.

• Dermatology training and quality control. Because IMA++ records expert versus novice skill levels, it supports studying how annotation skill affects segmentation quality and how to design training or review processes that reduce inter-annotator disagreement.

• Benchmarking large foundation models. Segmentation is used as a benchmark task for evaluating large vision models; a dataset with this scale and metadata richness offers a harder and more realistic evaluation target than single-annotator datasets.

Industry relevance. Any organization building dermatology AI — from medical device companies to telemedicine platforms — faces the question of how much annotation variability affects model performance and how to model the fact that no single "ground truth" boundary exists. IMA++ provides a public testbed for training models that accommodate multiple valid answers rather than collapsing them into one mask.

Future Directions

1. Developing a truly groupwise inter-annotator agreement measure. The authors point out that all existing IAA measures for medical image segmentation are pairwise, so image-level agreement for N segmentations requires aggregating N-choose-2 pairwise values (by mean or median), which discards information about the full distribution. They suggest IMA++'s incomplete bipartite graph is an ideal testbed for validating such a new measure.

2. Explaining complete annotator disagreement. 23 images in IMA++ have entirely non-overlapping segmentations from multiple annotators. The authors explicitly flag understanding what leads annotators to disagree completely as a worthwhile direction.

3. Tracing segmentation variability into clinical decision rules. Given the reported association between lower IAA and malignant lesions, the paper raises the question of how segmentation variability affects ABCD feature computation and the resulting diagnosis derived from such rules.

4. Exploiting the incomplete bipartite structure. Because not every image is segmented by every annotator, methods need to handle this realistic missing-data setting; the paper frames this as more representative of real-world annotation than complete bipartite datasets, and as a challenge for future multi-annotator modeling work.

Target Audience

Researchers in medical image analysis and computer vision working on multi-annotator segmentation, segmentation uncertainty, or consensus modeling; dermatology AI and skin lesion analysis groups who need a large, metadata-rich segmentation resource; machine learning practitioners interested in annotator-specific preference modeling and inter-rater agreement; and clinical researchers studying diagnostic criteria such as ABCD that depend on lesion boundary delineation. The paper is also useful to anyone evaluating whether to adopt IMA++ as a benchmark, since it documents data quality caveats, metadata completeness levels, and predefined splits.

Authors’ abstract

Multi-annotator medical image segmentation is an important research problem, but requires annotated datasets that are expensive to collect. Dermoscopic skin lesion imaging allows human experts and AI systems to observe morphological structures otherwise not discernable from regular clinical photographs. However, currently there are no large-scale publicly available multi-annotator skin lesion segmentation (SLS) datasets with annotator-labels for dermoscopic skin lesion imaging. We introduce ISIC MultiAnnot++, a large public multi-annotator skin lesion segmentation dataset for images from the ISIC Archive. The final dataset contains 17,684 segmentation masks spanning 14,967 dermoscopic images, where 2,394 dermoscopic images have 2-5 segmentations per image, making it the largest publicly available SLS dataset. Further, metadata about the segmentation, including the annotators' skill level and segmentation tool, is included, enabling research on topics such as annotator-specific preference modeling for segmentation and annotator metadata analysis. We provide an analysis on the characteristics of this dataset, curated data partitions, and consensus segmentation masks.