A dataset of medication images with instance segmentation masks for preventing adverse drug events

AI summary

Overview

Research area: Computer vision for healthcare — specifically instance segmentation and object detection datasets for automated pill recognition, aimed at reducing medication errors and adverse drug events (ADEs).

Technical level: Intermediate. The paper is primarily a dataset description with validation experiments; readers benefit from familiarity with object detection concepts such as mAP, IoU, bounding boxes, and few-shot learning.

Scope: The paper introduces MEDISEG, an instance-segmentation annotated medication image dataset of 32 pill types across 8,262 images (plus a focused 3-pill subset of 2,333 images), and validates it with YOLOv8, YOLOv9, and a few-shot FsDet protocol.

What This Paper Is About

Adverse drug events are a serious patient-safety problem, and AI-based pill recognition could help, but existing pill image datasets mostly show single, unobstructed pills under controlled lighting and rarely include complete instance segmentation labels. The authors build and release MEDISEG, a dataset of medication images captured in realistic conditions — multiple pills, dosette boxes, overlap, occlusion, varied lighting and reflections — with pixel-level segmentation masks for every pill. They then test whether models trained on this data work well, and whether pretraining on it transfers to unseen pill types when only a few labelled examples are available.

Key Contributions

1. A new dataset (MEDISEG) with instance segmentation masks for 32 distinct pill types across 8,262 images, split into a MEDISEG (3-Pills) subset (2,333 images, 3 pill categories) and a MEDISEG (32-Pills) subset (8,262 images, 32 categories), distributed in COCO format under a CC BY 4.0 license.
2. Realistic acquisition conditions: images captured with an iPhone 12 Pro Max using a standard four-by-seven dosette box, with manipulated artificial lighting intensity and angle to create authentic shadows, reflections, and highlights, plus overlapping tablets and partial obstruction by dosette box walls.
3. Supplementary regulatory metadata: a metadata.csv file linking each pill to its Hong Kong drug registration number, brand name and dosage strength, certificate holder and address, active ingredients, sale requirement code, registration date, and a link to the Hong Kong Drug Office product page.
4. Benchmarking and few-shot transfer evaluation: supervised validation with YOLOv8 and YOLOv9, plus a controlled FsDet (Faster R-CNN with ResNet backbone and FPN) few-shot protocol comparing base training on MEDISEG against base training on the CURE dataset.

Main Findings

• Supervised detection performance: Training YOLOv8 and YOLOv9 on MEDISEG achieved mAP@50 of 99.5% on the 3-Pills subset and 80.1% on the 32-Pills subset.
• YOLOv9 versus YOLOv8: YOLOv8 converged faster in early epochs on both subsets, but YOLOv9 outperformed it in mAP@50-95, indicating stronger localisation at stricter IoU thresholds. On the 3-Pills validation results at epoch 20, YOLOv8 reached mAP@50 of 0.99432 and mAP@50-95 of 0.95002, while YOLOv9 reached mAP@50 of 0.99462 and mAP@50-95 of 0.96473.
• Residual errors come from visual similarity: Misclassifications on the 3-Pills subset predominantly stemmed from subtle colour variation between Pill B and Pill C, whose side profiles appear strikingly similar even though their top-down views are distinguishable. YOLOv8 exhibited slightly higher misclassification rates overall.
• Few-shot accuracy on overlap-only images (1-shot): A model base-trained on MEDISEG reached foreground classification accuracy of 0.406 versus 0.131 for the CURE-base-trained model, with a false negative rate of 0.513 versus 0.816.
• Few-shot accuracy at 5-shot and 10-shot: MEDISEG base training gave foreground classification accuracy of 0.625 versus 0.372 at 5-shot and 0.740 versus 0.558 at 10-shot, with false negative rates of 0.246 versus 0.465 and 0.210 versus 0.342 respectively.
• Lower few-shot losses with MEDISEG: Base training on MEDISEG was associated with lower region proposal and total detection losses across all few-shot configurations. Total loss was 0.963 versus 1.326 (1-shot), 0.680 versus 0.844 (5-shot), and 0.445 versus 0.674 (10-shot). Classification loss was 0.383 versus 0.351 at 1-shot, 0.279 versus 0.421 at 5-shot, and 0.191 versus 0.320 at 10-shot.
• Standard evaluation set does not discriminate between datasets: On the standard (non-overlap-restricted) evaluation set, both training configurations achieved high foreground classification accuracy even in the 1-shot setting, so this evaluation did not strongly separate the datasets.
• Existing dataset limitations: NIH Pillbox contains 4,392 reference images and 133,774 consumer images but no segmentation labels, and the Pillbox program was discontinued on January 29, 2021, ending creation of the dataset, its image repository, and related APIs. CURE has 8,973 images with partial instance segmentation labels and includes synthetic images. The comparison table in the paper lists Lee et al. (15,031 images, no labels), Wong et al. (5,284 images, no labels), and Tan et al. (5,131 images, bounding box labels), none of which are listed as publicly available.
• Hyperparameter tuning: A genetic algorithm over 70 iterations produced a best fitness score of 0.81253 at iteration 66, using learning rate (lr0) 0.01009, final learning rate (lrf) 0.01, momentum 0.94023, weight decay 0.00048, warmup epochs 2.9974, warmup momentum 0.66015, and box loss weight 7.26074.
• Clinical validation is not claimed: The authors state explicitly that the experiments demonstrate dataset utility and representation transfer but do not constitute clinical validation.
• Not reported in the provided content: Per-class precision, recall, and F1-score tables for the 32-Pills configuration (Tables 6 and 7 are referenced but their contents were not included in the supplied text), and any test-set mAP figures distinct from the validation-set results, which the paper states are the basis for reported results.

Methodology in Plain English

The team photographed medications using an iPhone 12 Pro Max, arranging pills in a standard four-by-seven dosette box and varying the artificial lighting in both intensity and angle across sessions to produce realistic shadows, reflections, and highlights. Each dosette box photo was cropped into individual slot images to reduce background distraction and introduce slight camera-angle variation, then padded and resized to a uniform 640×640 pixels.

Annotation was done manually with COCO Annotator. Every pill instance was outlined with a segmentation mask that traces pill borders even where pills overlap or are partially occluded. Multiple annotators followed strict inter-annotator guidelines, and each annotated image was checked manually by the original annotator and a second reviewer. The pills were selected to be difficult: in the 3-pill version, Pill A and Pill B share a similar shape, forcing the model to rely on colour and fine shape detail, while Pill C's peach-like hue closely resembles Pill B's colour.

To check that the dataset is usable, the authors trained YOLOv8 and YOLOv9 using a hold-out split of 70% training, 20% validation, and 10% testing, with reported results relative to the validation set, and measured precision, recall, F1-score, mAP@50, and mAP@50-95.

For the few-shot experiment, they used the FsDet framework, which separates training into a base learning stage and a few-shot fine-tuning stage. The detector is Faster R-CNN with a ResNet backbone and a Feature Pyramid Network. Base classes are disjoint from the novel classes. In fine-tuning, the backbone extractor and most of the region proposal network are frozen; only the region-of-interest heads for classification and bounding box regression are fine-tuned, with the classification layer re-initialised for the novel classes. Two base training configurations were compared — MEDISEG versus CURE — with all other architectural components, optimisation settings, and fine-tuning schedules kept identical. Evaluation was on held-out multi-pill images, including a challenging subset containing only images with substantial pill overlap or occlusion.

Why This Matters

Impact on research. MEDISEG supplies the combination that prior pill datasets lack: complete instance segmentation labels and genuinely cluttered multi-pill scenes. The few-shot results suggest that exposure to realistic multi-object interactions during base training produces more transferable representations, especially for localisation and discrimination in visually cluttered scenes — a structural realism the authors argue is not well represented in earlier pill datasets.

Real-world applications.

• Bedside medication administration: verifying that the right pills are being given when several are arranged in close proximity, often partially occluded or rotated.
• Dosette box preparation: checking filled trays in community and residential care settings, where pills overlap and sit against box walls.
• Home medication adherence: supporting older adults on complex regimens — the paper cites that more than a third of adults between 75 and 85 take at least five prescription medications daily — where confusion or misidentification, rather than intentional refusal, drives errors.
• Regulatory and traceability workflows: the accompanying metadata links images to Hong Kong drug registration numbers, ingredients, sale-requirement categories (0 for Over-the-Counter, 1 for Pharmacy Only, 2 for Prescription Only), and official product pages, enabling analyses that connect visual characteristics to ingredients or dosage forms.

Industry relevance. The dataset is COCO-format and works directly with Detectron2, PyTorch, TensorFlow, and the COCO API, so it drops into existing pipelines. The code is released as Jupyter Notebooks in Python 3 on GitHub using the Ultralytics library. The finding that a model can adapt to new pill classes with 1, 5, or 10 labelled examples is directly relevant to pharmacies and healthcare providers, for whom new formulations appear regularly and labelled data is expensive. The authors caution that prospective evaluation in live healthcare or pharmacy environments would be needed before deployment.

Future Directions

1. Expand pill coverage and acquisition devices. The authors suggest extending MEDISEG to include additional pill types and diversified acquisition devices, since the current images were captured with a single iPhone 12 Pro Max.
2. Test additional learning paradigms. They propose evaluating the dataset under semi-supervised, continual learning, and domain adaptation settings, which matter for systems that must absorb new drug formulations over time.
3. Clinical and workflow validation. Prospective evaluation in live healthcare or pharmacy environments is needed, particularly to assess performance across broader device variability, lighting conditions, and workflow integration constraints.
4. Exploit the metadata link. The paper notes that merging metadata.csv with the annotation data could support investigation of associations between visual characteristics and specific ingredients or dosage forms, and downstream automated drug identification tasks — an avenue the current work does not pursue.

Target Audience

Researchers and engineers working on medical computer vision, object detection and instance segmentation, and medication-safety AI. It is most useful to those building pill recognition or medication verification systems who need a labelled dataset that reflects real-world clutter, and to methodologists studying few-shot object detection and transfer learning under limited supervision. Clinicians, pharmacists, and health informatics teams evaluating the feasibility of automated medication checks will find the framing and the stated limitations relevant. Beginners can follow the broad argument but will need background in detection metrics to interpret the validation results.

Authors’ abstract

Medication errors and adverse drug events (ADEs) pose significant risks to patient safety, often arising from difficulties in reliably identifying pharmaceuticals in real-world settings. AI-based pill recognition models offer a promising solution, but the lack of comprehensive datasets hinders their development. Existing pill image datasets rarely capture real-world complexities such as overlapping pills, varied lighting, and occlusions. MEDISEG addresses this gap by providing instance segmentation annotations for 32 distinct pill types across 8262 images, encompassing diverse conditions from individual pill images to cluttered dosette boxes. We trained YOLOv8 and YOLOv9 on MEDISEG to demonstrate their usability, achieving mean average precision at IoU 0.5 of 99.5 percent on the 3-Pills subset and 80.1 percent on the 32-Pills subset. We further evaluate MEDISEG under a few-shot detection protocol, demonstrating that base training on MEDISEG significantly improves recognition of unseen pill classes in occluded multi-pill scenarios compared to existing datasets. These results highlight the dataset's ability not only to support robust supervised training but also to promote transferable representations under limited supervision, making it a valuable resource for developing and benchmarking AI-driven systems for medication safety.