DART: A Structured Dataset of Regulatory Drug Documents in Italian for Clinical NLP

AI summary

Overview

Research area: Clinical and biomedical Natural Language Processing (NLP), specifically Italian-language regulatory text mining. Technical level: Intermediate — assumes familiarity with NLP concepts such as section segmentation, few-shot prompting, and named-entity-style information extraction, but the pipeline itself is straightforward to follow. Scope: The paper introduces DART, the first structured Italian corpus of Summaries of Product Characteristics (RCPs), built from roughly 16,000 documents scraped from the Italian Medicines Agency (AIFA), with a demonstration application in automated drug–drug interaction checking.

What This Paper Is About

Most biomedical NLP resources for pharmacology — DrugBank being the standard example — are English-only, which means countries like Italy have no machine-readable counterpart aligned with their own regulatory language or drug labeling conventions. The authors address this gap by building DART, a structured corpus of Italian regulatory drug documents (RCPs, the Italian equivalent of the EU Summary of Product Characteristics) retrieved directly from the AIFA portal. Their goal is both to release the resource and to show that grounding an LLM in these structured regulatory fields measurably improves drug-interaction detection.

Key Contributions

1. A new dataset: DART, the first structured Italian corpus of RCPs, comprising 16,029 documents and over 95 million tokens, drawn from the official AIFA repository and released under CC BY 4.0.
2. A reproducible construction pipeline: an end-to-end, open-source workflow covering undocumented AIFA REST API retrieval, PyMuPDF text extraction, regex-based segmentation into standardized regulatory sections, and validation.
3. LLM-generated clinical summaries: each document is paired with a structured summary (max 450 words) produced by LLaMA 3.1-405B using few-shot prompting and low-temperature decoding, intended to make long regulatory sections usable within LLM context limits.
4. A validated downstream application: an LLM-based drug–drug interaction checker that uses DART summaries as grounding, with a comparative evaluation against four commercial web interaction checkers and seven standalone LLMs.

Main Findings

• Dataset scale and coverage: DART contains 16,029 successfully segmented RCPs (74.55% of the 21,502 initially retrieved), spanning six ATC Level 1 therapeutic classes and over 95 million tokens with a 102,749-term vocabulary.
• High section completeness: Key regulatory sections are present in over 90% of documents — therapeutic indications (97.4%), posology (95.9%), contraindications (94.1%), interactions (92.7%), and undesirable effects (93.2%). Pregnancy/lactation is lowest at 89.6%.
• Segmentation accuracy: On a random sample of 300 documents, more than 97% of expected sections were correctly identified and segmented; residual errors came from non-standard formatting or scanned PDFs.
• Summarization reliability: A manual review of 100 generated summaries found 95% factual consistency with minimal hallucination; deviations were mostly stylistic or omissions of low-priority detail.
• DART grounding dramatically improves DDI recall: Standalone LLaMA-3.1-8B achieved a recall of only 0.229 on the DDI task. With DART summaries as context, recall rose to 0.843 (F1 0.781, accuracy 0.786) — outperforming GPT-4o (recall 0.786) and edging past the best web tool, Drugs.com (recall 0.812).
• Small open models can compete when grounded: Gemma-2-9b + DART roughly doubled its standalone recall (0.214 to 0.457), and Mistral-7B improved from 0.229 to 0.300, though neither matched LLaMA-3.1-8B with DART.
• Commercial tools remain strong baselines: Drugs.com and Medscape achieved F1 scores of 0.756 and 0.734 respectively, showing the problem is not trivially solved, but the gap narrows substantially with regulatory grounding.
• Failure modes are documented: About 4.1% of collected PDFs were excluded because they lacked an embedded text layer (scanned images), and 25.45% of retrieved documents were dropped overall due to incomplete or malformed content.

Methodology in Plain English

The authors built the dataset in three stages. First, because the AIFA website is a single-page JavaScript application that resists static scraping, they inspected browser network traffic to find two undocumented but publicly reachable REST endpoints. One endpoint takes a zero-padded drug code and returns metadata; the second converts that metadata into a direct PDF download link. A Python spider seeded with known drug codes walked through this two-step process and downloaded each RCP.

Second, they extracted text from the PDFs using PyMuPDF, which preserves reading order and rough spatial layout, then split each document into sections by matching the standardized regulatory headers (for example, "04.1 Therapeutic Indications" and "04.8 Undesirable Effects") with a regular expression robust to spacing, punctuation, and capitalization variation. Tables inside section 04.8 were not parsed structurally but their raw text was retained line by line.

Third, they cleaned and standardized the text with regex, discarded documents with empty or corrupt content, and dropped the "05.0 Pharmacological Properties" section entirely because it was too often missing or unusable. What remained was mapped into a table where each row is a drug and each column is a regulatory section.

For the summarization step, seven key sections per drug were fed to LLaMA 3.1-405B via the NVIDIA NIM API at temperature 0.2, using handcrafted few-shot examples to enforce regulatory tone and structure. Output was capped at 450 words. To test the dataset's value, they built a two-stage DDI checker: drug pairs are first represented by their DART summaries, then passed to an LLM that classifies the pair as absent or interacting (with four-level severity collapsed to binary for comparability with clinical tools). Evaluation used 100 manually annotated examples, with recall emphasized because false negatives are the dangerous error in clinical settings.

Why This Matters

This work fills a concrete void: biomedical NLP has been overwhelmingly English-centric, and previously there was no structured, machine-readable resource capturing how Italian regulators actually describe drug indications, contraindications, and interactions. DART makes that regulatory language directly usable by language models, and the DDI experiment shows the payoff is not marginal — recall improved roughly fourfold for one open model simply by grounding it in structured regulatory summaries rather than raw or absent context.

Real-world applications:

• Prescription safety checking: Integrating DART-derived interaction logic into hospital pharmacy systems or e-prescribing software to flag risky drug combinations at the point of care.
• Pharmacovigilance automation: Drafting adverse-event reports and cross-checking regulatory language against national registries such as the Italian Pharmacovigilance Network (RNF) or VALORE.
• Clinical decision support: Populating electronic health record modules with regulatory-grade alerts, contraindication rules, and monitoring recommendations.
• Regulatory document processing: Automating classification, summarization, and controlled rewriting of Italian technical leaflets and RCPs.

Industry relevance: Pharmaceutical companies, hospital IT vendors, and clinical software developers operating in Italy or other non-English European markets gain both a training resource for domain-specific models and a template pipeline that could be replicated for other national regulatory agencies. The finding that small open-source models become viable when grounded in structured regulatory text is directly relevant to organizations that cannot or will not route clinical data through proprietary APIs.

Future Directions

• OCR integration for scanned documents: Roughly 4.1% of PDFs had no embedded text layer; adding optical character recognition would recover them and improve coverage across drug categories.
• Structured table parsing: Section 04.8 (undesirable effects) relies heavily on tabular layouts that were only preserved as raw line-by-line text; layout-aware tools such as pdfplumber or camelot would enable true columnar extraction.
• Expert validation of summaries: The current 95% consistency figure comes from a single manual review of 100 summaries; a formal expert-based validation protocol is described as under development.
• Expansion to other document types and severity granularity: The authors plan to add additional regulatory document classes and therapeutic areas, and to move beyond the binary DDI evaluation back toward the full four-level severity taxonomy.

Target Audience

Clinical NLP and biomedical text-mining researchers, particularly those working on non-English or multilingual resources; pharmacovigilance and drug-safety data scientists; and practitioners building clinical decision support or e-prescribing systems for the Italian or broader European market. The paper is also useful for anyone studying dataset construction pipelines around regulatory sources or evaluating whether retrieval grounding improves LLM reliability in high-stakes domains.

Authors’ abstract

The extraction of pharmacological knowledge from regulatory documents has become a key focus in biomedical natural language processing, with applications ranging from adverse event monitoring to AI-assisted clinical decision support. However, research in this field has predominantly relied on English-language corpora such as DrugBank, leaving a significant gap in resources tailored to other healthcare systems. To address this limitation, we introduce DART (Drug Annotation from Regulatory Texts), the first structured corpus of Italian Summaries of Product Characteristics derived from the official repository of the Italian Medicines Agency (AIFA). The dataset was built through a reproducible pipeline encompassing web-scale document retrieval, semantic segmentation of regulatory sections, and clinical summarization using a few-shot-tuned large language model with low-temperature decoding. DART provides structured information on key pharmacological domains such as indications, adverse drug reactions, and drug-drug interactions. To validate its utility, we implemented an LLM-based drug interaction checker that leverages the dataset to infer clinically meaningful interactions. Experimental results show that instruction-tuned LLMs can accurately infer potential interactions and their clinical implications when grounded in the structured textual fields of DART. We publicly release our code on GitHub: https://github.com/PRAISELab-PicusLab/DART.