Toward Automatic Filling of Case Report Forms: A Case Study on Data from an Italian Emergency Department

AI summary

Overview

Research area: Natural Language Processing applied to clinical text — specifically, information extraction from Italian emergency department notes into structured Case Report Forms (CRFs).

Technical level: Intermediate. Readers will benefit from familiarity with standard NLP evaluation (precision, recall, F1), prompting-based LLM use, and basic clinical-research vocabulary, though the paper explains its task and metrics clearly.

Scope: The paper introduces a new annotated Italian clinical-note dataset aligned to a 134-item CRF schema, defines the CRF-filling task and its evaluation metrics, and reports zero-shot pilot results with an open-source LLM.

What This Paper Is About

Clinical research depends on Case Report Forms, structured templates that record patient data in a consistent format, but filling them from free-text clinical notes is still largely manual. Progress toward automating this has been held back by a scarcity of annotated CRF data drawn from realistic hospital settings. This paper addresses that gap by releasing an annotated dataset of 290 Italian emergency-care notes paired with a 134-item CRF, and by testing whether a state-of-the-art open-source Large Language Model can fill the form automatically.

Key Contributions

1. A new annotated clinical resource. A dataset of 290 emergency-care notes from the San Giovanni Bosco Hospital in Turin, covering adult patients presenting between 1 January 2021 and 31 December 2023, annotated by clinicians against a 134-item CRF schema. The authors state this is, to the best of their knowledge, the first public dataset available for the CRF-filling task on data coming from hospitals. It is released under a Creative Commons licence.

2. A task definition with evaluation metrics. The paper formalizes CRF filling as the automatic population of each item with a value from a predefined closed set, evaluated by exact match using micro-F1 (equivalent to overall accuracy in this single-label setting) and macro-F1, with "unknown" always a valid answer.

3. A characterization of the data. An exploratory analysis documenting annotation density, CRF coverage, redundancy, conflicts, note-length variation, and the distribution of annotations across CRF groups.

4. A pilot experiment demonstrating feasibility. A zero-shot study with an open-source LLM, compared against a most-frequent-value baseline, showing both that the approach is viable and that it is not yet sufficient for clinical quality.

Main Findings

• The dataset is sparse and imbalanced. Each note contains on average 5.7 annotated items, and about 20% (58/290) are not filled at all. Label density rarely exceeds 15 items per note, with a few cases reaching up to 40. Since the CRF has 134 fields, well over 120 items per note on average are assigned "unknown". Notes average 131 words, with outliers up to 400.

• CRF coverage is incomplete. Roughly three-quarters of the 134 items appear at least once as an annotation; the remaining quarter never appear in the data.

• Annotations reflect real documentation behavior. 28% of notes contain repeated annotations of the same item with the same value, and about 0.8% of notes assign the same item more than once with different values. The authors state these are not annotation errors but reflect how clinical concepts are documented over time.

• Annotation distribution across CRF groups is highly skewed. Clinical examination (24 items, 676 labels) and history taking (46 items, 661 labels) dominate, together accounting for over 80% of all observed annotations (1337/1648). Laboratory results (24 items, 148 labels), imaging results (10 items, 85 labels), treatment (7 items, 29 labels), final diagnosis (20 items, 27 labels), and diagnostic test results (3 items, 22 labels) contribute far less.

• The baseline is deceptively strong. The most-frequent-value baseline, which always predicts "unknown", achieves a micro-F1 of 0.963 but a macro-F1 of only 0.353, showing it fails to recover informative but less frequent values.

• The LLM trades a little micro-F1 for much better macro-F1. The pilot model obtains a micro-F1 of 0.920 against the baseline's 0.963, while improving macro-F1 to 0.548 versus 0.353.

• Accuracy is dominated by the majority class. 90.42% of the model's predictions correctly identify "unknown", and items with a non-"unknown" gold label are rarely recovered (1.57% correct). In the dataset overall, 96.38% of values are "unknown" and 3.62% are not.

• Precision and recall are balanced but modest. Macro-averaged precision is 0.562 and recall 0.569, suggesting the model does not strongly favor one over the other.

• The model respects the schema. It consistently selected only from the predefined set of valid values.

Methodology in Plain English

The researchers worked with an Italian Emergency Department to build a realistic test bed. Clinicians used Label Studio software to annotate 290 anonymized notes, marking the exact text fragments that support each CRF item and the value assigned to it. All identifiers — names, phone numbers, addresses, ID codes — were removed by a third-party collaborator, and no pseudonyms or internal patient codes were retained, which means notes from the same patient cannot currently be linked together.

The CRF itself has 134 items split into seven topical groups: History taking, Clinical examination, Diagnostic test results, Laboratory test results, Imaging test results, Treatment, and Final diagnosis. Each item has a closed set of allowed values, typically binary (yes/no, present/absent), ordinal (for example, below norm / within norm / above norm for body temperature), or a measured/unknown distinction. For every item, "unknown" is always a valid answer.

The task is framed as a zero-shot prompt-based problem. Each note-item pair is queried independently, so the same note is processed 134 times, once per CRF item, which avoids cascading errors across fields. Prompts contain four parts: a system role description, a brief introduction to the item group, an item-specific instruction listing allowed values, and the full Italian clinical note. CRF items and values are in English while notes are in Italian, so the model must handle cross-lingual comprehension.

Inference used vLLM on a single NVIDIA A40 GPU (48 GB VRAM), with a maximum context length of 4096 tokens and temperature set to 0 for deterministic output. The baseline assigns every item its most common value, which in all cases is "unknown".

Why This Matters

Impact on research. The paper's central claim is that the field has lacked annotated CRF data from realistic hospital settings, which has limited progress. By releasing a dataset drawn from actual emergency-care documentation — with its heterogeneity, sparsity, and imbalance — the authors provide a challenging and realistic benchmark rather than a synthetic one. The finding that LLM outputs are affected by biases, such as a cautious behavior that favors "unknown" answers, gives the community a concrete failure mode to target.

Real-world applications:

• Clinical trial data collection — automating the population of electronic Case Report Forms from narrative notes, a step beyond the paper-based-to-electronic transition that already improved usability and reduced transcription errors.
• Emergency department documentation — reducing manual effort in high-pressure settings where vital signs and history are recorded for nearly all patients but detailed test results and diagnoses only for some.
• Electronic health record integration — structured extraction that could feed directly into EHR systems rather than requiring separate data entry.
• Clinical research on specific conditions — the CRF design reflects a documented use case focused on dyspnea and transient loss of consciousness, showing how extraction can be tailored to a study's objectives.

Industry relevance. The work sits at the intersection of clinical research infrastructure and applied NLP. Pharmaceutical and academic trial sponsors, EHR vendors, and clinical NLP developers all depend on reliable conversion of unstructured narratives into structured fields. The paper's framing of evaluation — arguing against relying on micro-F1 alone when most values are legitimately "unknown" — is directly useful to anyone building deployment-grade extraction systems.

Future Directions

• Correct the systematic biases. The authors conclude explicitly that LLM biases, including a cautious tendency toward "unknown", need to be corrected.
• Move beyond out-of-the-box models. The results indicate that off-the-shelf tools are not sufficient and that dedicated LLM-based solutions must be engineered to reach clinical quality requirements.
• Scale up experimentation. The paper calls for systematic, large-scale experimentation to develop models that can meet clinical standards, and the framework is described as open to evaluating a wide range of open and closed LLMs in future studies.
• Enable cross-note linkage. Future dataset releases are planned with refined anonymization procedures so that multiple notes from the same patient can be connected without compromising confidentiality, which would support longitudinal reconstruction and may make redundancy and conflict phenomena more prominent.

Target Audience

Clinical NLP researchers working on information extraction and structured data generation; medical informatics and health-data-science teams evaluating LLMs for real-world deployment; clinical researchers and trial coordinators interested in automating CRF population; and NLP practitioners who want a concrete, well-documented example of a sparse, highly imbalanced multi-label extraction task where simple accuracy metrics are misleading.

Authors’ abstract

Case Report Forms (CRFs) collect data about patients and are at the core of well-established practices to conduct research in clinical settings. With the recent progress of language technologies, there is an increasing interest in automatic CRF-filling from clinical notes, mostly based on the use of Large Language Models (LLMs). However, there is a general scarcity of annotated CRF data, both for training and testing LLMs, which limits the progress on this task. As a step in the direction of providing such data, we present a new dataset of clinical notes from an Italian Emergency Department annotated with respect to a pre-defined CRF containing 134 items to be filled. We provide an analysis of the data, define the CRF-filling task and metric for its evaluation, and report on pilot experiments where we use an open-source state-of-the-art LLM to automatically execute the task. Results of the case-study show that (i) CRF-filling from real clinical notes in Italian can be approached in a zero-shot setting; (ii) LLMs' results are affected by biases (e.g., a cautious behaviour favours "unknown" answers), which need to be corrected.