Customizing Open Source LLMs for Quantitative Medication Attribute Extraction across Heterogeneous EHR Systems

AI summary

Overview

Research area: Clinical natural language processing and applied large language model (LLM) engineering for Electronic Health Record (EHR) data harmonization, specifically for medications for opioid use disorder (MOUD).

Technical level: Intermediate. The paper assumes familiarity with instruction-tuned LLMs, JSON-constrained generation, and basic clinical data concepts, but its central ideas are explained plainly.

Scope: The paper evaluates whether locally deployable open-source LLMs can extract a unified set of MOUD prescription attributes from five differently formatted clinic EHR systems and compute a standardized per-prescription medication coverage metric ("MOUD days").

What This Paper Is About

MOUD prescription information in real-world health systems is scattered across differently named structured columns and free-text clinical notes, so comparing medication use across clinics normally requires custom, site-specific extract-transform-load (ETL) pipelines that break whenever field names or documentation habits change. The authors build a framework in which an open-source LLM acts as a "universal translator" that converts each clinic's raw records into a single fixed JSON schema, then computes MOUD days from that standardized output. The goal is to measure how well different open-source models perform this extraction quantitatively, and to identify which errors persist and how to fix them.

Key Contributions

1. A practical, privacy-preserving framework that uses instruction-tuned open-source LLMs with constrained JSON generation to extract MOUD prescription attributes (prescription date, drug name, duration, total quantity, daily quantity, refills) from heterogeneous, site-specific EHR data without per-site ETL engineering.

2. A quantitative benchmark: the system is evaluated on prescription-level EHR data from five clinics in a national OUD study — 25,605 records from 1,257 patients in the full dataset, with a manually annotated ground truth of 10,369 records from 776 patients, annotated by UCLA medical professionals.

3. A standardized metric, "MOUD days," computed per record as (number of refills + 1) × duration when duration is provided, or (number of refills + 1) × (total quantity / daily quantity) when duration is missing.

4. A documented error taxonomy with concrete fixes: imputing missing dosage fields from within-drug norms, handling monthly or weekly injectables (e.g., Vivitrol) by setting duration from the documented schedule, and adding unit checks so mass units (e.g., "250 g") are not misread as daily counts.

Main Findings

• Best overall models: Qwen2.5-32B achieved 93.4% overall coverage with 93.0% overall exact-match accuracy across clinics. MedGemma-27B achieved a comparable 93.1% coverage and 92.2% accuracy.

• Large models generally win: Among the evaluated models, the larger ones produced the best results overall. Qwen3-32B reached 75.6% coverage / 90.9% accuracy; Gemma 3-27B reached 77.0% / 91.1%.

• Smaller models were competitive on accuracy but inconsistent on coverage: Qwen3-8B reached 69.4% coverage / 85.2% accuracy and Qwen3-4B reached 54.0% / 91.0%; MedGemma-4B reached 34.3% / 68.1%; Gemma 3-4B reached 65.5% / 78.0%. The paper's prose states the Qwen3-8B overall coverage "dropping to 48.0%," which does not match the 69.4% reported for that model in Table 2.

• Sub-1B models were excluded: Initial experiments with sub-1B models revealed significant difficulties in generating reliable structured outputs, leading the authors to focus on larger models.

• Coverage varied sharply by clinic: For Clinic B and Clinic D, most models achieved excellent coverage (often >95%) with high accuracy (>90%). Clinic A and Clinic C showed wide variance — the paper's text states that one model scored 0% on Clinic A, which does not match the table entry for Qwen3-32B on Clinic A (42.86% coverage / 66.67% accuracy). Clinic E was intermediate: Qwen3-32B reached 93.66% coverage there, while Gemma 3-4B reached 19.95%.

• High accuracy on what is processed, but coverage gaps: The authors summarize two primary findings: significant performance variation across clinic datasets (reflecting data heterogeneity), and a consistent trend in which models achieve high exact-match accuracy on records they successfully process but often struggle with overall record coverage.

• Three recurring error modes with domain-expert fixes:

  • Clinic B had substantial missingness in dosage instructions (SIG) and daily quantity, and gaps in total prescribed quantity — about 20% of records had key missing entries. Daily quantity was imputed using the typical value observed for the same drug name, and total quantity was imputed with the median value.
  • About 5% of prescriptions in Clinic C and Clinic E were monthly or weekly extended-interval injections (e.g., Vivitrol). LLMs frequently failed to translate these schedules into structured daily and total quantities, so duration was manually set from the documented dosing schedule.
  • Unit-related outliers appeared in LLM outputs: daily quantity is usually in tablets or films, but occasional entries listed mass units (e.g., "250 g"), which models misread as a daily count (e.g., 250). Unit normalization and plausibility checks were added, including capping extreme values.

Methodology in Plain English

The framework has three stages. First, data preparation: raw prescription records are collected from five clinics (labeled A through E) participating in a multi-site national study on OUD. Each clinic stores its attributes in its own differently named fields — for example, Clinic A uses BRAND_NAME, UNIT_DOSE, and DOSAGE_INSTRUCTIONS, while Clinic B uses epic_medication_name, dose_unit, and dose_instructions. The full dataset contains 25,605 records from 1,257 patients, and a manually annotated subset of 10,369 records from 776 patients serves as ground truth.

Second, extraction: models are prompted with clinic-specific templates and constrained to emit output matching a predefined Pydantic-based JSON schema, rather than free text. This guarantees syntactically correct structured output. A shared system prompt establishes a "medical expert" persona, asks the model to reason about the answer in a reasoning field, and provides interpretation rules — for example, converting fraction notations like "3 1/2" to 3.5, treating "X10" as 10 days under specified conditions, summing multiple doses in a day (e.g., "one tab in morning, half tab at night" = 1.5), and converting frequency terms such as weekly = 7 days and monthly = 30 days. Because of HIPAA and privacy requirements, only models deployable locally were considered; closed-source APIs such as GPT-4 or Gemini were excluded. Models were served with vLLM on a single node with 4 NVIDIA A6000 GPUs (48GB VRAM each), using quantization where applicable and tensor parallel processing.

Third, post-processing and evaluation. Outputs are type-normalized (dates, integers), validated with rule-based cross-field checks (for example, total quantity must not be less than daily quantity), and used to compute MOUD days. Evaluation reports two metrics: coverage (the percentage of ground truth records for which the model produced a parsable output matchable on a composite key of clinic name, patient ID, prescription date, and drug name) and record-level exact-match accuracy (the percentage of covered records where extracted attributes and MOUD days perfectly match ground truth after normalization).

Why This Matters

Impact on research: The framework removes the need for brittle, site-specific ETL code and supports local deployment, which matters for multi-site studies where data cannot leave a clinic. This enables consistent cross-site analyses of MOUD exposure, adherence, and retention in real-world settings, particularly in the rural communities the authors highlight as underserved.

Real-world applications:

• Multi-site comparative studies of MOUD prescribing patterns, treatment retention, and adherence without building separate extraction pipelines per health system.
• Quality improvement and monitoring programs that need standardized medication coverage durations across clinics with incompatible record formats.
• Privacy-preserving clinical analytics in which models run on local hardware rather than sending protected health information to external APIs.
• Feasibility work for extending the same constrained-JSON extraction pattern to other medication classes or clinical concepts, with additional prompt engineering and evaluation.

Industry relevance: The paper shows that models capable of running on a single GPU can harmonize prescription data across five disparate clinical systems without site-specific engineering, which is relevant to health systems, health IT vendors, and payers that need to integrate heterogeneous records. It also gives concrete, reusable engineering practices — constrained JSON generation, Pydantic schema validation, cross-field consistency checks, unit normalization — plus an explicit caution that human-in-the-loop review is essential for high-stakes clinical use.

Future Directions

1. Broader validation across EHR systems: The framework was tested on five diverse clinics, but generalizability to a wider range of EHR systems and documentation styles remains to be validated.

2. Testing reasoning-focused models: The authors expect newer reasoning models will handle heterogeneous EHRs with less prompt customization and list evaluating that hypothesis as a priority.

3. Efficient fine-tuning for smaller specialized models: They plan to explore techniques such as LoRA to create smaller, specialized models, and to benchmark against reasoning-focused models that have shown strong performance on verifiable tasks like math and coding.

4. Extending beyond MOUD and building a review interface: Extending the approach to other medication classes or clinical concepts requires further prompt engineering and evaluation, and developing a user-friendly interface for expert review and error adjudication is described as a key priority for clinical readiness.

Target Audience

Clinical informatics researchers and health data engineers who need to harmonize medication data across multiple health systems; applied machine learning practitioners interested in constrained structured generation with open-source LLMs; opioid use disorder and addiction medicine researchers conducting multi-site observational studies; and health IT or regulatory-adjacent readers evaluating whether locally deployable LLMs can substitute for traditional ETL pipelines under privacy constraints. Readers seeking deployment-ready clinical tooling should note the authors' stated limitations: the models are not clinical-grade, only MOUD prescriptions were tested, and human-in-the-loop review is described as essential for high-stakes applications.

Authors’ abstract

Harmonizing medication data across Electronic Health Record (EHR) systems is a persistent barrier to monitoring medications for opioid use disorder (MOUD). In heterogeneous EHR systems, key prescription attributes are scattered across differently formatted fields and freetext notes. We present a practical framework that customizes open source large language models (LLMs), including Llama, Qwen, Gemma, and MedGemma, to extract a unified set of MOUD prescription attributes (prescription date, drug name, duration, total quantity, daily quantity, and refills) from heterogeneous, site specific data and compute a standardized metric of medication coverage, \emph{MOUD days}, per patient. Our pipeline processes records directly in a fixed JSON schema, followed by lightweight normalization and cross-field consistency checks. We evaluate the system on prescription level EHR data from five clinics in a national OUD study (25{,}605 records from 1{,}257 patients), using a previously annotated benchmark of 10{,}369 records (776 patients) as the ground truth. Performance is reported as coverage (share of records with a valid, matchable output) and record-level exact-match accuracy. Larger models perform best overall: Qwen2.5-32B achieves \textbf{93.4\%} coverage with \textbf{93.0\%} exact-match accuracy across clinics, and MedGemma-27B attains \textbf{93.1\%}/\textbf{92.2\%}. A brief error review highlights three common issues and fixes: imputing missing dosage fields using within-drug norms, handling monthly/weekly injectables (e.g., Vivitrol) by setting duration from the documented schedule, and adding unit checks to prevent mass units (e.g., ``250 g'') from being misread as daily counts. By removing brittle, site-specific ETL and supporting local, privacy-preserving deployment, this approach enables consistent cross-site analyses of MOUD exposure, adherence, and retention in real-world settings.