Building Patient Journeys in Hebrew: A Language Model for Clinical Timeline Extraction

AI summary

Overview

• Research area: Clinical natural language processing; temporal relation classification (TRC) over electronic health records (EHRs) in Hebrew, a low-resource language.
• Technical level: Intermediate. The paper assumes familiarity with encoder-based language models, masked language modeling, tokenization, and fine-tuning, but its applied framing is accessible to clinical informatics readers.
• Scope: The paper describes HeMed, a Hebrew medical encoder language model built by continually pre-training DictaBERT 2.0 on approximately five million de-identified hospital records, and evaluates it on two newly annotated Hebrew temporal relation classification datasets, Med-TRC and Onc-TRC.

What This Paper Is About

Clinical notes record a patient's symptoms, diagnoses, treatments, and outcomes over time, but the temporal ordering of these events is buried in unstructured text. The authors formulate patient-journey extraction as temporal relation classification: given pairs of medical events in a note, the model decides whether one occurred before, after, or at the same time as the other (or is vague). Because all documentation at their medical center is written in Hebrew and existing medical language models are almost exclusively English, they build the first Hebrew medical TRC model trained on real clinical notes and create the first Hebrew benchmark datasets for this task.

Key Contributions

1. First Hebrew medical TRC model trained and evaluated on real-world clinical notes from the Tel Aviv Sourasky Medical Center (referred to as "the hospital"), supporting time-aware clinical NLP in a low-resource setting.
2. A large-scale Hebrew clinical corpus: approximately five million de-identified records (3GB uncompressed), containing around 0.5 billion words, or roughly 0.7 billion tokens under DictaBERT's tokenizer. This corpus is used to adapt a foundational Hebrew model via continued pre-training into the resulting model, HeMed.
3. Two new Hebrew TRC datasets spanning different clinical domains — Med-TRC (internal medicine and emergency departments) and Onc-TRC (breast oncology clinics) — providing the first benchmark for temporal reasoning in Hebrew EHRs.
4. Evidence that continued pre-training on clinical notes significantly improves performance over the original base model, alongside systematic study of vocabulary adaptation, segmentation, and de-identification choices.

Main Findings

• Adapted models beat the baseline: In both Med-TRC and Onc-TRC, the continually pre-trained models consistently outperformed DictaBERT 2.0 fine-tuned on each task without the clinical pre-training step.
• Low-data benefit (Med-TRC): On Med-TRC, the improvement diminished as training set size increased, suggesting continual pre-training helps most when labeled data is scarce.
• Sustained benefit (Onc-TRC): On Onc-TRC, the gain remained consistent for some model variations even as training size grew.
• No single winner across tasks: Model variations performed similarly on Med-TRC, with AdaLM lagging slightly behind; on Onc-TRC, AdaLM outperformed the others by a small margin.
• Vocabulary adaptation improves token efficiency: On the 10K test split, DictaBERT's original vocabulary produced a corpus token count (CTC) of 1,339,002 and compression rate of 1.45. Simple adaptation gave 1,318,682 and 1.43; Simple+FLOTA gave 1,321,248 and 1.43; AdaLM gave 1,232,328 and 1.34 — AdaLM improved both metrics over simple adaptation.
• New tokens added: The simple approach added 2,011 tokens beyond those already in DictaBERT's 128K vocabulary; AdaLM added 3,000 new tokens.
• FLOTA segmentation had no significant impact on performance across tasks when applied during both continual pre-training and fine-tuning.
• Extra epochs: Moving from one epoch to two or three slightly improved Med-TRC but had no effect on Onc-TRC.
• De-identification is safe for performance: Comparing the simple vocabulary-adaptation model trained on the de-identified corpus versus the original corpus, performance degradation was minimal despite replacing approximately 2.3M substrings, about 5% of the words in the corpus.
• English models underperformed: English medical encoder models run as general baselines all performed worse than the Hebrew baseline models.
• Comparable to English results: The authors state the performance achieved is comparable to results reported on standard English pairwise TRC datasets.
• Robustness: Each fine-tuning process was repeated 15 times with different random seeds; standard deviations were all within the range of 1–2 minor points.

Methodology in Plain English

The work proceeds in two steps.

Step 1 — Building the medical language model. The authors collected textual EHRs from several hospital divisions chosen for diversity and low duplication: internal medicine departments, the emergency room, the children's emergency room, and the emergency room for labor and delivery. The included record types were visit summaries, status descriptions, discharge letters, follow-ups, and diagnoses. The final corpus is 5,232,028 records, 546,606,802 words, and 3,185,476,400 characters (word counts estimated by splitting strings on white spaces). They started from DictaBERT 2.0 — chosen after preliminary experiments showed HeBERT performing worse and DictaBERT 2.0 slightly outperforming AlephBERTGimmel — and continued pre-training it with the masked language modeling objective.

They also explored adapting the tokenizer, training each new tokenizer on one million randomly sampled records. The "simple" method trains a 10,000-token WordPiece tokenizer on the clinical corpus and takes the union with DictaBERT's 128K vocabulary, adding 2,011 genuinely new tokens. The AdaLM method (delta = 0.1, step size 1,000) added 3,000 tokens, with new token embeddings initialized as the average of their sub-token embeddings. For segmentation, they compared standard WordPiece (longest token from the start of a word) against FLOTA (greedy longest available token anywhere in a word).

For privacy, they first tried the open-source Safe Harbor Hebrew de-identification system, but found it missed some names and locations and over-masked — for example, misclassifying blood-pressure readings like "120–90" as identity numbers and replacing disease, procedure, and drug names with personal-name placeholders. They replaced it with a custom rule-based approach that pulls patient, relative, contact, and doctor names from hospital databases and searches for them via regular expressions that account for Hebrew morphology, masks organizations only from a manually compiled institution list, and swaps detected entities for realistic fake alternatives rather than generic placeholders. In total, 2,335,950 substrings were replaced.

Step 2 — Fine-tuning on temporal relations. Two professional annotators, both experienced nurses, labeled two datasets following MATRES annotation guidelines as adapted to Hebrew by Yanko et al. (2023). Events are marked in advance, and each anchored event pair receives one of four labels: before, after, equal, or vague. An extra "invalid" label captured pairs where at least one event was wrongly identified. In Med-TRC, events were identified as verbs using a Hebrew part-of-speech tagger from the Stanza library; in Onc-TRC, a predefined list of about 300 oncological terms (procedures, medications, biomarkers, and other key terms) was matched with a simple algorithm plus regular expressions for morphological variation. Only event pairs within a sliding window of two sentences (delimited by hard punctuation) were annotated.

Annotation quality was checked by having both annotators independently label 15% of records; Cohen's Kappa for Onc-TRC was 0.8, and for Med-TRC no formal Kappa was measured but qualitative assessment indicated high agreement. Both datasets contain more before pairs than any other class, so the authors clipped the number of before instances to match the count of the second most common class, then split into train and test sets: Med-TRC has 1,829 training and 503 test samples; Onc-TRC has 1,078 training and 283 test samples.

Fine-tuning used the event start state (ESS) architecture from Yanko et al. (2023) and the weighted average F1 metric, plus a relaxed F1 that disregards errors where non-vague predictions are made on vague instances. All models were trained with a learning rate of 3×10⁻⁶, a batch size of 16, on a single NVIDIA A40 GPU (48GB), with each epoch taking approximately 63 hours. Experiments used training-set sizes of 25, 50, 75, 100, 250, and so on. Code and supplementary material are released under an Apache-2.0 license.

Why This Matters

Impact on research. The paper extends temporal relation classification — heavily studied in English through resources like MATRES (12,736 training and 837 test instances) — to Hebrew, where previously only one publicly available TRC dataset existed (Yanko et al., 2023, with 7,260 annotated samples). It provides a reusable Hebrew medical encoder, a five-million-record clinical corpus recipe, and two annotated benchmarks, while reinforcing the finding that encoder-based models remain competitive on TRC relative to generative LLMs and are smaller and cheaper to deploy.

Real-world applications:

• Patient timeline construction: Turning scattered notes into an ordered sequence of events to summarize a patient's journey.
• Clinical summarization and disease progression modeling: Using the temporal graph to condense long histories and track how conditions evolve.
• Diagnosis prediction and treatment evaluation: Sequencing events to understand what was done when and what followed.
• Temporal reasoning in decision support: Contextualizing and prioritizing interventions based on the order in which events occurred.

Industry relevance. Encoder models require fewer optimization steps and less training data than large generative models, and can be fine-tuned on internal hospital equipment — an important practical constraint. The finding that de-identification does not degrade downstream performance (despite replacing about 5% of words) shows that privacy-conscious clinical model development is viable without sacrificing utility, and that vocabulary adaptation makes more efficient use of a fixed 512-token context window for dense, time-sensitive narratives.

Future Directions

• Releasing the datasets. Med-TRC and Onc-TRC cannot be publicly released because they may contain sensitive information; finding a way to share or replicate them would broaden benchmarking.
• Extending beyond pairwise TRC. The authors restrict themselves to pairwise classification because encoder models perform poorly on global TRC (predicting a full document temporal graph in one step); closing this gap is an open problem.
• Additional clinical domains. Only internal medicine/ER and breast oncology were covered; other specialties, and the corpus's other record types, remain untested.
• Longer pre-training and broader model comparison. Extra epochs helped Med-TRC slightly but not Onc-TRC, and no single vocabulary-adaptation method won on both datasets, leaving room to explore when each adaptation strategy is preferable.

Target Audience

• Clinical NLP and medical informatics researchers, especially those working on temporal reasoning, clinical timelines, and domain adaptation of language models.
• Hebrew and low-resource language researchers seeking a case study in adapting a general-purpose model to a specialized, under-served language.
• Hospital IT and clinical data science teams considering whether to build in-house encoder models for structured extraction from EHR notes.
• Privacy, ethics, and regulatory reviewers interested in evidence that de-identified clinical corpora can support performant models; note the model is released for research use only, restricted to formal medical institutions under IRB-approved protocols.

Authors’ abstract

We present a new Hebrew medical language model designed to extract structured clinical timelines from electronic health records, enabling the construction of patient journeys. Our model is based on DictaBERT 2.0 and continually pre-trained on over five million de-identified hospital records. To evaluate its effectiveness, we introduce two new datasets -- one from internal medicine and emergency departments, and another from oncology -- annotated for event temporal relations. Our results show that our model achieves strong performance on both datasets. We also find that vocabulary adaptation improves token efficiency and that de-identification does not compromise downstream performance, supporting privacy-conscious model development. The model is made available for research use under ethical restrictions.