NOAH: Learning the Full Patient Journey. A Longitudinal Multimodal Time-Aware Model for Representation and Forecasting

AI summary

Overview

Research area: Medical AI / clinical machine learning — longitudinal multimodal patient modeling, generative sequence modeling, and representation learning for electronic health records.

Technical level: Intermediate. The core ideas (transformers, latent variable models, autoregressive forecasting) are explained in terms accessible to readers with some ML background, though the paper assumes familiarity with transformer architectures and variational inference.

Scope: The paper introduces NOAH, a time-aware generative transformer pretrained on 559 million clinical events that jointly represents and forecasts full multimodal patient journeys across imaging, waveforms, signals, text, and structured records.

What This Paper Is About

Electronic health records contain a patient's entire medical life — vitals, imaging, ECGs, lab values, clinical notes, medications, procedures, and administrative events — collected at irregular intervals over decades. Existing AI models capture only fragments of this picture: they tend to be discriminative rather than generative, limited to one or two modalities, restricted to fixed code vocabularies, or treat time as a simple monotonic ordering that ignores how relevant a past event is to a prediction at a specific future horizon.

NOAH's goal is to build a single, task-agnostic generative foundation model that consumes the actual content of every clinical modality across a patient's lifetime, learns a latent space of patient states and transitions, and can then both describe the present state and roll forward plausible futures — including simulating what would happen under a different treatment.

Key Contributions

1. A holistic generative architecture for patient journeys. NOAH is a causally masked transformer decoder that natively ingests medical images (X-ray, echocardiogram video), ECG waveforms, numeric and categorical measurements, free text, and structured clinical records — in arbitrary order, count, and temporal spacing — within a single token stream.

2. Bidirectional time integration. Temporal information is injected both at the input tokens and inside the attention module, where backward-oriented time features are added to keys and forward-oriented features (including the time delta to the next predicted event) to queries. This lets the model weight a patient's history differently depending on how far ahead the prediction reaches, and enables optional time control during generation.

3. A context-conditional variational latent space. A prior distribution conditioned only on the previous transformer state and a posterior conditioned additionally on the current event are aligned via a KL divergence, creating a learned space of patient states and transitions. This supplies stochasticity at inference time while keeping the components of each generated event (type, value, timing) semantically coherent.

4. Demonstrated versatility across clinical tasks. The same pretrained model supports autoregressive forecasting with optional time control, zero-shot outcome classification via Monte Carlo rollouts, counterfactual intervention simulation, and highly informative frozen patient-state embeddings for probing and survival analysis.

Main Findings

• The latent space is clinically organized. Patient state trajectories unfold and diverge over time according to care level and risk. Patients with similar starting conditions but different outcomes separate in embedding space. Regions emerge for long-term ICU, general hospital, emergency department, and outside care, with a distinct high-risk zone adjacent to the ICU region.

• Model "surprise" tracks clinical reality. The divergence between the model's prior expectation and posterior belief at each transition spikes shortly before formally charted stay onsets, peaking for emergency department vitals and then drug administration and immediate procedures. Surprise decreases as context accumulates, and is highest for content-rich modalities like ECG and X-ray.

• Forecasting is accurate and improves with time control. Across 13,551 test patients, NOAH achieves AUROC of 0.83–0.95 and Brier scores of 0.03–0.11 for predicting which event types and modalities occur within a horizon, beating a persistence baseline. Calibration is reasonable (ECE 0.08–0.094) with slight overconfidence at high probabilities. Providing the ground-truth time delta to the next token improves AUROC by up to +0.19 in short horizons, though type-count error grows over long horizons under enforced timing.

• Zero-shot clinical decisions via Monte Carlo rollouts. Sampling multiple future trajectories per patient yields AUROC 0.97 (balanced accuracy 0.82) for 72-hour mortality and 0.74 (0.68) for length of stay ≥ 72 hours when prompted at 48 hours after admission. Thirty-day readmission is much harder (AUROC 0.61), attributed to autoregressive drift over long horizons and matching clinical intuition about the difficulty of that task.

• Counterfactual treatment simulation matches a real trial. For sepsis patients, swapping saline for lactated Ringer's as the first intravenous fluid produced a simulated effect in favor of Ringer's on mortality, length of stay, and MAKE-30, matching the sign of the effect in the SMART trial's sepsis subgroup at roughly twice its magnitude. NOAH overpredicts mortality, which the authors trace to rollout drift rather than to the within-patient contrast itself.

• Frozen embeddings carry rich clinical information. Linear probes on final stay representations reach AUROC 0.72–0.94 across 15 ICD chapters and 29 Quan-Elixhauser comorbidities, and 0.87, 0.98, and >0.99 for ICU length of stay, hospital length of stay, and mortality. A regression MLP recovers the NEWS2 stability score with MAE 1.34 and AUROC 0.94/0.91 for risk separation.

• Survival analysis improves with more longitudinal context. Time-to-death prediction from embeddings extracted later in the stay shows higher time-dependent C-index while remaining well-calibrated by D-calibration.

• The model learns physiology, not just structure. Of 144 attention heads, 141 specialize in clinically meaningful relations: 76 link events of the same type across time (maintaining per-modality history) and 65 connect observations to their reporting. Ablations show the time-content attention term dominates after the first layer, meaning the model weights history differently depending on prediction horizon, and attribution analysis shows measurements drive next-token prediction rather than administrative structure alone.

Methodology in Plain English

The researchers merged six components of the MIMIC data family (MIMIC-IV, MIMIC-ED, MIMIC-Note, MIMIC-CXR-JPG, MIMIC-IV-ECG, MIMIC-ECHO) on patient identifiers to build lifetime timelines covering 299,000 patients and 431,000 hospital visits. Every charted data point becomes a standalone timestamped event; events that span time, like procedures, get explicit end tokens inserted at their real termination time. Dense ICU vital measurements are compressed with a sliding window that keeps the minimum and maximum per vital type to prevent high-frequency monitoring from dominating the sequence.

Preprocessing splits patients 70/15/15 with no patient appearing in two splits. Training sequences are sliding windows of at most 2,048 tokens with a fixed demographic prefix (date of birth, race, gender, insurance, language, marital status) and a masked overlap region. Heavy modalities are handled by frozen external encoders: RAD-DINO for images, BioLORD for text, OTIS for ECG waveforms, and EchoPrime for echocardiogram videos. Numeric scalars become Fourier features, and lightweight trainable projection layers map everything into a shared token space.

The architecture is a causally masked transformer decoder. Instead of a standard positional encoding, time enters twice: once at the input and again inside attention, where backward-looking time features are added to keys and forward-looking features to queries. Between the transformer output and the decoding heads sits a variational bottleneck — a prior from the previous timestep and a posterior that also sees the current event — trained with a reconstruction loss plus a KL term. Each event component (type, time, value, free-text specifics) has its own decoding head, but all are sampled from a single shared latent state so generated events stay internally consistent. At inference, the model rolls forward autoregressively, optionally told the true time delta to the next event, and clinical outcome probabilities are estimated by drawing many trajectories and counting outcomes, Monte Carlo style.

Why This Matters

Impact on research. NOAH consolidates several lines of prior work — multimodal health record tokenization (Tamme, Apollo), variational sequence modeling (VRNNs), and rotary time encoding — into a single generative framework that is simultaneously a representation encoder and a forecasting engine. It pushes clinical foundation models past discriminative classification toward task-agnostic generation with an explicit source of stochasticity, which is a structural shift for the field. The detailed interpretability analysis (attention head specialization, temporal ablation, integrated-gradients attribution) also sets a precedent for auditing medical foundation models rather than treating them as black boxes.

Real-world applications:

• Risk stratification at admission. Frozen patient-state embeddings feed lightweight linear models that retrieve ICD chapters, comorbidities, and deterioration scores without any downstream system ever needing to process raw imaging or waveforms directly.
• Zero-shot prognosis and length-of-stay planning. Monte Carlo rollouts estimate mortality and prolonged-stay probabilities from a patient's history alone, without task-specific training data.
• Counterfactual treatment exploration and in silico trials. The sepsis saline-versus-Ringer's demonstration shows the model can generate plausible alternative courses under a changed intervention, supporting hypothesis generation before prospective studies.
• Longitudinal monitoring. Because the model consumes outside-hospital measurements and medication data, it can react to abnormal outpatient readings that precede formal admissions — useful for early-warning systems and remote patient management.

Industry relevance. Health systems, EHR vendors, and clinical AI companies looking for a single pretrained backbone that handles heterogeneous data without bespoke per-task pipelines have an obvious architectural target here. The 559-million-event pretraining scale and the task-agnostic interface are directly aligned with the economics of deploying one model across many clinical use cases, and the ability to ingest unseen drug names and new code values via free-text type specifics reduces the maintenance burden of closed-vocabulary systems.

Future Directions

1. Mitigating exposure bias and rollout drift. Training currently predicts one token ahead under teacher forcing, causing errors to compound over long autoregressive horizons. Scheduled sampling, reinforcement learning from sequence-level objectives, or post-training on rollout data could extend reliable forecasts beyond the 72-hour window where readmission prediction currently falls short.

2. Extending counterfactual simulation toward causal inference. The authors explicitly note that their intervention-conditioned forecasts are not causal modeling. Establishing when these simulated treatment effects can be trusted, and how to calibrate them down from the roughly twofold overestimate seen against SMART, is an open methodological question.

3. Deepening evaluation across the full task suite. The paper demonstrates selected use cases within each downstream task for feasibility. Head-to-head comparisons against specialized discriminative models, more thorough evaluation on underrepresented patient subgroups, and broader modality coverage would strengthen claims of general superiority.

4. Scaling beyond MIMIC. The dataset is U.S.-centric and ICU-heavy. Validating on other health systems, non-English clinical text, and primary-care-dominated populations is necessary before claims about lifelong, population-scale patient modeling generalize. The authors frame NOAH as "ready-to-scale," which invites exactly this replication work.

Target Audience

This paper is most valuable to machine learning researchers working on clinical foundation models, multimodal representation learning, and generative sequence modeling, particularly those interested in how transformer architectures must be adapted for irregular, heterogeneous, time-stamped data. It is also directly relevant to clinical informatics researchers and health system data scientists evaluating pretrained backbones for downstream deployment, and to medical AI practitioners interested in zero-shot prognosis, risk stratification, or treatment simulation. Readers without a background in transformers or variational inference will find the high-level findings accessible, but the methodological detail assumes some familiarity with modern deep learning.

Authors’ abstract

The digitization of healthcare has generated vast, longitudinal, and multimodal patient records over a lifetime, yet fully exploiting these data to represent and predict patient state trajectories remains a critical challenge. Current AI models often struggle to capture the complex, irregular temporal dynamics and inherent stochasticity of real-world multimodal patient data. Existing AI approaches for modeling longitudinal patient records are predominantly discriminative, limited to a few modalities, constrained by closed categorical vocabularies, treating time as a monotonic inductive bias, or they are limited in forecasting future patient states. We introduce NOAH, a time-aware, task-agnostic, generative transformer model representing and forecasting the full multimodal patient journey. NOAH features a novel bidirectional time integration and a variational latent space to capture the continuous evolution of patient states and the stochasticity of clinical trajectories. Built from over 559 million clinical events from 431,000 hospital visits of 299,000 patients across the MIMIC dataset family, NOAH natively processes medical images, time-series and numeric signals, categorical events, as well as structured and unstructured clinical records. NOAH is the first truly holistic generative model in its field, enabling autoregressive forecasting with optional time control, zero-shot classification, and counterfactual intervention simulation. It generates highly informative and predictive patient state representations that demonstrate strong performance in probing for clinical outcomes, 15 ICD chapters, and 29 comorbidities, as well as in time-to-event prediction. Seamlessly handling diverse modalities and complex temporal dynamics, NOAH provides a versatile, task-agnostic, scalable foundation for intelligent predictive systems in personalized clinical care and digital medicine.