Toward Valid Generative Clinical Trial Data with Survival Endpoints

AI summary

Overview

Research area: Generative machine learning for health, specifically synthetic clinical trial data with time-to-event (survival) endpoints.

Technical level: Intermediate to Advanced. Readers need familiarity with variational autoencoders, censoring, and survival analysis concepts such as the log-rank test, hazard ratios, and type I error.

Scope: This paper introduces a variational autoencoder that jointly generates mixed-type covariates and censored survival outcomes, then systematically tests whether the resulting synthetic control arms preserve valid downstream statistical inference rather than only looking realistic.

What This Paper Is About

Clinical trials in oncology and rare diseases struggle with slow enrollment, small patient populations, and high costs, which has motivated interest in synthetic control arms generated by AI. The hard part is generating time-to-event outcomes, which are the primary endpoints in most late-phase trials but are difficult to model under censoring and small samples. The authors ask not only whether synthetic data looks like real data, but whether it leads to statistically valid conclusions when substituted for or added to a real control arm.

Key Contributions

1. A joint VAE for covariates and survival outcomes. The authors extend the HI-VAE (Heterogeneous and Incomplete Variational Autoencoder) of Nazábal et al. (2020) to model censored survival times within a single unified latent variable framework, without assuming independent censoring. Two variants are proposed: HI-VAE_Weibull and HI-VAE_piecewise.

2. A calibration-oriented evaluation framework. Beyond the standard fidelity, utility, and privacy metrics, the authors assess type I error and power in downstream survival analyses, comparing empirical rates against the theoretical power given by Schoenfeld's (1983) formula.

3. Two realistic trial scenarios evaluated. Data sharing under privacy constraints (synthetic controls substitute for original data) and control-arm augmentation (synthetic patients mitigate imbalance between treated and control groups).

4. A post-generation selection procedure. The authors propose selecting the best generated control dataset from among N_gen = 200 candidates, defined as the one yielding the highest log-rank p-value against the original training controls, which improves calibration.

Main Findings

• Superiority on classical metrics: The HI-VAE variants consistently achieved lower divergence from the original distributions and higher fidelity, particularly on survival outcomes, while maintaining competitive privacy scores, compared to Surv-GAN and Surv-VAE.

• Systematic miscalibration across all methods: Type I error rates were consistently inflated and power curves often deviated sharply from theoretical expectations for all methods, including the authors'. Good fidelity scores or usual utility metrics did not guarantee valid downstream inference.

• Post-generation selection improves calibration: In the independent case, the selected datasets closely reproduced the target theoretical power for a control size of N_C = 300 (the blue line), even when trained on as few as N_train = 100 or 200 original controls, indicating a genuine augmentation effect, particularly in the ×3 augmentation scenario. This effect was attenuated or nonexistent in the dependent setting. Type I error rates remained somewhat inflated in augmentation settings, with inflation increasing with the augmentation factor.

• Reliability of the candidate pool: The authors' method achieved at least one generated dataset not rejected by the multiple-testing-adjusted log-rank test in at least 80 percent of Monte Carlo replications across all experimental settings, and often considerably more.

• Baseline behaviour after selection: Surv-GAN generally failed to reproduce expected type I error rates or powers. Surv-VAE attained the target theoretical power in three of six experiments, most notably in the replacement scenario, but fell short in the others and did not adequately control type I error in most cases.

• Candidate-pool size matters: Reducing the candidate pool to 50 or 100 datasets did not alter the conclusions, but selecting the top 20 percent of generated datasets led to poor performance for all methods. Other performance metrics were not degraded by the selection procedure.

• Discrimination versus calibration: When a Cox proportional hazards model was trained separately on synthetic versus real control data and evaluated on an independent real test set using the C-index and integrated Brier score, the authors' method produced models whose performance closely matched real controls in both discrimination and calibration, whereas Surv-GAN and Surv-VAE achieved good discrimination but poorer calibration.

• Hyperparameter search strategy: Strategies that evaluated performance on generated synthetic datasets of full control size (methods 1 and 3) proved more robust than relying solely on validation-set evaluation (method 2). Seed choice still influenced absolute performance within each algorithm.

• Training on controls only versus controls plus treated: On simulated data, resemblance and utility metrics were generally superior when training on controls only, while privacy metrics improved when both arms were used. Surv-VAE was the only model to show a modest benefit from including treated data. On real datasets, all models performed better on average when trained solely on controls.

• Insufficient privacy for open release: Posterior sampling yielded K-map values of about 4–6 and NNDR values around 0.2–0.4, below the EMA Policy 0070 benchmark recommending K-map values above 11. Switching to prior-based sampling produced only minor differences, with no consistent privacy benefit. A preliminary differential privacy attempt using Opacus with noise_multiplier = 2.0 did not substantially improve privacy metrics.

Methodology in Plain English

The authors treat each trial participant as a triple: a vector of covariates, an observed time, and an event indicator telling whether that time was an actual event or a censoring time. They build a generative model that learns the joint distribution of all three together.

The model has three latent pieces per patient. A one-hot vector picks one of several mixture components, a continuous latent vector captures individual variation, and a neural network maps these into a shared representation. The prior over components is a mixture of Gaussians with uniform mixing probabilities, which lets the model capture heterogeneity in clinical data. From this shared representation, separate neural networks output the parameters of each covariate's distribution depending on its type (Gaussian for continuous, log-normal for positive-valued, Poisson for counts, multinomial-logit for categorical).

Crucially, the time-to-event part is modeled by parameterizing the complete density, not just the distribution of event times. One neural network handles the event-time density and another handles the censoring density, so the model does not need to assume that censoring is independent of the covariates or the event process. Two density choices are implemented: a Weibull distribution (HI-VAE_Weibull) and a piecewise-constant model on a discretized time axis (HI-VAE_piecewise).

Training maximizes the evidence lower bound using the Adam optimizer with early stopping, with the Gumbel-Softmax trick for the discrete component and the Gaussian reparameterization trick for the continuous one. Hyperparameters were selected with Optuna by minimizing the distance between Kaplan-Meier curves from observed and synthetic data, using up to 150 trials.

Baselines were SurvivalGAN and SurvivalVAE from the synthcity library (referred to as Surv-GAN and Surv-VAE), which use a modular two-stage pipeline that separates covariate generation from survival modeling, in contrast to the authors' unified framework. Generators were trained on a fraction υ ∈ {1/3, 2/3, 1} of available control patients, then 200 synthetic control arms were generated, each of size equal to the treated group. Evaluation covered simulated data with 100 Monte Carlo replications and four phase III clinical trial datasets.

Why This Matters

Impact on research. The paper argues that evaluating generative models for health only on fidelity and privacy is insufficient. A model that looks strong by standard machine learning metrics may still fail when used for clinical inference, because inflated type I error and biased power would change trial conclusions. This is presented as the first systematic evaluation of type I error and power in generative survival models.

Real-world applications:

• External control arms: using synthetic controls to support single-arm trials or regulatory submissions when a randomized control group is impractical.
• Privacy-preserving data sharing: releasing synthetic versions of control-arm data for external validation or meta-analysis without exposing patient-level records.
• Control-arm augmentation: supplementing small control groups in rare disease, highly targeted therapy, or early-phase studies where N_C is much smaller than N_T.
• Benchmarking and trial simulation: generating realistic trial data for methodological development and planning.

Industry relevance. The authors explicitly frame the work as informing ongoing regulatory initiatives at the European Medicines Agency (EMA) and the Food and Drug Administration (FDA) on the evaluation and potential integration of AI-based and model-based evidence in clinical development. The privacy findings also bear directly on EMA Policy 0070 requirements for public data release.

Future Directions

• Stronger privacy-preserving techniques, such as calibrated epsilon-delta differential privacy, and diffusion- or transformer-based generators with privacy-aware objectives. The authors note that adapting these models to survival settings requires further methodological development.

• Calibration-aware training strategies. Rather than only improving fidelity, generators could be trained with downstream statistical validity as an explicit objective.

• Better calibration of type I error. Post-generation selection restored power in most settings but type I error remained partially elevated, especially as the augmentation factor increased.

• Broader validity frameworks. Bridging generative modeling with domain-specific notions of validity, such as error control in survival analysis, is described as essential for reliability in real-world applications.

Target Audience

This paper benefits machine learning researchers working on tabular and survival data generation, biostatisticians and clinical trial methodologists interested in external controls and synthetic arms, and regulatory scientists evaluating AI-based evidence in clinical development. Practitioners who want to use synthetic control arms in practice will find the calibration results and the privacy shortfall particularly important, while method developers will find the unified latent-variable formulation and the evaluation framework the most directly reusable parts.

Authors’ abstract

Clinical trials face mounting challenges: fragmented patient populations, slow enrollment, and unsustainable costs, particularly for late phase trials in oncology and rare diseases. While external control arms built from real-world data have been explored, a promising alternative is the generation of synthetic control arms using generative AI. A central challenge is the generation of time-to-event outcomes, which constitute primary endpoints in oncology and rare disease trials, but are difficult to model under censoring and small sample sizes. Existing generative approaches, largely GAN-based, are data-hungry, unstable, and rely on strong assumptions such as independent censoring. We introduce a variational autoencoder (VAE) that jointly generates mixed-type covariates and survival outcomes within a unified latent variable framework, without assuming independent censoring. Across synthetic and real trial datasets, we evaluate our model in two realistic scenarios: (i) data sharing under privacy constraints, where synthetic controls substitute for original data, and (ii) control-arm augmentation, where synthetic patients mitigate imbalances between treated and control groups. Our method outperforms GAN baselines on fidelity, utility, and privacy metrics, while revealing systematic miscalibration of type I error and power. We propose a post-generation selection procedure that improves calibration, highlighting both progress and open challenges for generative survival modeling.