SurvHTE-Bench: A Benchmark for Heterogeneous Treatment Effect Estimation in Survival Analysis

AI summary

Overview

Research area: Causal machine learning for survival analysis, specifically heterogeneous treatment effect (HTE) estimation from right-censored time-to-event data.

Technical level: Advanced. The paper assumes familiarity with causal identification assumptions (ignorability, positivity, consistency), censoring mechanisms, and survival modeling concepts such as proportional hazards and restricted mean survival time (RMST).

One-sentence scope: The authors introduce the first standardized benchmark for evaluating how well machine learning methods estimate individualized treatment effects when outcomes are censored over time, spanning 40 synthetic, 10 semi-synthetic, and 2 real-world datasets.

What This Paper Is About

When clinicians want to know whether a treatment works differently for different patients, they need methods that estimate heterogeneous treatment effects. When the outcome is time-to-event data (e.g., death, disease progression) that is often censored because some patients drop out or the study ends early, estimation gets much harder. The problem is that published methods each rely on their own bespoke simulations and evaluation protocols, making it impossible to compare them fairly or know which one is robust when its underlying assumptions break down. This paper builds a common playing field—SurvHTE-Bench—so it becomes possible to compare these methods systematically under consistent conditions.

Key Contributions

1. Method unification. The authors organize existing survival HTE methods into three families—outcome imputation methods, direct-survival causal methods, and survival meta-learners—and implement 53 method variants in a modular, reusable codebase. This is the first framework that gathers these approaches under one roof.

2. Synthetic benchmark with known ground truth. They design 40 synthetic datasets that cross 8 causal configurations (varying randomization, unobserved confounding, positivity, and informative censoring) with 5 survival scenarios (varying event-time distributions and censoring rates from under 30% to over 70%). Because both potential outcomes are generated, true individual-level treatment effects are known.

3. Semi-synthetic and real data. They add 10 semi-synthetic datasets combining real covariates (from the ACTG HIV trial and MIMIC-IV ICU records) with simulated treatments and outcomes, plus two real datasets: the Twins dataset (with known ground truth because both twins are observed) and the ACTG 175 HIV clinical trial (without ground truth).

4. Comprehensive empirical comparison. They run the first rigorous head-to-head evaluation of survival HTE methods under controlled assumption violations, identifying when each family succeeds and where it fails.

Main Findings

• No method wins everywhere. Performance is strongly dependent on the causal configuration, censoring level, and survival distribution. Rankings shift substantially across scenarios, meaning practitioners cannot assume a single dominant estimator.

• Low-censoring, randomized settings favor outcome imputation. In balanced randomized trials with little censoring, methods like X-Learner and Double-ML perform best. Double-ML also leads in Scenario C-type settings for the ACTG semi-synthetic data (RMSE ≈ 10.65).

• Survival meta-learners dominate as censoring rises. S-Learner-Survival and Matching-Survival climb to the top as censoring increases. By Scenario D (high censoring), S-Learner-Survival and Matching-Survival dramatically outperform all other approaches.

• Robustness under simultaneous assumption violations. S-Learner-Survival (average rank 5.17 of 53 variants; 3.30 of 11 families) and Matching-Survival (5.42; 3.48) were the most consistent top performers overall, particularly resilient under unobserved confounding, positivity violations, and informative censoring together.

• DeepSurv is a strong base learner. Survival meta-learners built on DeepSurv consistently outperformed those built on Random Survival Forests or DeepHit, driven by higher time-dependent concordance indices.

• Margin imputation is the most reliable preprocessing choice. Among outcome imputation approaches, Margin imputation achieved the lowest imputation error and degraded least under heavy censoring, translating into more frequent top rankings.

• Positivity violations hurt tree-based causal methods. Causal Survival Forests suffered a large ranking drop when treatment assignment was deterministic in regions of covariate space, while Double-ML and X-Learner remained competitive.

• High censoring with AFT distributions breaks nearly everything. In Scenario D, almost all estimators failed across all causal configurations, highlighting remaining fundamental challenges under extreme censoring.

• Real data reveals behavioral differences without ground truth. On the Twins dataset, S- and DR-Learners with imputation and S-Learner-Survival achieved the lowest CATE RMSE (approximately 7.2 days), while Double-ML was surprisingly worst. On the HIV trial under artificially increased censoring, Causal Survival Forests produced tightly clustered estimates, imputation methods concentrated their predictions, and survival meta-learners showed substantial drift.

Methodology in Plain English

The researchers built a software framework that wraps many different survival treatment-effect estimators behind a common interface, so every method can be trained and evaluated identically. For the synthetic portion, they wrote data generators that simulate patients with five covariates, randomly or non-randomly assigned treatments, and event times drawn from different survival distributions. Crucially, the generators produce both the treated and untreated outcome for every simulated patient, so the true treatment effect is known exactly. They then systematically toggle causal assumptions on and off—introducing unmeasured confounding, removing overlap between treated and untreated groups, and making censoring depend on event times. For the semi-synthetic datasets, they keep real patient covariates (from HIV trials and ICU records) but simulate the treatment assignment and outcomes themselves. Finally, they evaluate on two real datasets, using the Twins study where one twin per pair receives the "treatment" (being heavier at birth) so both outcomes are observable. Evaluation uses CATE RMSE, ATE bias, and auxiliary measures like imputation error and survival model concordance, aggregated across 10 random train/validation/test splits and summarized with Borda count rankings and Wilcoxon signed-rank tests.

Why This Matters

Impact on research. Before this benchmark, survival HTE papers each used their own simulations, so reported improvements could not be trusted as comparable. SurvHTE-Bench makes fair comparison possible, exposes where methods actually break down, and gives the field a shared, extensible testbed. It also explicitly tests assumption violations rather than idealized conditions, which is closer to how real clinical data behaves.

Real-world applications.

• Precision medicine: Choosing which cancer therapy or antiviral regimen is likely to extend survival most for an individual patient, not just on average.
• Clinical trial design: Stress-testing candidate analysis methods before locking in a statistical analysis plan, reducing the risk of misleading subgroup conclusions.
• Electronic health record analysis: Estimating treatment effects from observational ICU or hospital data where censoring is severe and patients are not randomized.
• Health policy: Evaluating whether a public health intervention (e.g., screening program) produces different benefits for different populations, accounting for loss to follow-up.

Industry relevance. Pharmaceutical companies, health insurers, and hospital systems all make decisions that depend on individualized survival predictions. The finding that survival-aware methods are more robust under high censoring directly informs which tools analytics teams should deploy on real-world EHR data, where censoring rates commonly exceed 70 percent.

Future Directions

1. Graded assumption violations. The current benchmark treats violations as binary (present or absent). Extending it to a continuum—for example, varying the strength of unmeasured confounding using Rosenbaum's Γ, or the degree of overlap violation—would reveal the thresholds at which each estimator breaks down.

2. Broader estimands. The paper focuses primarily on RMST. Extending native support to conditional median survival time, time-varying survival probabilities, and other task-specific targets would make the benchmark more clinically flexible.

3. More complex causal structures. Time-varying treatments, longitudinal covariates, instrumental variables, and interference are all out of scope here. These are common in practice and would substantially expand the benchmark's usefulness.

4. Integration of newer method classes. Generative causal margin modeling and synthetic-control-based survival methods are acknowledged but not yet benchmarked. Adding these would keep the benchmark current as the field evolves.

Target Audience

This paper is most valuable for causal inference methodologists and survival analysis researchers who develop or evaluate treatment-effect estimators. Biostatisticians working on clinical trials and observational comparative effectiveness studies will find the empirical rankings directly actionable. Machine learning practitioners building personalized medicine or outcomes-prediction systems in healthcare will also benefit, especially from the practical guidance on when to prefer survival meta-learners over imputation-based approaches. Beginners may find the paper dense due to its reliance on causal identification assumptions and survival modeling jargon, but the framing of method families and failure modes is accessible to anyone with a grounding in statistics or machine learning.

Authors’ abstract

Estimating heterogeneous treatment effects (HTEs) from right-censored survival data is critical in high-stakes applications such as precision medicine and individualized policy-making. Yet, the survival analysis setting poses unique challenges for HTE estimation due to censoring, unobserved counterfactuals, and complex identification assumptions. Despite recent advances, from Causal Survival Forests to survival meta-learners and outcome imputation approaches, evaluation practices remain fragmented and inconsistent. We introduce SurvHTE-Bench, the first comprehensive benchmark for HTE estimation with censored outcomes. The benchmark spans (i) a modular suite of synthetic datasets with known ground truth, systematically varying causal assumptions and survival dynamics, (ii) semi-synthetic datasets that pair real-world covariates with simulated treatments and outcomes, and (iii) real-world datasets from a twin study (with known ground truth) and from an HIV clinical trial. Across synthetic, semi-synthetic, and real-world settings, we provide the first rigorous comparison of survival HTE methods under diverse conditions and realistic assumption violations. SurvHTE-Bench establishes a foundation for fair, reproducible, and extensible evaluation of causal survival methods. The data and code of our benchmark are available at: https://github.com/Shahriarnz14/SurvHTE-Bench .