Composable Causality: A Toolkit for Systematic Time-Series Causal Discovery and Treatment-Effect Benchmarking

AI summary

Overview

Research area: Causal machine learning — specifically time-series causal discovery and treatment-effect (counterfactual) estimation, with a focus on benchmarking infrastructure.

Technical level: Intermediate. The tooling and benchmark results are accessible to anyone with basic causal-inference background; the identifiability propositions in Section 3.2 are Advanced.

Scope (one sentence): The paper introduces DTCD, a configurable data-generation library that builds multivariate time series with known contemporaneous and lagged causal graphs and exactly paired counterfactual twins, plus a reference benchmark of eight discovery algorithms and four treatment-effect estimators.

What This Paper Is About

Research on finding causal structure in multivariate time series, and on estimating treatment effects from it, relies on synthetic data because real data almost never comes with a verified ground-truth causal graph. The tools available for making that synthetic data come in two flavours: fixed corpora that cannot be changed after release, and collections of generator scripts where each script hardcodes one combination of mechanism, confounding, and missingness. Neither lets a researcher vary one property — sparsity, noise distribution, or missingness pattern — while holding everything else fixed, which is exactly what controlled method comparisons require. The paper builds a toolkit where every piece of the data-generating process is an independent, swappable configuration option, and uses it to run a systematic benchmark that surfaces failure modes averaged metrics conceal.

Key Contributions

1. A composable framework rather than a corpus or script collection. Topology generator, structural mechanism, noise distribution, observational mask, and intervention policy are each first-class API parameters that compose orthogonally. Sensitivity sweeps and factorial ablations that would require new code in script-based benchmarks require only configuration changes here.

2. Exact counterfactual twin generation. The CounterfactualSuite caches the stochastic realization N during the factual rollout and re-injects it during the interventional rollout, so the individual treatment effect at any time t for any downstream node is exact to floating-point precision. The authors state that no prior temporal benchmark provides ground-truth counterfactuals.

3. Explicit identifiability documentation. Identifiability status is stated as propositions for the linear mechanisms (linear Gaussian VAR: lagged adjacency supports identified, contemporaneous matrix only up to Markov equivalence; VAR-LiNGAM: both uniquely identified under non-Gaussian innovations), and the recursive nonlinear power-transform mechanism is explicitly marked as not provably identifiable and used only as a robustness stressor. No claim is made for the remaining mechanisms.

4. A reference benchmark and diagnostic protocol. Eight discovery algorithms across nine graphs (d ∈ {4, 6, 8}, τ_max ∈ {2, 3, 4}) and nine DGP variants, plus four CATE estimators on two variants, tuned with Optuna and evaluated with standardized metrics (SHD, normalized SHD, F1, TPR, FDR, AUPRC, PEHE), with recovered graphs visualized next to ground truth.

5. Open-source release with reproducibility infrastructure. Native exporters to tigramite and causal-learn, Datasheet for Datasets documentation, Croissant metadata, and single-command reproduction of every paper table and figure.

Main Findings

• DYNOTEARS dominates clean conditions, but cannot run on missing data. It reaches F1 ≈ 0.99 on the Linear and Nonlinear variants and 0.98 on Drift, with SHD 0.02, 0.02 and 0.04 respectively. Its nonlinear performance is attributed to the recursive sign-power transform being strictly monotonic and amplitude-bounded, not to general nonlinear robustness.

• Missing values defeat most methods regardless of mechanism. Six of eight discovery methods cannot accept NaN inputs at all; the runner records a structured skipped_no_missing_support status rather than silently emitting an empty graph. The two that opt in complete every cell: CUTS+ (F1 = 0.75 on MCAR, 0.62 on MNAR) and VARLiNGAM_imp (F1 = 0.61 on MCAR, 0.50 on MNAR). The 0.14-point MCAR–MNAR gap quantifies the cost of informative missingness for CUTS+.

• A marginal-variance shortcut inflates linear-variant scores. Contemporaneous varsortability on the raw benchmark averages 0.93 (0.5 is uninformative, 1.0 perfectly sortable). A SortnRegress control reaches F1 = 0.93 against DYNOTEARS at 0.99 on raw data. Standardizing drives varsortability to exactly 0.5, DYNOTEARS falls to 0.82, SortnRegress converges to RandomRegress (0.81 versus 0.79), and DYNOTEARS' margin over the best competitor shrinks from 0.23 to 0.05. The effect is material (>0.05) on six of the nine variants, largest on seasonal (0.79 to 0.46), absent on the trend variants, and exactly zero for every scale-invariant method (PCMCI-ParCorr, VARLiNGAM, GrangerF).

• Linear conditional-independence methods are insensitive to monotonic and additive perturbations. At evaluation seed 42, PCMCI-ParCorr and GrangerF return the same SHD and predicted-edge count on the Linear and Nonlinear versions of a graph in 15 of 18 (graph, method) pairs.

• TARNet wins the treatment-effect comparison on both variants. PEHE = 0.491 (Linear) and 0.427 (Nonlinear) at about 1.6 seconds per fit. Ranking is ordinally stable across variants: TARNet < SNet ≈ FlexTENet < X-Learner. FlexTENet takes about 6.5 seconds and trails TARNet by 0.09 to 0.17 PEHE; SNet's roughly 12 seconds buys no PEHE advantage over FlexTENet.

• All four CATE estimators improve from Linear to Nonlinear (PEHE drops 13–25% across all four), which the authors attribute to the saturating sign-power transform compressing the range of treatment-effect modifiers.

• Dimensionality is not uniformly detrimental for CATE. On the Linear DGP, X-Learner's PEHE falls with d (0.84, 0.82, 0.61 for d = 4, 6, 8), TARNet's does not rise monotonically (0.44, 0.59, 0.44), while SNet's and FlexTENet's rise.

• Some low scores reflect scope mismatches, not method failure. CD-NOD's near-zero recall (TPR 0.02 to 0.03) follows from its design as a contemporaneous-structure method that does not detect lagged edges. GrangerF serves as the adversarial floor (F1 ≈ 0.43, FDR ≈ 0.67).

• Run accounting. Of 6,480 attempted (method, graph, variant, seed) combinations, 4,809 succeeded, 960 were skipped for declaring supports_missing=False, 540 were skipped by PCMCI-GPDC's compute-budget guard at d > 4, and none raised an exception. An earlier run had 171 exceptions (2.6%), all in VARLiNGAM_imp, traced to a wrapper defect and fixed in the reported regeneration.

Methodology in Plain English

The authors reframe the benchmarking problem as one of composition rather than curation. Rather than shipping datasets, they ship a generator whose behaviour is controlled by five independent dials: how the causal graph is wired, what functional form each variable's equation takes, what noise distribution drives it, how data goes missing, and whether an intervention is applied.

Because the dials are independent, changing one and holding the rest fixed is a configuration edit rather than a coding task. The generator runs a structural rollout that outputs four things at once: the time series, the true adjacency matrices (contemporaneous plus each lag), the observation mask, and — if an intervention policy is supplied — a paired counterfactual trajectory.

The counterfactual trick is the technically distinctive part. Normally you can never observe both what happened and what would have happened to the same unit under the same random draws. DTCD sidesteps this by storing every random noise value it draws during the factual simulation and replaying those exact values during the interventional simulation, changing only the intervened variable. Any difference between the two runs is therefore pure causal effect, exact to floating-point precision, which is what makes a metric like PEHE computable against ground truth.

They then run a factorial benchmark: nine hand-curated graphs crossed with nine data-generating variants (clean linear, nonlinear, drifting coefficients, MNAR, MCAR at 20% per entry, trend, seasonal, trend+seasonal, and a combined nonlinear+trend+seasonal), each with ten evaluation seeds. Structural weights are sampled once per graph and held fixed across all variants, seeds, and methods, so reported standard deviations reflect only noise-realization variance. Hyperparameters are tuned with Optuna at 30 trials per cell on a held-out seed 41 trajectory — with a reduced budget of 5 trials for PCMCI-GPDC and DYNOTEARS — and the authors openly flag that this tuning objective is an oracle using the ground-truth graph, making reported numbers optimistic.

Finally, they audit their own generator for a known artifact rather than assuming it away. Because additive-noise generators with bounded coefficients accumulate variance along causal paths, simply sorting variables by marginal variance can recover much of the graph without causal reasoning. They measure varsortability directly, add SortnRegress and RandomRegress controls, and ship standardization as a first-class option.

Why This Matters

Impact on research. The paper argues that the fixed-artifact paradigm is the wrong abstraction for methodological work. Questions like "at what sparsity does constraint-based discovery break?" or "how does ranking change as nonlinearity intensifies?" require factorial experiments, and the paper delivers infrastructure for them plus a working demonstration that aggregate metrics hide method-specific artifacts. The varsortability audit is the clearest case: a shortcut worth 0.23 F1 in margin was invisible in headline numbers and only surfaced when measured deliberately.

Real-world applications (domains named in the paper as motivating time-series causal discovery):

• Healthcare — recovering physiological or clinical causal structure from longitudinal patient records, and estimating effects of time-varying treatments.
• Algorithmic finance — discovering lead-lag dependencies among correlated market series.
• Climate science — disentangling causal links among interacting geophysical time series.
• Method development for non-stationary settings — the drift, trend, and seasonal variants probe robustness to artifacts common in deployed data but absent from canonical synthetic benchmarks.

Industry relevance. Practitioners choosing a causal discovery method have little guidance on how performance degrades under missingness or non-stationarity, and the results here are blunt: six of eight standard methods cannot accept missing values at all, which is an input-validation barrier rather than an identifiability problem. For teams building treatment-effect systems, the finding that architectural complexity does not buy accuracy (SNet at roughly twelve seconds producing worse PEHE than TARNet at roughly 1.6 seconds) is directly actionable.

Future Directions

• Adding machine-readable identifiability tags to the core generators. Currently only the spatio-temporal module emits them, and construction-time warnings are planned; configurations with unestablished identifiability are treated as unknown rather than flagged.

• Wrapping the deferred CATE baselines. CRN, G-Net, and the Causal Transformer are excluded from the reference CATE benchmark because each needs a wrapper exposing a unified fit(X, A, mask) -> tau_hat interface.

• Weight-sensitivity analysis and confounded assignment. The current protocol holds weights fixed per graph and varies only noise; varying weights at fixed noise is supported but deferred, as are confounded-assignment variants beyond the current Bernoulli(0.5) policy.

• Extending scope beyond d ∈ {4, 6, 8}. Larger graphs and the spatio-temporal module are supported but outside the reference benchmark, and PCMCI-GPDC is restricted to d = 4 on compute grounds — leaving open how rankings shift at higher dimensionality.

Target Audience

Researchers and engineers working on time-series causal discovery or treatment-effect estimation who need controlled, reproducible comparisons rather than one-off datasets. It is most valuable to (a) method developers who want to ablate difficulty axes without writing generator code, (b) benchmarking and evaluation researchers interested in the varsortability diagnostic and the exact-counterfactual protocol, and (c) practitioners choosing among the eight discovery methods and four CATE estimators covered, who will benefit from the concrete failure-mode reporting. Readers primarily interested in new causal discovery theory will find the identifiability treatment deliberately conservative — propositions for the linear mechanisms, and explicit non-claims elsewhere.

Authors’ abstract

Real-world multivariate time series rarely come with verified ground-truth causal structure, so research on time-series causal discovery and treatment-effect estimation depends on synthetic data. Existing tools are either fixed datasets that cannot be modified after release or script-based generators in which each script fixes one combination of mechanism, confounding, and missingness. Neither supports controlled experiments that vary a single property, such as sparsity,…