The Robustness of Differentiable Causal Discovery in Misspecified Scenarios

AI summary

Overview

Research area: Causal discovery — specifically the empirical robustness of differentiable (gradient-based) causal discovery algorithms when their underlying modeling assumptions are violated.

Technical level: Advanced. The paper assumes familiarity with structural causal models, constraint-based/score-based/functional causal model taxonomy, DAG constraints such as NOTEARS' Tr(e^{W∘W}) − d, and metrics like SHD and SID.

Scope in one sentence: Twelve mainstream causal discovery algorithms are benchmarked across eight model-assumption-violation scenarios on synthetic linear and nonlinear data, with theoretical analysis of why linear differentiable methods hold up in several of those scenarios but not under scale variation.

What This Paper Is About

Causal discovery algorithms are built on assumptions — linear or nonlinear mechanisms, i.i.d. data, no latent confounders, no measurement error, faithfulness — that real-world data routinely break, and these assumptions cannot be verified in practice. The authors ask whether differentiable causal discovery methods, which convert the combinatorial DAG search into a smooth constrained optimization, remain reliable when those assumptions fail. They answer with a large empirical benchmark covering eight misspecified scenarios and twelve algorithms, supplemented by theory explaining the observed behavior.

Key Contributions

1. A large-scale empirical evaluation of twelve prominent causal discovery algorithms across eight model-assumption-violation scenarios, comprising over 70,000 experiments on more than 2,400 synthetic datasets.
2. Expansion of the benchmark landscape for misspecification to include under-explored challenging scenarios — heterogeneity, scale variation, missing data, and mechanism violation — in addition to confounded, measurement error, unfaithful, and autoregressive settings.
3. The first assessment of gradient-based (differentiable) causal discovery methods across a wide array of misspecified scenarios, filling a gap left by prior benchmarks that either omitted differentiable methods or omitted broad misspecification.
4. Theoretical analysis of linear differentiable causal discovery under measurement error, unfaithful distributions, and missing data, using Theorems 7 and 9 of Loh & Bühlmann (2014) and a noise-ratio quantity r.

Main Findings

• Differentiable methods are broadly robust, with one clear exception. The authors report that differentiable causal discovery algorithms "almost always achieve optimal or competitive performance in commonly used misspecified scenarios other than scale variation," judged by SHD and SID of the inferred graphs.

• Scale variation breaks linear differentiable methods. Under scale-variant data, NOTEARS, GOLEM, NoCurl, and DAGMA degrade sharply and perform worse than PC and GES. On the 10-node ER-2 linear benchmark, NOTEARS reaches SHD 18.0 ± 1.2 / SID 60.5 ± 7.3 and GOLEM 17.5 ± 1.2 / SID 64.4 ± 6.8, against PC at SHD 12.4 ± 3.1 / SID 40.9 ± 13.4 and GES at SHD 13.8 ± 7.8 / SID 32.0 ± 13.6. This matches the earlier observations of Reisach et al. (2021).

• Nonlinear differentiable methods also degrade under scale variation, but remain above PC and GES. The paper states this scenario had not previously been explored for nonlinear methods, and that unlike the linear case, nonlinear differentiable results are almost always superior to PC and GES — though almost always lower than CAM.

• Strong vanilla performance sets the baseline. On the 10-node ER-2 linear vanilla model, NOTEARS scores SHD 1.5 ± 1.6 / SID 1.8 ± 4.2, GOLEM 1.4 ± 1.4 / SID 0.4 ± 1.2, and DAGMA 1.2 ± 1.2 / SID 3.3 ± 5.3.

• Confounded, measurement error, autoregressive, and heterogeneous data are handled well. Under these scenarios (heterogeneous evaluated at P₁ = 0.5, γ = 0.1), differentiable methods achieve optimal or competitive performance relative to other algorithm families. For the nonlinear Gaussian process mechanism, CAM performs better, but the authors argue CAM has limitations compared to differentiable methods.

• Missing data barely changes results. With MCAR missingness, algorithm results are close to the vanilla model; differentiable methods are consistent with traditional methods including PC and GES. In the linear missing setting, DAGMA reaches SHD 1.5 ± 1.4 / SID 4.5 ± 7.1 and GOLEM SHD 1.7 ± 1.7 / SID 6.2 ± 10.8.

• Mechanism violation favors the matching method class. In the linear setting, linear differentiable methods remain competitive despite PC and GES being able to handle nonlinear mechanisms. In the nonlinear setting, NOTEARS-MLP, GraN-DAG, and DAGMA outperform other algorithm types; on the nonlinear mechanism-violation column, DAGMA reaches SHD 3.3 ± 3.1 / SID 9.5 ± 11.6 and NOTEARS-MLP SHD 5.9 ± 2.5 / SID 19.7 ± 8.7, versus CAM at SHD 17.8 ± 4.4 / SID 45.9 ± 17.0.

• CAM's advantage is tied to the data-generating mechanism. CAM excels in the nonlinear vanilla setting (SHD 4.7 ± 1.9 / SID 16.3 ± 9.5) because the Gaussian process mechanism aligns with its assumptions, but it does not perform well under mechanism violation. In an MLP-based comparison, NOTEARS-MLP achieves better performance under almost all model assumption violations.

• Unfaithful distributions hurt variance-sorting methods. Var-SortnRegress and R²-SortnRegress show a significant performance drop under unfaithful distributions: when the direct effect of X_i → X_k cancels in a triplet X_i → X_j → X_k ← X_i, the variance of X_k changes substantially, reducing Var-Sortability. In the nonlinear unfaithful setting, SHD generally declines somewhat across algorithms, consistent with Montagna et al. (2023), who found path cancellation makes structural inference of sparse graphs easier.

• Theory links measurement error to a worse noise ratio. Theorem 7 of Loh & Bühlmann (2014) says minimizing the least squares score returns the true DAG in the large-sample limit for a linear model with equal noise variance; Theorem 9 gives a sufficient condition r < 1 + ξ/d, where ξ is the score gap to the next-best DAG. Under measurement error the noise ratio becomes r̃ = max(1 + δ·Var(X_i), …) / min(1 + δ·Var(X_i), …), which the authors argue exceeds r = 1, so the guarantee no longer holds — explaining the empirical decline.

Methodology in Plain English

The authors build synthetic data from two baseline ("vanilla") models. The linear vanilla model samples coefficients from U(−2, −0.5) ∪ U(0.5, 2) with additive standard Gaussian noise; the nonlinear vanilla model uses a Gaussian process with a radial basis function kernel of bandwidth one. They then deliberately break assumptions in eight ways: latent confounders, measurement error, unfaithful distributions, and autoregressive effects (following Montagna et al., 2023), plus their own four additions — heterogeneous multi-domain noise, scale variation, missing data under MCAR, and mechanism violation (feeding linear data to nonlinear methods and vice versa).

Data are simulated as ER and SF graphs with node counts d ∈ {10, 20, 50} and average degree k ∈ {2, 4}, plus GRP graphs with average degree 6. For each configuration and scenario, 10 datasets of 2000 samples are generated, and mean and standard deviation of the metrics over 10 trials are reported. Twelve algorithms spanning constraint-based (PC), score-based (GES), functional causal model-based (DirectLiNGAM, CAM, Var-SortnRegress, R²-SortnRegress), and gradient-based (NOTEARS, GOLEM, NOTEARS-MLP, GraN-DAG, NoCurl, DAGMA) families are compared. Hyperparameters are tuned per dataset: the sparsity coefficient λ₁ over {0.005, 0.01, 0.05, 0.5, 2, 5} and significance level α over {0.001, 0.01, 0.05, 0.1}. Performance is measured by Structural Hamming Distance (edge insertions, deletions, reversals needed to reach the true graph) and Structural Intervention Distance (differences in intervention distributions). Where an algorithm outputs a Markov equivalence class, undirected edges are assumed correctly oriented, following Zheng et al. (2018).

Why This Matters

Research impact. Prior benchmarks either covered only traditional methods under misspecification (Montagna et al., 2023), only differentiable methods without misspecification (Ng et al., 2024), or were limited to linear SEMs, bivariate nonlinear settings, or Markov equivalence class outputs. This study is presented as the first broad misspecification benchmark that includes recent differentiable causal discovery, and it adds theoretical grounding rather than reporting rankings alone.

Real-world applications where assumption violations are unavoidable:

• Biomedical data analysis, including the real-world Sachs protein-signaling dataset examined in the paper's appendix.
• Observational studies in the social and behavioral sciences, where randomized experiments are blocked by cost, risk, or ethics.
• Sensor and monitoring data subject to measurement error and missing records (the MCAR regime studied here).
• Multi-domain or multi-site datasets exhibiting distribution shift, corresponding to the heterogeneous scenario with varying noise variances across domains.

Industry relevance. Practitioners deploying causal discovery for decision support need to know when a method will silently fail. The paper's message is reassuring for mechanism mismatch, missingness, confounding, and heterogeneity, but cautionary for scale-variant feature sets — a warning that is easy to act on, since it points to preprocessing or method selection as mitigation. The finding that CAM's apparent superiority depends on the data-generating mechanism also encourages evaluation on mechanisms other than Gaussian processes before adopting a method.

Future Directions

1. Rehabilitating differentiable methods under scale variation. Since scale variation is the consistent failure mode, the open question is whether standardization, alternative scoring, or DAG-constraint variants can restore robustness without losing performance elsewhere.
2. Beyond the Gaussian process testing mechanism. The paper shows CAM benefits from a Gaussian process that matches its assumptions; a broader evaluation under neural-network functional mechanisms (as begun with the MLP setting) would clarify how conclusions transfer to unknown real-world mechanisms.
3. Extending theory beyond the linear case. The reported theoretical analysis covers linear differentiable causal discovery via the noise-ratio condition from Loh & Bühlmann (2014); the authors note unexplored performance of nonlinear methods under scale variation, indicating a corresponding theoretical gap.
4. Combined and other misspecification regimes. The paper points to appendices covering combined misspecified scenarios, non-Gaussian noise vanilla models, semi-synthetic data, runtime of benchmark methods, and the Sachs dataset — each representing a direction where robustness conclusions could be tested further.

Target Audience

Researchers and graduate students working on causal discovery, structural equation modeling, or structure learning who need to know which method families survive assumption violations; benchmark designers seeking evaluation protocols for misspecified settings; and practitioners in biomedicine, epidemiology, economics, or industrial monitoring who must run causal discovery on observational data whose assumptions cannot be verified. Readers without background in DAG constraints, SHD/SID, and causal identifiability will find the experimental sections accessible but the background and theory sections demanding.

Authors’ abstract

Causal discovery aims to learn causal relationships between variables from targeted data, making it a fundamental task in machine learning. However, causal discovery algorithms often rely on unverifiable causal assumptions, which are usually difficult to satisfy in real-world data, thereby limiting the broad application of causal discovery in practical scenarios. Inspired by these considerations, this work extensively benchmarks the empirical performance of various mainstream causal discovery algorithms, which assume i.i.d. data, under eight model assumption violations. Our experimental results show that differentiable causal discovery methods exhibit robustness under the metrics of Structural Hamming Distance and Structural Intervention Distance of the inferred graphs in commonly used challenging scenarios, except for scale variation. We also provide the theoretical explanations for the performance of differentiable causal discovery methods. Finally, our work aims to comprehensively benchmark the performance of recent differentiable causal discovery methods under model assumption violations, and provide the standard for reasonable evaluation of causal discovery, as well as to further promote its application in real-world scenarios.