Causal discovery needs explicit epistemic standards

AI summary

Overview

Research area: Causal discovery / structure learning from observational data (stat.ML), specifically the methodology and philosophy of how causal discovery methods are compared.

Technical level: Intermediate. The arguments are conceptual, but they rely on familiarity with DAGs, Markov equivalence classes, faithfulness, and asymptotic consistency.

Scope: A position paper arguing that causal discovery lacks explicit epistemic standards for choosing between algorithms, comparing identifiability assumptions, interpreting consistency guarantees, and designing benchmarks — illustrated with theory, counterexamples, and targeted simulation experiments.

What This Paper Is About

Causal discovery has produced a large and growing collection of algorithms for learning causal graphical structure from data, but few guiding principles for choosing between them. The authors argue that the field's three standard comparison lenses — identifiability, consistency, and benchmarks — each smuggle in untestable assumptions about the data-generating process, and that there are no shared standards for comparing, prioritizing, or even discussing those assumptions. The goal is to argue for making epistemic standards explicit and for designing benchmarks that stress-test and ablate assumptions rather than silently fixing which regimes matter.

Key Contributions

1. A diagnosis of implicit standards in identifiability research. The authors show, via two explicit counterexample SCMs, that faithfulness and covariance-faithfulness are logically independent — neither implies the other — yet faithfulness is treated as the baseline while covariance-faithfulness receives comparatively little attention, a preference they attribute to convention rather than principle.

2. A critique of pointwise consistency as a design target. They argue pointwise consistency is easy to obtain and has kept algorithm design conservative, and they show that a candidate-generating procedure can be made consistent simply by running a consistent baseline such as GES in parallel and returning whichever output attains the better score.

3. An account of benchmarks as performing epistemic work. Through experiments with FLOP, DirectLiNGAM, and VarSortRegress, they show that benchmark design (graph family, edge density, noise distribution, edge weights, metric) quietly determines which guarantees come into play, which algorithms win, and what counts as success.

4. A concrete proposal for practice. Make epistemic standards for identifiability and consistency assumptions explicit, including their failure modes and scope conditions, and design benchmarks that operationalize those standards by stress-testing and ablating both.

Main Findings

• Faithfulness and covariance-faithfulness form no hierarchy. Example 1 gives an SCM with N₁ ~ Unif({0,1,2}) and N₂, N₃ ~ Unif({−1,1}), uniform over 12 configurations on a fully connected triangle graph, where all entries of both the covariance matrix Σ and its inverse Σ⁻¹ are nonzero (so the distribution is covariance-faithful), yet X₂ ⟂ X₃ | X₁ holds (so it is not faithful). Example 2 gives X₁ = N₁, X₂ = X₁ + N₂, X₃ = 2X₁ − X₂ + N₃ with jointly independent, zero-mean, unit-variance, non-Gaussian noise of finite fourth moment, where Cov(X₂, X₃) = 0 (so covariance-faithfulness fails) yet no conditional independence holds, so the distribution is faithful by the Darmois–Skitovich theorem.

• The classical zero-measure justification applies to both equally. The traditional zero-measure argument was formulated for linear Gaussian SCMs, where faithfulness and covariance-faithfulness coincide, so it does not privilege one over the other.

• Covariance-faithfulness may cut across standard divides. The authors note it can hold under nonlinear functional forms or non-additive errors (their footnote example: X₁ = N₁, X₂ = X₁³ + N₂, X₃ = X₂(1 + N₃) with independent standard normal noise), which means the linear vs. nonlinear, Gaussian vs. non-Gaussian, and additive vs. non-additive distinctions stop organizing the space in the same way. FLOP, introduced for linear additive Gaussian models under faithfulness, could have been formulated under covariance-faithfulness alone.

• Assumptions that current practice would reject can hold in standard benchmarks. MEC size is often very small on Erdős–Rényi graphs, and varsortability / R²-sortability are high in commonly used benchmarks; a known temporal order is realistic in longitudinal or time-series settings.

• Pointwise consistency is a weak, easily satisfied design criterion. The BIC score has long been known to be pointwise consistent, which makes exact BIC-optimal search pointwise consistent; exact search is feasible only up to about 30 nodes, motivating GES, whose locally consistent score yields MEC recovery. XGES keeps GES's greedy insertion and deletion phases so it can reuse the GES proof, which the authors argue unnecessarily constrains the algorithm. BOSS and FLOP already decouple algorithm design from consistency proofs, and continuous approaches such as DAGMA make direct consistency proofs technically challenging.

• Uniform consistency requires additional untestable assumptions. Robins et al. (2003) show uniform consistency is unachievable under faithfulness alone; λ-strong faithfulness (Zhang and Spirtes, 2003) restores it, but Uhler et al. (2013) show that for linear Gaussian models the set of non-λ-strong-faithful distributions has strictly positive Lebesgue measure that grows rapidly with the number of nodes and graph density, so it cannot be defended by a zero-measure argument.

• Benchmark success can be incidental to benchmark design. With p = 50 nodes, n = 1000 samples, generalized normal noise with β = 1.4, and 50 sampled graphs per setting, FLOP outperforms DirectLiNGAM on Erdős–Rényi graphs with average degree four for DAG recovery (Figure 1(b)) even though the noise is non-Gaussian and DirectLiNGAM's assumptions are satisfied — because the MEC is small and few edges are left to a random draw. On chains without colliders (MEC size 50), FLOP recovers the skeleton with SHD ≤ 49 but orients edges at random and is outperformed by DirectLiNGAM (Figure 1(c)).

• Algorithm rankings flip with benchmark parameters. On p = 50 ER graphs with β = 1.2, FLOP's performance degrades with edge density while DirectLiNGAM and VarSortRegress remain stable (Figure 2(a)); the failure is attributed partly to score unreliability, since BIC optima no longer correspond to the true graph in that regime (Figure 5), consistent with Olko et al. (2025) empirically showing the proportion of λ-strong-faithful distributions decreases with graph density (the sentence is truncated in the supplied text). On p = 50 ER graphs with average degree 10, DirectLiNGAM is sensitive to β and fails close to β = 2 (Gaussian noise, which violates its assumptions), while FLOP and VarSortRegress are stable (Figure 2(b)). On p = 20 chains with β = 1.2, VarSortRegress improves with larger absolute edge weights while FLOP and DirectLiNGAM are stable (Figure 2(c)).

• Metric choice matters too. Results are reported with SHD in the main text and with the ancestor adjustment identification distance (AID) in Appendix C.3; the authors note the choice of metric shapes relative algorithm performance (Figure 7), and ancestor AID is zero when the found graph respects the true causal orders.

Methodology in Plain English

This is a position paper, not a new algorithm. The authors deliberately restrict themselves to a narrow, tractable setting — causal sufficiency, with the target being either a DAG or its Markov equivalence class — and focus on a representative corner: algorithms designed for linear additive noise models (DirectLiNGAM, VarSortRegress, FLOP). Within that setting they proceed in three steps. For identifiability, they construct small explicit structural causal models and compute covariance and precision matrices to demonstrate that faithfulness and covariance-faithfulness neither imply one another, and then compare this with how the community actually treats those assumptions. For consistency, they review the logical structure of pointwise and uniform consistency guarantees and show by construction how easily pointwise consistency can be bolted onto any candidate method. For benchmarks, they run controlled simulations on ER graphs and chains, varying average degree, noise peakedness (β), and edge weights, and evaluate with SHD (plus AID in the appendix), to show how rankings of the same three algorithms change with design choices. The deliberate narrowness is stated as a choice: the issues only become harder in the broader space of causal discovery.

Why This Matters

Impact on research: The paper argues that identifiability results, consistency guarantees, and benchmark leaderboards are currently accumulating as disconnected advances, because assumptions are studied in isolation, justified mainly by the positive results they enable, and rarely examined in light of when they fail. If adopted, its proposal would make failure modes and scope conditions first-class research outputs, and would let seemingly separate lines of work (linear vs. nonlinear, Gaussian vs. non-Gaussian) be re-read as living under a single assumption. It would also reframe benchmark construction: rather than removing artifacts such as varsortability, researchers would test whether such properties are exploited, and stress-test and ablate assumptions deliberately.

Real-world applications:

• Scientific discovery pipelines in biology, epidemiology, or economics where a practitioner must choose among structure learning methods with no principled guidance, and where the assumptions underlying each guarantee are untestable on the available data.
• Longitudinal studies and time-series experiments, where the authors argue known temporal order is a realistic identifiability assumption that the field nonetheless often treats as an artifact to be designed away.
• Downstream causal effect estimation, where the ancestor AID metric measures whether the learned graph yields valid adjustment sets for effect identification, tying graph recovery to a concrete decision task.
• Benchmark and tooling development for causal discovery, including openly maintained simulation suites whose graph families, densities, and noise settings currently encode unstated commitments.

Industry relevance: Teams applying causal discovery to product, operational, or clinical data must justify method choices to stakeholders. The paper gives them a vocabulary for that justification — explicit assumptions, stated failure modes, and benchmarks that actually probe them — and warns that published benchmark rankings may not transfer to their regime, since the same three algorithms change rank under small changes in density, noise shape, or edge weight magnitude.

Future Directions

• Define and agree on standards for what counts as a legitimate identifiability assumption, including how to weigh assumptions like MEC size of one, varsortability, R²-sortability, or known temporal order against established ones, and how to report when an assumption may not hold.
• Reformulate existing theory under weaker, more general assumptions — for instance, restating algorithms introduced for linear, additive, Gaussian models under covariance-faithfulness alone, and checking whether their empirical ablations are explained by that weaker condition.
• Decouple algorithm design from consistency proof templates so that new search heuristics, including continuous and relaxed approaches, are not structurally constrained by the proof strategy of a baseline such as GES.
• Design benchmarks that operationalize epistemic standards, systematically stress-testing and ablating identifiability and consistency assumptions (graph family, density, noise distribution, edge weights, metric) rather than rewarding methods aligned with one chosen regime — an agenda the paper only begins, since its supplied text is truncated partway through the benchmark section.

Target Audience

Causal discovery researchers and method developers, especially those working on structure learning theory, consistency proofs, or benchmark suites; statisticians and machine learning researchers who evaluate graph-recovery methods; and applied scientists or data practitioners who must justify a choice among causal discovery algorithms. Readers who want a concrete algorithm contribution will not find one here — the paper is aimed at people who shape how the field compares and evaluates methods.

Authors’ abstract

There are many causal discovery algorithms, but few guiding principles for choosing between them. Methods are usually compared through three lenses: identifiability, consistency, and benchmarks. Each introduces assumptions about the data-generating process that are often untestable. We argue that there are no shared standards for comparing, prioritizing, or even discussing those assumptions. We support this claim by showing that (i) identifiability assumptions are often preferred by convention rather than by principle, (ii) uniform consistency requires further untestable assumptions, while the weaker pointwise consistency has arguably narrowed algorithm design, and (iii) benchmarks are not neutral tests but carry an epistemic role of their own: they quietly fix which regimes matter, which guarantees come into play, and what counts as success. In doing so, they shape what algorithm comparisons are taken to show and thus perform some of the same epistemic work as theory, but less transparently. In this position paper we argue for a more transparent, integrated practice: Make the epistemic standards for discussing identifiability and consistency assumptions explicit, especially their failure modes and scope conditions; design benchmarks to operationalize those standards, by stress-testing and ablating both. Then, theoretical and empirical results jointly clarify when, why, and how algorithms may work rather than accumulate as disconnected advances.