CausalProfiler: Generating Synthetic Benchmarks for Rigorous and Transparent Evaluation of Causal Machine Learning

AI summary

Overview

Research area: Evaluation methodology for causal machine learning — specifically, the generation of synthetic benchmarks for causal inference methods.

Technical level: Intermediate. Readers need basic familiarity with structural causal models, interventions, and counterfactuals, though the paper's motivation is accessible without deep causal inference background.

Scope in one sentence: The paper introduces CausalProfiler, an open-source generator that randomly samples structural causal models, data, and queries from a user-specified "Space of Interest" to produce synthetic causal benchmarks with explicit assumptions and coverage guarantees.

What This Paper Is About

Causal machine learning methods are supposed to answer "what if" questions, but the field evaluates them on a small number of hand-crafted or semi-synthetic datasets whose underlying assumptions are rarely stated and even more rarely varied. This makes empirical conclusions brittle: minor changes to the data-generating process can flip method rankings. The paper's goal is to replace fixed datasets with a configurable generator that samples entire families of causal datasets under transparent, controllable assumptions, so that methods can be probed for failure modes and robustness rather than scored on a single leaderboard.

Key Contributions

1. CausalProfiler, an open-source synthetic benchmark generator. It is described as the first random generator of synthetic causal benchmarks with coverage guarantees and transparent assumptions that operates on all three levels of the Pearl Causal Hierarchy: observation (L1), intervention (L2), and counterfactual (L3). Code is available at https://github.com/panispani/causal-profiler.

2. The Space of Interest (SoI) abstraction. An SoI is defined as a tuple S = (M, Q, D) — a class of SCMs, a class of causal queries, and a class of data. This shifts the evaluation question from "which dataset should I use" to "which space of conditions do I want to evaluate over."

3. A coverage guarantee (Proposition 5.1). For a Space of Interest whose SCM class is a class of regional discrete SCMs with the maximum number of noise regions, any causal dataset has strictly positive probability of being generated. The paper is explicit that this is an expressivity statement, not a claim of uniform sampling.

4. An empirical demonstration. The generator is validated against causal axioms and then used to evaluate four recent causal inference methods across multiple SoIs, illustrating the kinds of robustness and failure-mode analyses the framework makes possible. The paper states that a causal dataset also contains the ground-truth graph, so it can be used for causal discovery evaluation by leaving the query empty.

Main Findings

• Correctness checks pass. On L1, the Markov property was tested by enumerating d-separated triplets (A, B, C) and testing conditional independence with Pearson's chi-squared test; about 5% of tests failed, mostly attributed to finite-sample variability. On L2, the three rules of do-calculus were tested by comparing interventional distributions; about 5.5% of tests failed, again mostly attributed to finite-sample variability. On L3, the structural counterfactual axioms of composition, effectiveness, and reversibility held exactly across all samples.

• CausalProfiler covers a much wider metric space than existing benchmarks. Applying two-dimensional t-SNE with perplexity 30, the 11 synthetic SCMs from Causal Normalizing Flows (CausalNF) occupy a very narrow region, whereas SoI sampling produces a much broader spread. The two bnlearn networks used for comparison (CANCER and EARTHQUAKE, selected from 24 available) overlap with the CausalProfiler embedding, showing the generator can reproduce familiar dataset characteristics while adding diversity.

• Method rankings change with the functional form of the causal mechanisms (ATE experiment, continuous SCMs). In the Linear-Medium SoI, Diffusion-based Causal Models (DCM) achieved the lowest mean error at 0.1530, but with a high standard deviation of 1.5289 and a maximum error of 33.9766. In NN-Medium, DCM became the weakest performer (mean 0.0276) while VACA achieved the lowest mean error (0.0090) and standard deviation (0.0077).

• VACA's failure rate depends strongly on the SoI. It reached a 53.40% failure rate in Linear-Medium, dropped to 5.00% in NN-Medium, and was 11.60% in NN-Large. Failures were attributed to numerical instabilities producing invalid floating-point values.

• Low-data regimes penalize some methods much more than others. Comparing NN-Large (1000 samples) with NN-Large-LowData (50 samples), DCM's mean error rose from 0.0267 to 0.0777 (nearly tripling) and its interquartile range expanded; CausalNF's mean error rose from 0.0159 to 0.0359. NCM (0.0101 to 0.0097) and VACA (0.0090 to 0.0103) were essentially unchanged, and VACA's failure rate fell to 0.00% in the low-data setting.

• On discrete counterfactual estimation, DCM had lower mean error but CausalNF was much faster. On D2-Reject, DCM's mean error was 0.1974 versus 0.5233 for CausalNF. On D2L-Unbias, it was 0.1895 versus 0.4340. On D4-Unbias, DCM had a moderately lower mean error (0.4087 versus 0.4783) while CausalNF had a lower standard deviation (0.3149 versus 0.3418) and a lower maximum error (1.0000 versus 2.0000). CausalNF was approximately 16x, 15x, and 21x faster than DCM on D2-Reject, D4-Unbias, and D2L-Unbias respectively.

• Sampling is not uniform. The paper states that sampled causal datasets are not uniformly distributed, and that certain classes of SCMs (for example, linear SCMs) remain unlikely to be sampled unless explicitly specified in the SoI. It also notes that uniform sampling over discrete mechanism functions biases toward non-bijective functions, since bijections are not dense in the function space.

• The experiments are framed as demonstrations, not definitive benchmarks. The paper repeatedly states that its findings are specific to the explored SoIs and should not be taken as general conclusions about the evaluated methods.

Methodology in Plain English

The researchers build a generator rather than a dataset. A user writes down a Space of Interest, which is a specification of what kinds of causal models, queries, and data are in scope. The generator then randomly draws a causal graph and causal mechanisms to build a structural causal model; it can mark some variables as hidden and applies Verma's latent projection so the user only sees the observable causal structure.

Mechanisms come in two flavors. Discrete (regional discrete) mechanisms are defined tabularly, mapping regions of the noise space to different parent-to-child functions, which allows controllable stochasticity including highly non-linear and non-invertible behavior. Continuous mechanisms use parametric families such as neural networks or linear functions with randomly initialized parameters.

From the sampled model, the generator forward-samples observational data in topological order, picks treatment/outcome/covariate variables according to the query class, and computes the true answer to each query — interventional queries by applying do-operations, and counterfactual queries via Pearl's three-step abduction-action-prediction procedure. The result is a dataset with a known query, known answer, known causal graph, and known assumptions.

To check that the machinery is sound, the authors ran consistency tests across all three levels of the causal hierarchy using discrete SCMs (which allow exhaustive enumeration of conditioning sets) and a parameter grid over the number of variables, edge density, cardinalities, and noise regions. For the method comparisons, each SoI was evaluated with five random seeds and 100 SCMs per seed; each SCM yielded one training set and five queries with ground-truth values, aggregated across SCMs and seeds. Errors were measured as mean squared error between predicted and true query values, alongside standard deviation, runtime, and failure rate. All experiments ran on a single Intel Core i9-14900K machine (24 cores, 32 threads, 96GB RAM), fully parallelized on CPU.

Why This Matters

Impact on research. The paper argues that narrow, static benchmarks give a false sense of reliability, and that implicit design choices in synthetic datasets can bias results toward particular methods. By turning evaluation into a controlled parameter sweep with a stated coverage guarantee, CausalProfiler makes it possible to systematically probe failure modes, generalization limits, and sensitivity to violated assumptions — analyses that current single-dataset evaluations cannot support. It also makes assumptions a first-class object of study rather than an unstated artifact.

Real-world applications (as framed by the paper):

• Medicine, where causal ML is described as enabling principled decision making.
• Public policy, cited alongside medicine as a field where intervention and counterfactual estimates matter.
• Safety-critical settings — the paper notes that occasional large errors in a well-performing method could be "problematic in safety-critical applications" matching that SoI.
• Causal discovery, since each generated dataset includes the ground-truth graph, allowing discovery algorithms to be assessed directly by leaving the query empty.

Industry relevance. The author list includes an industry affiliation (Ekimetrics, Paris), and the framework is aimed at practitioners who must decide which method suits their modeling context. The paper's central practical message is that rankings are SoI-specific: a method that wins on linear mechanisms can lose on neural ones, and the right question is not which method is best overall but which method is best under the conditions a practitioner actually faces. CausalProfiler also lowers the cost of that assessment, since randomized controlled trials are described as expensive, ethically constrained, and often small.

Future Directions

• Diversify the supported Spaces of Interest. The authors list scaled and mixed-variable SCMs, sampling interventional training data, more realistic generating scenarios (selection bias, measurement noise), and extension beyond tabular data to time-series or text.

• Automate SoI exploration. A stated goal is to search for assumption regimes that reveal a method's failure modes, reducing reliance on manual specification.

• Reduce distributional bias in sampling. Coverage does not imply uniformity. The authors propose stratified sampling as a promising avenue — weighted sampling over groups of SoIs for controllable parameters, and rejection sampling or new algorithms for emergent properties. They note that perfectly balanced coverage over all metrics is inherently impossible.

• Bridge the simulation-to-real gap. Alignment with real domains currently depends on manually specified SoIs. The authors propose automatically mapping real data to sets of SoIs to enable semi-synthetic pipelines shaped by empirical evidence. They caution that this mapping is fundamentally underconstrained, and that one caveat remains open: whether synthetic results transfer to real-world settings.

Note: the provided paper text ends mid-sentence in Section 7, so any further listed future directions are not reported in the available content.

Target Audience

Causal machine learning researchers and method developers who need to test their estimators under varied and explicitly stated assumptions; benchmark designers and reproducibility-focused methodologists; and applied data scientists in industry or policy who must choose a causal estimation method for a specific setting and want to know how sensitive that choice is to the data-generating regime. Readers whose primary interest is a leaderboard of causal inference methods will find the paper's deliberately non-definitive stance less useful than those interested in evaluation infrastructure.

Authors’ abstract

Causal machine learning (Causal ML) aims to answer "what if" questions using machine learning algorithms, making it a promising tool for high-stakes decision-making. Yet, empirical evaluation practices in Causal ML remain limited. Existing benchmarks often rely on a handful of hand-crafted or semi-synthetic datasets, leading to brittle, non-generalizable conclusions. To bridge this gap, we introduce CausalProfiler, a synthetic benchmark generator for Causal ML methods. Based on a set of explicit design choices about the class of causal models, queries, and data considered, the CausalProfiler randomly samples causal models, data, queries, and ground truths constituting the synthetic causal benchmarks. In this way, Causal ML methods can be rigorously and transparently evaluated under a variety of conditions. This work offers the first random generator of synthetic causal benchmarks with coverage guarantees and transparent assumptions operating on the three levels of causal reasoning: observation, intervention, and counterfactual. We demonstrate its utility by evaluating several state-of-the-art methods under diverse conditions and assumptions, both in and out of the identification regime, illustrating the types of analyses and insights the CausalProfiler enables.