Dynamic Expert-Guided Model Averaging for Causal Discovery

AI summary

Overview

Research area: Causal discovery (learning causal graphs / Bayesian networks from observational data), ensemble learning, and expert-in-the-loop methods using human experts or large language models (LLMs).

Technical level: Intermediate. Readers should be comfortable with directed acyclic graphs (DAGs), partially directed acyclic graphs (PDAGs), and standard causal discovery algorithm families, but the method itself is described procedurally.

Scope: The paper proposes and empirically evaluates a model-averaging (ensembling) method for causal discovery that resolves disagreements between algorithms by selectively querying an imperfect expert, treating edge existence and edge orientation as separate decisions.

What This Paper Is About

Practitioners of causal discovery face many competing algorithms with no clear best choice for a given use case, and real-world data frequently violate the assumptions those algorithms rely on. The authors build a flexible ensembling method that combines a diverse set of causal discovery algorithms and, when the algorithms disagree, dynamically asks an imperfect expert (human or LLM) to resolve the discrepancy, under a limited query budget. The goal is a final graph closer to the unknown ground-truth causal graph than what individual algorithms or existing ensembling approaches produce.

Key Contributions

1. A model-averaging strategy that uses dynamically requested expert knowledge to mediate discrepancies between component causal discovery algorithms. The authors state this is, to their knowledge, the first exploration of expert knowledge for model averaging or ensembling in causal discovery.
2. A formulation that treats edge existence and edge orientation as separate questions, querying the expert selectively in cases of greater disagreement between models in order to maximize the value of limited access to an imperfect expert.
3. An evaluation against strong baselines on well-known real-world Bayesian networks, using both clean data and noisy data, with statistical significance testing over 20 seeds.

Main Findings

• Broad improvements in overall accuracy metrics: Averaged across all networks, sample sizes, and seeds, the method shows broad improvements on all evaluated metrics except precision, with reliably higher Balanced Scoring Function (BSF) and F1 scores than baseline methods by several percentage points on both clean and noisy data.
• Structural metrics improve: Relative Structural Hamming Distance (SHD) and Structural Intervention Distance (SID) typically improve by wide margins, with the exception of the Oracle baseline, which the authors describe as an unrealistically high bar since the best individual method would not be known ahead of time in practice.
• Precision–recall trade-off relative to other ensembles: The method provides recall comparable to other ensembling methods with much higher precision.
• Outperforms expert-augmented single algorithm: The method widely improves over Tabu-AL on all metrics (save precision with noisy data) by at least 11 percent on average, which the authors present as evidence for the value of dynamic expert knowledge in ensembling rather than in augmenting one algorithm.
• Statistical significance: On clean data, improvements over GES and MMHC are significant on 100 percent of settings for BSF (28/28 each); over FCI on 96 percent (25/26) and over MMHC on 100 percent (28/28) for F1. On noisy data, improvements over FCI are significant on 100 percent (25/25) for F1. Significance was determined by a one-sided Wilcoxon signed-rank test over 20 seeds (p < 0.05).
• Individual algorithms retain higher precision: While individual algorithms often have higher precision, this is balanced by the greatly improved recall of the proposed method. A higher edge threshold θ1 can be used to achieve higher precision at the cost of lower recall (Appendix G in the paper).
• Query budget trade-off: As the budget B increases, recall increases while precision decreases. Across all settings, BSF goes from 0.40 ± 0.04 at B = 0.25N to 0.51 ± 0.06 at B = ∞; F1 peaks at 0.57 ± 0.05 for B = N and B = 2N; recall goes from 0.40 ± 0.04 to 0.59 ± 0.05 at B = ∞; precision falls from 0.84 ± 0.09 to 0.61 ± 0.07. A budget in the range N to 2N performs best overall.
• Expert quality matters: Performance increases linearly with expert correctness. The method surpasses Bayesys Model Avg on F1 at p_exist ≈ 50 percent and p_orient ≈ 50 percent, suggesting that using an expert in ensembling is valuable only when the expert performs better than random. Even at low expert correctness the method achieves higher precision than Bayesys Model Avg, while Bayesys Model Avg achieves higher recall up to around 80 percent expert correctness.
• Querying to resolve disagreement is better: Appendix B.4 reports that querying the expert for all edges (rather than prioritizing areas of disagreement) performs worse on all metrics save precision, at all tested query budgets.

Methodology in Plain English

The method takes as input the outputs of M causal discovery algorithms, each producing a PDAG, collected into a set of graphs. It then greedily builds a final graph by adding edges in order of how often they appear across the component models, avoiding cycles as it goes.

The key design choice is that edge existence and edge orientation are handled separately:

• For a candidate edge between variables x and y, any direct connection (x → y, y → x, or x − y) counts equally toward establishing that some connection exists, regardless of direction.
• A predetermined edge threshold θ1 sets the percentage of component models that must contain a connection before an edge is considered. If that percentage is above θ1 but below 0.5 (a majority vote), the expert is queried and may reject the edge.
• For accepted edges, orientation is determined next. If only one direction is valid without creating a cycle, that direction is used. If both are valid, the method checks whether the percentage of models with a particular orientation, out of the models containing a direct connection, exceeds an orientation threshold θ2. If so, that orientation is used; otherwise the expert is queried.
• A query budget B limits the total number of expert queries. Once the budget is exhausted, all remaining decisions default to majority voting.

The rationale for separating the two questions comes from two observations in the paper: (1) from observational data it is generally only possible to learn a CPDAG, so orientation is often genuinely ambiguous and algorithms may agree an edge exists while disagreeing on its direction; and (2) prior work suggests LLMs and human experts are better at judging causal direction between a pair of variables than at distinguishing causation from mere correlation, so the expert is only queried about edges the ensemble already detected, never to introduce new edges.

For experiments the authors use a simulated expert rather than a real human or LLM. Based on the ground-truth graph, this expert answers existence queries correctly in p_exist of cases and orientation queries correctly in p_orient of cases. Existence answers are based on whether there is some chain x → ... → y (which may be indirect) rather than a direct edge, to reflect the practical difficulty of eliciting accurate expert knowledge. When both variables have no causal relation in the ground truth, an arbitrary orientation is returned.

Experiments use six networks from the Bayesys repository plus the SimSUM network, with clean data and the noisy data generated in prior work, at sample sizes of 100, 1000, 10000, and 100000. Because variable ordering can affect results, each experiment is repeated over twenty random orderings of the dataset columns. The ensemble consists of PC-Stable, FCI, Tabu, GES, MAHC, BOSS, MMHC, and DAG-GNN. Baselines include the ensembling approaches of Constantinou (2019a), referred to as Bayesys Model Avg, and Aslani and Mohebbi (2023); Tabu-AL as a dynamically expert-queried single algorithm; and an Oracle that selects the best individual method for each given setting. Metrics reported are BSF, F1, precision, recall, SHD, and SID; reversed or undirected edges count as partial matches (0.5 of a false negative and 0.5 of a true positive), except for SID, which is computed differently.

Main results use an expert with p_exist = 70 percent and p_orient = 90 percent, which the paper states is in line with LLM performance on a subset of the benchmark networks (Appendix F). The main configuration uses θ1 = 0.0, θ2 = 0.9, and a query budget of 2N, where N is the number of nodes; the authors state these values were chosen to optimize BSF and F1 scores.

Benchmark networks used, with nodes, edges, parameters, and domain as reported in the paper: Asia (8 nodes, 8 edges, 18 parameters, reasons causing dyspnea); Sports (9, 15, 1049, football match outcomes); Property (27, 31, 3056, property investment); Alarm (37, 46, 509, ALARM monitoring system); Formed (88, 138, 912, forensic / criminal psychology); Pathfinder (109, 195, 71,890, diagnosis of lymph-node diseases); SimSUM (16, 35, 3368, respiratory diseases).

Why This Matters

The paper targets a practical gap: no single causal discovery algorithm consistently outperforms the others on real-world data, and the assumptions behind them are often violated. By combining algorithms and using an imperfect expert only where models disagree, the work offers a way to get more out of limited, noisy data and limited expert access.

Potential real-world applications (grounded in the benchmark domains the paper uses and its stated motivation):

• Healthcare and clinical diagnosis, including the dyspnea-related, lymph-node disease, and respiratory disease networks used in the benchmarks, where causal structure can inform which randomized controlled trials are most worth running.
• Forensic and criminal psychology, represented by the Formed network, where understanding causal mechanisms matters for interpretation.
• Property investment, represented by the Property network, where decisions depend on understanding drivers rather than only prediction.
• Industrial and safety monitoring, represented by the ALARM monitoring system network, where identifying dependency structure among alarms supports diagnosis.

Industry relevance: The method is designed around realistic constraints encountered in deployment — an explicit query budget, an imperfect expert, noisy data, and a choice of expert that can be a human or an LLM. Because the runtime of the ensembling step is effectively dominated by query latency (expected runtime Θ(L × B), where L is average query latency and B is the query budget), the approach is designed to be practical when expert or LLM access is the bottleneck. The authors also report that in practice the overall runtime is dominated by the component algorithms.

Future Directions

• Applying the method to other data types and noise distributions, including real data, since the current experiments involve only synthetic, categorical data and synthetic noise.
• Adapting the weights given to different models during averaging, rather than the current equal weighting, based on predicted performance of individual algorithms.
• Using the model averaging method across multiple runs of the same algorithm rather than across an ensemble of disparate algorithms, for improved stability (for non-deterministic algorithms or those sensitive to variable ordering) or efficiency across different data partitions.
• Further exploration of dynamic expert knowledge for model averaging, which the authors identify as an avenue they expect to have substantial potential.

Target Audience

Researchers and practitioners working on causal discovery, probabilistic graphical models, and Bayesian networks who need robust structure learning on real, noisy data; applied scientists in domains such as healthcare, forensics, and industrial monitoring who must combine algorithmic output with domain expertise; and researchers interested in using LLMs as experts within algorithmic pipelines, including those studying expert-in-the-loop and active querying methods.

Authors’ abstract

Would-be practitioners of causal discovery face a dizzying array of algorithms without a clear best choice. This abundance of competitive methods makes ensembling a natural strategy for practical applications. At the same time, real-world use cases frequently violate the assumptions on which common causal discovery algorithms are based, forcing reliance on expert knowledge. Inspired by recent work on dynamically requested expert knowledge and large language models (LLMs) as experts, we present a flexible model averaging method that integrates selective expert querying to ensemble a diverse set of causal discovery algorithms. Crucially, we distinguish between edge existence and orientation, enabling the method to leverage the complementary strengths of data-driven discovery and expert input. We further consider the realistic setting of limited access to an imperfect expert, using disagreement among algorithms to query the expert in cases of greater uncertainty. Experiments demonstrate that our method consistently outperforms strong baselines on both clean and noisy data. Code and data are available at https://anonymous.4open.science/r/expert-cd-ensemble-3282/.