Linear-LLM-SCM: Benchmarking LLMs for Coefficient Elicitation in Linear-Gaussian Causal Models

AI summary

Overview

Research area: Machine learning / causal inference — specifically evaluating large language models as quantitative causal-effect estimators inside linear-Gaussian structural causal models (SCMs).

Technical level: Intermediate. Readers should be comfortable with directed acyclic graphs (DAGs), linear regression coefficients, and basic causal-inference vocabulary; no deep mathematical background is required to follow the argument.

Scope: The paper introduces an open-source benchmark that asks whether LLMs can assign numerical coefficients to a pre-specified causal DAG using only structure and variable semantics, without any observational data, and tests that ability on seven real-world DAGs with four LLM models.

What This Paper Is About

Prior work suggests LLMs can identify qualitative causal relationships ("X causes Y") from prior knowledge, but their ability to produce quantitative effect sizes in continuous domains is largely untested. The paper builds a framework that hands an LLM a known DAG, node-by-node, and asks it to write the regression-style structural equation for each child given its parents. The goal is to measure how close those elicited coefficients are to real ground-truth parameters, and how badly they break under adversarial perturbations.

Key Contributions

1. A plug-and-play benchmarking framework (Linear-LLM-SCM) that decomposes any pre-specified DAG into local parent-child elicitation tasks, prompts an LLM for a linear structural equation per node, and aggregates the extracted coefficients into a full parameterized SCM.
2. Four evaluation metrics (M1–M4) comparing LLM-elicited coefficient vectors against ground truth: an unnormalized L2 distance (M1), a node-wise normalized L2 distance (M2), an L2 distance excluding single-parent edges (M3), and a relative effect-size ordering count (M4).
3. An empirical study over seven real-world DAGs from the BnRep Bayesian Network repository in continuous linear-Gaussian settings (cachexia1, expenditure, foodsecurity, algal2, lexical, liquefaction, stocks), each run against four models — Gemini 2.5 Flash, Llama 3.1 8B, Llama 3.3 70B, and GPT-5.4 — with 25 independent runs per (DAG, model) condition.
4. Robustness experiments under adversarial conditions: unit rescaling on cachexia1 (μM to nM) and simulated DAG misspecification on expenditure via four spurious-edge variants (S1–S4).

Main Findings

• Best overall performer: Gemini 2.5 Flash performs best overall on M3 and M4 across the tested DAGs. Under M3, Gemini 2.5 Flash and GPT-5.4 each perform best on three DAGs; under M4, GPT-5.4 is best on four DAGs and Gemini 2.5 Flash on three, including ties.
• Scale sensitivity of M1: M1 shows high variability across DAGs because it is not scale-invariant with respect to variable ranges. On expenditure, M1 values range from 2463.377 ± 0.000 (Llama 3.1 8B) to 148084.559 ± 22359.281 (Gemini 2.5 Flash), which is why the authors focus discussion on M2–M4.
• Model scale helps within a family: Comparing Llama 3.1 8B and Llama 3.3 70B, the larger model usually performs better, suggesting a benefit from larger parameter counts for this task.
• Small-model parsing failure: Llama 3.1 8B failed to generate parsable equations for algal2, so no value is reported for it on that DAG.
• Stochasticity at temperature zero: Despite temperature zero, the larger models (Gemini 2.5 Flash, Llama 3.3 70B, GPT-5.4) still show substantial stochasticity across metrics, with non-Gaussian distributions at n = 25, possibly due to hardware or software factors beyond the authors' control via external APIs. The authors note this aligns with prior work on LLM non-determinism and flag it as a failure mode under RQ3.
• A small model was more consistent but less accurate, while the three large models were inconsistent — variance the authors describe as concerning for safety-critical domains such as healthcare.
• Unit tweaking produced counter-intuitive results: In some M2–M4 cases, changing cachexia1 units from μM to nM improved parameterization rather than degrading it. A possible explanation offered is increased numerical precision in textual form, e.g., 1 μM becoming 1000 nM; the authors state they plan to extend empirical coverage to test this.
• DAG misspecification degrades performance: Under M3, the original expenditure DAG yields the best result for Gemini 2.5 Flash, Llama 3.1 8B, and Llama 3.3 70B, while GPT-5.4 performs best on variant S1. Under M4, the original DAG performs best for Gemini 2.5 Flash and Llama 3.1 8B, while for Llama 3.3 70B and GPT-5.4 variant S4 is slightly better, with the original remaining second best. Overall, adversarial conditions tend to lower M4, indicating degraded performance under spurious edges.
• Spurious edges tested: (S1) Owner → Expenditure, (S2) Majorcards → Dependents, (S3) Owner → Share, (S4) Majorcards → Selfemp, each added while preserving acyclicity.

Methodology in Plain English

The framework takes four inputs: a DAG structure, short textual descriptions of each variable including units, hard numeric bounds (lower and upper) for each variable, and an overview of the real-world phenomenon. Variable bounds were sourced from original literature or author judgment.

Processing works through the graph in topological order. For each target node, the system collects that node's direct parents and builds a prompt in three parts: (1) a domain-expert persona plus a phenomenon summary and variable units; (2) the parameterization task with a linear equation template and variable ranges; (3) output-format instructions, requiring reasoning tokens followed by the equation as a string. The model is asked to return JSON, and the system parses only the coefficients and intercept from the returned structural equation, discarding the explanatory text.

To keep the per-node answers globally consistent with the DAG, an iterative feedback loop applies interval arithmetic. Given elicited coefficients and known parent bounds, the validator computes the worst-case deterministic range of the target and rejects any proposal whose predicted minimum or maximum falls outside the target's hard bounds, feeding the failed proposal and failure reason back into the next prompt; after a loop budget of n iterations it returns the last available coefficients. The noise term is excluded from this interval calculation.

Ground truth comes from real-world DAGs in the BnRep Bayesian Network repository, restricted to continuous linear-Gaussian settings with at most 15 nodes, excluding DAGs whose variable names were letters with numbered suffixes. Some variables (for example Card in the expenditure DAG) were originally binary but were treated as continuous because the ground-truth network learned the effect as continuous.

For each (DAG, LLM) condition, 25 independent runs were performed with temperature zero, and mean M1–M4 values are reported with 95% confidence intervals. Gemini models were called via Google AI Studio APIs, Llama models via Groq APIs, and all APIs through the instructor Python package, which enforced the JSON format and handled rate limiting with default exponential backoff. Seeds were not explicitly set (both providers describe seeding as best-effort); for GPT-5.4, reasoning_effort was set to none to permit temperature zero.

Why This Matters

Impact on research. The paper moves the LLM-causality conversation from structural discovery to quantitative parameterization in continuous domains, which prior work had largely left unexplored. It situates itself against Zhang et al.'s "LLMs' Causal Hierarchy," where Type 3 (quantitative estimation of consequences) is the open challenge, and distinguishes itself from Bynum and Cho (2025) and Nafar et al. (2025), which estimate effects via sampling-based conditional distributions rather than directly eliciting structural parameters. By releasing the benchmark as open source, the authors aim to make it straightforward for other groups to plug in their own DAGs, variable metadata, and LLMs.

Real-world applications (domains the paper's DAG selection and framing point to, with the paper's own caveat that current results do not support deployment):

• Healthcare and clinical decision support — the abstract and Impact Statement name this as the motivating safety-critical domain.
• Economics — the expenditure and stocks DAGs cover consumer expenditure and financial variables.
• Environmental and earth science — the algal2 and liquefaction DAGs cover water/ecosystem processes and soil or seismic phenomena.
• Genetics and social sciences — the cachexia1, foodsecurity, and lexical DAGs cover metabolic syndrome, food security, and language-related social measures.

Industry relevance. The results supply a concrete caution: the observed run-to-run variability at temperature zero in frontier models, plus degradation under spurious edges, makes these models unreliable as drop-in coefficient estimators in high-stakes pipelines. The authors state explicitly that LLMs should not be used for quantitative causal effect estimation in real healthcare or clinical decision support, and that further safety testing and improvements are needed.

Future Directions

• Extend the framework to non-linear functional forms, since the current study is restricted to linear causal effects.
• Investigate ways to mitigate structural noise, given that spurious edges degrade effect-ordering accuracy.
• Examine whether textual uncertainty outputs (confidence intervals) from LLMs are calibrated — the authors did not use them here but plan to test this.
• Expand empirical coverage of the unit-tweaking condition to determine whether improved metrics under changed units truly stem from increased numerical precision (e.g., 1 μM to 1000 nM).
• Explore reducing the stochasticity of large LLMs at temperature zero, identified as a failure mode and a barrier for safety-critical use.

Target Audience

Researchers and practitioners working at the intersection of LLMs and causal inference — particularly those interested in causal effect estimation, structural causal models, and clinical or epidemiological machine learning. It is also useful for safety-focused ML engineers and evaluators who need evidence on where frontier and open-weight LLMs fail on quantitative reasoning tasks, and for benchmark designers looking for an existing open-source harness they can extend to their own DAGs and domains.

Authors’ abstract

Large language models (LLMs) have shown potential in identifying qualitative causal relations, but their ability to perform quantitative causal reasoning---estimating effect sizes that parametrize functional relationships---remains underexplored in continuous domains. We introduce Linear-LLM-SCM, a plug-and-play framework for evaluating LLMs on Linear Gaussian structural causal model parametrization when a directed acyclic graph (DAG) is given. The framework decomposes a DAG into local parent-child sets and prompts an LLM to produce a regression-style structural equation per node, which is aggregated and compared against available ground-truth parameters. Our experiments with seven real-world DAGs effect ground truth illustrate limitations of LLMs as quantitative causal parameterizers. Across most models, we observe variability in coefficient estimates and sensitivity to structural perturbations. We open-sourced the framework to further encourage the community to work on studies toward the use of LLM for causal effect elicitation in safety-critical domain, e.g., healthcare.