Causal Foundation Models

AI summary

Note: the supplied paper content is truncated mid-Section 2.6 (it ends inside the description of the prior-data loss). Sections 3, 4, and 5 — including all benchmark results — are therefore not present in the content provided. Every place where a result, dataset, or number is expected but absent is marked explicitly below as "not reported in the available content."

Overview

Research area: Causal inference at the intersection of machine learning, specifically a new class of pretrained transformer models that perform causal effect estimation via in-context learning.

Technical level: Intermediate. The paper is written as a practical, hands-on introduction, but it develops formal background in the Neyman-Rubin potential outcomes framework, identifiability, structural causal models, and Bayesian posterior predictive distributions before reaching the core material.

Scope: A tutorial-style introduction to causal foundation models (CFMs) — their motivation, the causal inference and machine learning background required to understand them, an intended benchmark comparison of existing CFMs against traditional estimators, and a survey of emerging developments — accompanied by example code, Jupyter notebooks, and a public codebase.

What This Paper Is About

Causal inference traditionally requires a bespoke pipeline for each new problem: propose a causal mechanism, select a compatible estimator, tune hyperparameters, then train on the data — with no reuse of tuned models or transfer of knowledge across tasks. This paper introduces causal foundation models as an alternative: pretrained neural networks that estimate causal quantities such as the average treatment effect on entirely new datasets using in-context learning, without any model updates or fine-tuning. The goal is to give practitioners the background, tools, and working examples needed to use CFMs in their own work.

Key Contributions

1. A practical, hands-on introduction to CFMs. The paper is framed explicitly as an accessible entry point, with example code and Jupyter notebooks linked throughout and a full codebase released at github.com/layer6ai-labs/cfms.

2. A consolidated background bridging causal inference and machine learning. The paper develops the potential outcomes framework, causal estimands (CEPO, ATE, CATE, ITRC), identifiability conditions, structural causal models, Bayesian posterior predictive distributions, and prior-data fitted networks in one linear narrative aimed at newcomers to CFMs.

3. A comparative benchmark of the first openly available CFMs. Section 4 is described as benchmarking "the first three openly available CFMs" (Robertson et al., 2025; Balazadeh et al., 2025; Ma et al., 2026) together with popular traditional causal inference models. (The benchmark results themselves are not present in the available content.)

4. A survey of new developments and cross-disciplinary applications. Section 5 reviews recent developments in the field as promising areas for future growth, including applications of CFMs in other scientific areas.

Main Findings

• CFMs require no training or fine-tuning at inference. The central claim is that CFMs are pretrained neural networks that can be applied immediately to any causal inference task. The accompanying code sample shows .fit() being called only to load context data, with no training required, followed by estimate_cate() on query data.

• CFMs provide both speed and performance gains. The paper states that CFMs "bring not only a vast increase in inference speed, but also improved performance," and that they have demonstrated top performance on causal inference tasks. (No specific speedup factor, metric, or numeric performance comparison is reported in the available content.)

• Prediction happens through in-context learning on labeled examples. Labeled context data (X_ctx, T_ctx, Y_ctx) is supplied as examples so the model can produce causal estimates on unlabeled query data (X_qry).

• CFMs perform amortized Bayesian inference. Rather than approximating a posterior and then computing a posterior predictive distribution, CFMs are trained on tasks sampled from a prior over possible data-generating processes and causal mechanisms.

• Traditional causal inference does not transfer across problems. The paper notes that for methods such as Bayesian additive regression trees, double machine learning, causal forests, and S-, T-, and X-learners, the full pipeline must be repeated for every new problem, with no opportunity to reuse tuned models or transfer knowledge between tasks.

• Identifiability is the central obstacle. Without additional assumptions, causal estimands (ATE, CATE, CEPO) are not identifiable given the full set of data-generating processes. The paper formalizes this with observational equivalence classes: two distributions are indistinguishable from observational data no matter how much data is available.

• Three assumptions unlock the backdoor setting. Ignorability (conditional unconfoundedness), positivity (overlap), and SUTVA (no interference, no hidden versions of treatments) together guarantee that the conditional interventional distribution and CEPO are identifiable and recoverable from the observational distribution via backdoor adjustment. The paper notes a key limitation: ignorability is an untestable assumption in practice.

• Bayesian uncertainty is decomposed into two types. The conditional interventional distribution expressed as a posterior predictive distribution separates epistemic uncertainty (which data-generating process produced the data, including structural uncertainty from observationally equivalent DGPs) from aleatoric uncertainty (randomness in the outcome under a fixed DGP).

• Prior-data fitted networks go directly from dataset to posterior predictive distribution. PFNs are transformers that skip explicit posterior approximation, which means uncertainty quantification is produced in the same forward pass used for prediction. Their training relies on three ingredients: the prior-data loss, tractable yet highly diverse priors generating unlimited synthetic tasks, and a transformer architecture enabling in-context learning.

Methodology in Plain English

The paper's approach is expository rather than experimental in its opening sections. It builds a conceptual scaffold in a deliberate order:

First, it motivates the distinction between prediction and causation using a classic illustration: across European countries, regions with more storks tend to have higher human birth rates. A purely predictive model exploits this correlation and forecasts well — but asked what would happen if a region doubled its stork population, it fails, because that is an interventional question. The paper uses this to argue that a predictive model answers "what outcome do we expect for this individual, as-is?" while a causal model answers "what outcome would we expect if we intervened?"

Second, it formalizes causal inference in the Neyman-Rubin potential outcomes framework. For every possible treatment value t, each individual is assumed to have a potential outcome Y(t). The consistency assumption ties potential outcomes to observed outcomes (T = t implies Y = Y(t)). The fundamental problem of causal inference is that we can never observe the counterfactual — once an individual receives treatment t, we cannot know what would have happened under a different treatment t'. Because of this, causal quantities are expressed as expectations rather than pointwise counterfactuals: the conditional expected potential outcome (CEPO), the average treatment effect (ATE), the conditional average treatment effect (CATE), and, for continuous treatments, the individual treatment-response curve (ITRC).

Third, it explains why causal effects are hard to identify. The full joint distribution over covariates, treatments, and potential outcomes is generally unknown; only the observational distribution over (X, T, Y) is accessible. Many different joint distributions can produce the same observational distribution, so practitioners must restrict the space of allowable data-generating processes by making assumptions. The paper lays out the standard three — ignorability, positivity, and SUTVA — and shows they yield the backdoor adjustment formula that recovers interventional distributions from observational data. It also notes two alternatives where identification holds under different assumptions: instrumental variables (which give only partial identifiability, narrowing the estimand to a finite interval) and the frontdoor setting (less used in practice).

Fourth, it introduces a generative machinery for data-generating processes. Bayesian networks represent dependencies between variables via a directed acyclic graph. Augmenting a Bayesian network with explicit structural equations and exogenous noise variables produces a structural causal model (SCM), which can simulate observational data by propagating samples through the equations in topological order. Interventions are modeled simply by replacing the structural equation for the treatment variable with a fixed constant — the paper gives a worked three-variable example with x = u₁, t = x² + 1 + u₂, and y = t − x + u₃ under standard normal noise. SCMs can also generate counterfactuals by fixing the exogenous variables while intervening.

Fifth, it connects to Bayesian inference. A prior over candidate data-generating processes is refined by observed data into a posterior, which induces a posterior predictive distribution over any causal estimand of interest — including the CEPO-PPD, ATE-PPD, CID-PPD, and CDTE-PPD. These generally lack closed-form solutions and have traditionally been approximated via Markov Chain Monte Carlo, variational inference, or neural posterior estimation.

Finally, it introduces prior-data fitted networks as a way to skip posterior approximation entirely. PFNs are trained with a prior-data loss — the expected negative log-likelihood of the model's output distribution at the correct label, given a context dataset — which is tractable because it never requires evaluating the true posterior predictive distribution. This training framework, combined with diverse synthetic task priors and transformer in-context learning, is the foundation on which CFMs are built.

Why This Matters

Impact on research. The paper argues that CFMs represent a paradigm shift for causal inference comparable to what foundation models did elsewhere in machine learning: pretrained once at scale, applied to new tasks without fine-tuning. If causal effect estimation can be amortized this way, it removes the per-problem engineering burden that has historically limited the deployment of causal methods. The paper also frames the field as young and expects "time will only demonstrate further transformational applications."

Real-world applications (drawn from the paper's own motivating examples and cited domains):

• Medicine and healthcare: estimating how effective a medication is at preventing a disease; the paper cites Alaa and van der Schaar (2017) and Shalit et al. (2017) on medical applications, and notes that the ITRC is also called the individual or conditional dose-response curve in medical settings.
• Economics and policy-making: determining whether a policy caused a decrease in unemployment; the paper cites Athey and Imbens (2017) and Chernozhukov et al. (2018).
• Monetary policy and consumer behavior: asking what the effect on consumer spending will be if a central bank raises interest rates.
• Marketing and finance: the paper cites Bottou et al. (2013) and Gordon et al. (2019) for marketing, and gives a financial example where a continuous treatment denotes a price or interest rate offered to a customer.

A recurring practical caveat: interventions may be expensive, impractical, or unethical — for instance, giving a patient a treatment that will probably harm them — which is why causal inference so often has to work from observational data alone.

Industry relevance. The authors are affiliated with Layer 6 AI (Toronto, Canada) and TD Bank Group (Toronto, Canada), placing the work squarely in industry applied-AI research. The financial framing of continuous treatments (price, interest rate offered to a customer) and the explicit goal of removing per-problem pipeline engineering point to commercial motivations: faster deployment, lower per-task cost, and reuse of a single pretrained model across many business problems. The paper explicitly invites "researchers and practitioners to give them a try."

Future Directions

1. Broadening beyond the backdoor setting. The paper highlights that ignorability is untestable in practice and that the backdoor setting has this key drawback. Extending CFMs to instrumental variable and frontdoor settings — where identification rests on different assumptions — is a natural open question. The IV setting is noted to give only partial identifiability, which raises the question of how a CFM should report or represent an interval-valued estimand.

2. Scaling up uncertainty quantification. CFMs inherit from PFNs the ability to produce posterior predictive distributions in the same forward pass as prediction. How well that translates into reliable, calibrated causal uncertainty on new datasets — including separating the epistemic from the aleatoric component — is not resolved in the available content.

3. Standardizing evaluation of CFMs. The paper benchmarks the first three openly available CFMs against popular traditional causal inference models, which raises the question of how CFM performance should be measured across the diverse data-generating processes, treatment types (binary, multi-armed, continuous), and estimands (ATE, CATE, ITRC) the paper describes. (The benchmark outcomes are not present in the available content, so no conclusions about comparative performance can be drawn here.)

4. Expanding into other scientific areas. The paper states that Section 5 reviews "applications of CFMs in other scientific areas," positioning cross-domain scientific deployment as a major growth area.

Target Audience

Primary audience: practitioners and applied researchers in causal inference who want to use CFMs on their own problems — particularly those in industry settings such as finance, healthcare, marketing, and policy analytics, who face the per-problem pipeline cost the paper describes.

Secondary audience: machine learning researchers interested in how the foundation-model paradigm transfers to new problem classes, and how in-context learning and prior-data fitted networks relate to Bayesian inference.

Prerequisite background: readers will benefit from basic familiarity with probability, statistical estimation, and neural networks. The paper supplies its own background in causal inference and Bayesian inference, making it accessible to those new to causal methods, but it does move into formal territory — potential outcomes notation, identifiability definitions, structural causal models, and posterior predictive distributions — so it is not an entry-level read.

Authors’ abstract

Causal inference is the practice of estimating the effect of a treatment or intervention from data. It traditionally requires a bespoke pipeline for every new problem: first proposing a causal mechanism, selecting a compatible estimator, and finally training it. Meanwhile, across diverse settings and modalities, much of machine learning has shifted to the paradigm of foundation models: networks pretrained once at scale and applied to new tasks without fine-tuning. Causal foundation models (CFMs) bring this paradigm to causal inference. CFMs are pretrained neural networks that estimate causal quantities, such as the average treatment effect, on entirely new datasets using in-context learning without requiring model updates. This work provides a practical introduction to this emerging area. We summarize the necessary background in causal inference and machine learning before discussing CFMs. Throughout, we include example code and Jupyter notebooks.