Future Horizons

The Causal Turn: Why the Next Frontier of AI Is Learning to Ask "Why"

An in-depth exploration of causal AI — from Judea Pearl's Ladder of Causation to counterfactual fairness, causal representation learning, and the future of medicine.

Gabriele Masetti · 2026-08-24

The Ceiling Above Pattern Recognition

Every major leap in modern AI — from ImageNet classifiers to large language models that draft contracts and debug code — has been a triumph of the same underlying trick: finding statistical regularities in enormous datasets and exploiting them to predict what comes next. That trick has been extraordinarily productive. It has also, according to a growing body of research anchored in the work of UCLA computer scientist Judea Pearl, run into a wall that no amount of additional data or parameters can breach on its own.

The wall is structural, not a matter of scale: pure association, however sophisticated, cannot answer questions about intervention or counterfactual alternatives, because those questions live at a different level of a hierarchy that statistics alone cannot climb.

That gap is the essence of the "causal turn" now underway in parts of the AI research community — a shift from asking "what is correlated with what" to asking "what would happen if we acted differently" and "what would have happened had things been otherwise."

The thesis of this piece is specific and falsifiable: the next meaningful capability gain in AI systems, especially any system aspiring toward general or superintelligent reasoning, depends less on scaling today's associative architectures than on giving machines an explicit formal grammar for cause and effect — the kind Pearl and collaborators have spent four decades building, and that fields from epidemiology to economics to algorithmic fairness are already using in production.

Without it, systems that look fluent can still be causally illiterate in ways that matter enormously for trust, safety, and generalization.

Pearl's Ladder of Causation

Pearl's central conceptual device, popularized in his 2018 book The Book of Why, co-written with mathematician-turned-science-writer Dana Mackenzie, is what he calls the ladder of causation, arranged in three rungs of increasing cognitive demand.

The first rung is association — the level of "what if I see." That is the domain of conditional probability: observing that patients who take a drug tend to recover faster, or that a word tends to follow another word in text. Nearly all of contemporary machine learning, including the transformer architectures behind today's large language models, operates at this rung. It is powerful because vast amounts of observational data are cheap to collect and because many practical prediction tasks genuinely only require finding stable patterns.

The second rung is intervention — "what if I do." That is the level of the question "if I make the intervention, how will the outcome change?" Answering it requires more than observing a correlation; it requires knowing what happens when an external actor forces a variable to a particular value, severing whatever natural causes would otherwise have determined it.

A classic illustration is Simpson's paradox: a treatment can appear harmful when data are pooled across groups and beneficial within every subgroup, depending entirely on how confounding variables were generated — a distinction invisible to conditional probability alone but resolvable once one specifies which variables were manipulated versus merely observed.

The third and highest rung is counterfactuals — "what if I had done." That is the level of retrospective, imagined worlds: what would have happened to a specific patient who died, had she not been given the drug? Counterfactual reasoning requires positing two parallel realities, the one that occurred and the one that did not, and is consequently the hardest to formalize and the most demanding of assumptions.

Pearl's framework treats each rung as requiring strictly more information or structural assumptions than the one below it — you cannot deduce the answer to a rung-two or rung-three question from rung-one data alone, no matter how much of it you have.

The Formal Machinery: Do-Calculus

Pearl did not merely describe this hierarchy philosophically; he built mathematical tools to operate within it. Working through directed acyclic graphs called structural causal models, he introduced in the mid-1990s a set of three symbolic inference rules known as do-calculus. The "do" refers to the do-operator, written do(x), which represents the act of externally setting a variable to a value — as opposed to conditioning on having observed that value, which is what ordinary probability theory does.

The do-calculus rules specify exactly when and how a query expressed with the do-operator (an interventional question) can be rewritten in terms of ordinary observational quantities that can, in principle, be estimated from non-experimental data. That rewriting matters enormously in cases where running a genuine randomized experiment is unethical, impossible, or prohibitively expensive — you cannot randomly assign people to smoke, for instance, but under the right graphical assumptions you may still be able to identify the causal effect of smoking on health from observational data alone.

The completeness of do-calculus — the proof that if the three rules cannot eliminate the do-operator from an expression, the causal effect genuinely cannot be identified from observational data under the assumed graph — was established in 2006, independently, by Yimin Huang and Marco Valtorta and by Ilya Shpitser and Judea Pearl.

That result turned do-calculus from a useful heuristic into a decision procedure: given a causal graph and a query, there is now a systematic way to determine whether the question is answerable at all from the available data, before ever touching a dataset.

Why "Correlation Is Not Causation" Undersells the Problem

The folk warning that correlation does not imply causation is often treated as a slogan rather than a technical claim, and that undersells how deep the gap actually is. The problem is not simply that spurious correlations exist and need to be filtered out with more careful statistics.

It is that the object a system needs — an intervention or counterfactual quantity — is often not a function of the observational data distribution at all, no matter how it is manipulated, unless the underlying causal structure (which variables cause which, and which are confounded by unobserved common causes) is specified separately.

A model trained purely to predict P(Y | X) can be an excellent predictor while being formally silent, even wrong, about P(Y | do(X)) — the effect of actually changing X. Pattern learning optimizes for the former; decision-making, policy design, and scientific explanation require the latter.

Precisely for that reason, fields with high stakes for getting causality wrong — medicine, economics, public policy — have built entire methodological traditions around it rather than relying on prediction accuracy alone.

Why Large Language Models Get Stuck on the First Rung

Large language models are trained overwhelmingly on unstructured text that encodes correlations and human descriptions of causal relationships, but not the underlying structural machinery that generated those relationships, and not experimental manipulations of the world. Several recent research efforts have tried to directly measure the consequence of this gap.

One widely cited 2023 benchmark, Corr2Cause, constructed a large-scale test — over 200,000 examples — in which a model is given a set of purely correlational statements about a group of variables (which variables are statistically associated with which, which are independent) and asked to determine what causal relationships, if any, are logically entailed.

The results were stark: the best models of the time such as GPT-4 scored only marginally above a random-guessing baseline, with GPT-4 achieving an F1 score around 29 against roughly 20 for uniform random guessing on the benchmark's evaluation. In other words, when the surface-level linguistic cues that models can pattern-match against are stripped away and only the pure logical structure of causal inference remains, performance collapses toward chance.

Reasoning-trained models have since moved that number. Kacper Kadziolka and Saber Salehkaleybar, in a paper posted to arXiv on 31 July 2025, measured OpenAI's o3-mini at an F1 of 66.28 on Corr2Cause from a single prompt, against 33.38 for the best fine-tuned conventional baseline, and lifted it to 83.83 with a four-stage in-context pipeline; DeepSeek-R1 went from 64.57 to 79.83 the same way. Their own reading is that the structured decomposition, rather than scale, is what does the work.

Related work on causal reasoning surveys has converged on similar diagnoses: language models tend to reproduce causal-sounding language patterns memorized from training text rather than performing the structural inference the questions actually require, they generalize poorly outside the specific phrasing and variable names seen during training, and they particularly struggle when causal relationships are implicit rather than explicitly stated.

None of this means language models are useless for reasoning tasks that touch on causality — they can be effective at retrieving and articulating causal claims that humans have already written down. It means they are not, by virtue of scale alone, performing the graph-based structural inference that do-calculus formalizes.

The Tools Already in Production

The response to this gap has not been purely theoretical. A parallel applied discipline, causal machine learning, has matured into working software and standard practice in several fields.

Microsoft Research built and released DoWhy, an open-source Python library whose name is a direct nod to Pearl's do-calculus. DoWhy structures any causal analysis into four explicit steps: modeling the problem using a causal graph and stated structural assumptions, identifying whether the desired causal effect is even estimable given that model, estimating the effect using a chosen statistical method, and — distinctively — refuting the estimate through sensitivity analyses and robustness checks that actively try to break the assumed model.

That last step is the library's signature contribution: rather than treating a causal estimate as a finished number, DoWhy is built to make the fragility of causal assumptions visible rather than hidden. The project has since migrated to an independent open-source governance structure under the PyWhy organization, reflecting how far causal ML tooling has moved beyond a single company's research agenda.

In economics, Stanford's Susan Athey, working with Guido Imbens and later with Stefan Wager, adapted the random forest algorithm — originally built for prediction — into causal forests, designed to estimate not just whether an intervention works on average but which subpopulations it helps, hurts, or leaves unaffected. Heterogeneous treatment effect estimation has become foundational in applied econometrics precisely because policymakers rarely need to know only an average effect; they need to know for whom a minimum-wage change, a job-training program, or a pricing policy will help or harm.

In medicine, Harvard epidemiologist Miguel Hernán has championed target trial emulation: rather than treating an observational dataset as a grab-bag to mine for correlations, researchers first specify the protocol of the hypothetical randomized trial they wish they could run — eligibility criteria, treatment strategies, follow-up, and outcomes — and then use observational data to emulate that trial as faithfully as possible.

The method is explicitly designed to avoid errors like immortal time bias and misaligned start times that have historically made observational medical studies produce causally misleading conclusions even when their statistical associations were perfectly real. Notably, Hernán and others are careful to note that target trial emulation does not solve every problem — unmeasured confounding and incomplete data remain real limits — which is itself consistent with Pearl's point that climbing the ladder requires stronger assumptions, not just cleverer statistics.

Counterfactual Fairness: Causality as an Ethical Instrument

Perhaps the sharpest illustration of why the causal hierarchy matters practically, not just philosophically, comes from algorithmic fairness. In 2017, Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva introduced counterfactual fairness at the Neural Information Processing Systems conference, proposing a definition of fair prediction grounded explicitly in Pearl's counterfactual rung rather than in statistical parity between groups.

Their criterion holds that a decision about an individual is fair only if it would have been the same in a counterfactual world where that individual had belonged to a different demographic group, with all the downstream causal consequences of that difference properly propagated through a specified causal model — not merely with the demographic label itself changed while everything correlated with it stays fixed.

The authors demonstrated the framework on predicting success in law school, showing that a model can look fair by conventional correlational metrics while still encoding causal pathways through which protected attributes influence outcomes indirectly.

The point of the paper is not that counterfactual fairness is a solved problem — it depends on a causal model whose assumptions cannot be read off from data alone — but that fairness itself is, in an important sense, a rung-three question, and treating it with rung-one tools quietly ports in a model's own biased assumptions about which correlations are permissible to use.

Why the Causal Turn Matters for the AGI Debate

The relevance of all this to discussions of artificial general intelligence and superintelligence is not incidental. Pearl has argued, across his research and in The Book of Why, that a system confined to associations cannot originate genuinely novel explanations, plan robustly under interventions it has never observed, or reason about responsibility, blame, and imagined alternatives — capacities that are arguably constitutive of what people mean by general intelligence, not decorative extras layered on top of it.

A system that only knows what has co-occurred in its training distribution has no principled way to predict what happens under an action it has never seen taken, which is exactly the situation any agent deployed into a changing world, or asked to plan a novel intervention, will constantly face. Robust out-of-distribution generalization, scientific discovery, and long-horizon planning under real uncertainty all live, in Pearl's framework, above the first rung.

The current wave of interest — from the Corr2Cause-style benchmarks stress-testing language models, to research efforts explicitly combining large language models with causal graphs and structural priors, to production causal ML libraries like DoWhy being adopted alongside predictive AI pipelines in industry — should therefore be read as more than a niche methodological subfield catching up to deep learning's momentum.

It is a bet that the missing ingredient in current systems is not more of the same kind of data, but a different kind of formal structure: an explicit representation of what causes what, disciplined by the same do-calculus machinery that lets epidemiologists and economists tell the difference between what merely follows from what and what would actually change if the world were rearranged.

Whether hybrid architectures that graft causal reasoning onto large-scale pattern learning succeed is an open empirical question, not a settled one — but the diagnosis of what is missing, unlike the marketing claims that often surround AI progress, rests on genuinely well-established mathematics and three decades of applied results across medicine, economics, and fairness research.