Causal Trees and Generalized Random Forests

Asked which students a school intervention helped most, a causal forest gave an answer that looked decisive. The data were the shared ACIC 2018 workshop dataset, derived from the National Study of Learning Mindsets: 10,391 student records across 76 schools. The organizers simulated those records from a model fit to the National Study data, to protect student privacy. They are not the original study records. Athey and Wager published the run in 2019, grf version 0.10.2, with the numbers left in.

Fit without accounting for school-level clustering, the forest's best-linear-prediction output reported differential.prediction 0.634163, standard error 0.132700, t = 4.7789, p = 1.786e-06. Then the same test was run cluster-robust. Same 10,391 records, same 76 schools. Differential prediction 0.321932. t = 1.0495. p = 0.294. A third run repeated the best-linear-prediction test on out-of-fold predictions from a 5-fold split, the 76 schools divided into 5 cluster-aligned estimation folds. Its heterogeneity t-statistic was 0.058.

The paper puts it in one sentence: “In other words, if we use evaluation methods that are robust to clustering, then the apparent gains from non-cluster-robust forests wash away.”

Nothing had broken. The forest did exactly what it was asked to do. What changed between t = 4.7789 and t = 0.058 was not the model. It was whether the evaluation admitted that the students arrived in 76 schools rather than one at a time. The place where a forest speaks loudest is not automatically the place where it has evidence.

A causal tree asks something different at every split than a prediction tree does. It is not hunting for a cut that makes the outcome easier to guess. It is hunting for a cut where the two sides react differently to being treated. Athey and Imbens set that machinery out in PNAS in 2016: one sample constructs the partition, a second estimates the treatment effect in each leaf.

That honest division looks wasteful and is not, and the paper prices both sides of it. The abstract states the gain: “Through a simulation study, we show that for our preferred method honest estimation results in nominal coverage for 90% confidence intervals, whereas coverage ranges between 74% and 84% for nonhonest approaches.” The body reports coverage “as low as 69%” for adaptive estimation. The cost was 7% to 22% in mean squared error of treatment effects, paid because the model is fit on a smaller sample. That is the whole trade in two numbers. Intervals that mean what they say, bought with a fifth of the precision at worst. Skip the separation and the tree finds a split because the noise there happened to be flattering. Then it hands the same flattering noise back as the effect it discovered.

One honest tree gives you a leaf. A forest gives you something better.

Grow many honest trees. Count how often each training observation lands in the same leaf as the point you care about. Those counts become weights — a neighborhood drawn fresh for every target point, instead of fixed in advance by a radius someone chose. Wager and Athey formalised the requirement in 2018: “An example of an honest tree is one where the tree is grown using one subsample, while the predictions at the leaves of the tree are estimated using a different subsample”. Then they measured the adaptive neighborhood against a fixed one. Aggregated over 500 simulation replications, their Table 1 shows the causal forest holding mean squared error at 0.02 as the dimension d rose from 2 to 30. Over the same range, 10-nearest-neighbour matching degraded from 0.21 to 0.33, and 100-NN from 0.09 to 0.13. Their abstract states the conclusion plainly: “In experiments, we find causal forests to be substantially more powerful than classical methods based on nearest-neighbor matching, especially in the presence of irrelevant covariates.”

The generalized version keeps those weights and changes what they are used for. The generalized random forests paper, in The Annals of Statistics in 2019, describes the substitution exactly: “Following the literature on local maximum likelihood estimation, our method considers a weighted set of nearby training examples; however, instead of using classical kernel weighting functions that are prone to a strong curse of dimensionality, we use an adaptive weighting function derived from a forest designed to express heterogeneity in the specified quantity of interest.” The weights then solve any quantity identified as the solution to a set of local moment equations. The same machinery is instantiated for three different parameters: non-parametric quantile regression, conditional average partial effects, and heterogeneous treatment effects via instrumental variables. It ships as the grf package for R and C++ on CRAN. The forest's job never changes. It decides who counts as nearby.

Estimating altitude by averaging nearby survey points has the same shape and the same failure. Picking which points count as nearby, separately for each location, genuinely beats a fixed grid. That is the 0.02 against 0.21 to 0.33 above. But where the surveyors never walked, no scheme for choosing neighbors will conjure a measurement. The same Table 1 shows the second half of the lesson. Against a 0.95 target, causal-forest coverage fell from 0.95 at d = 2 to 0.85 at d = 30, and in the harder design of Table 3 it fell to 0.73 at d = 8. The authors attribute that to bias rather than variance. Adapting the neighborhood improves the estimate. It does not manufacture the evidence the interval is supposed to be reporting. The causal case is stricter still. A useful neighbor is not merely a nearby unit. It has to be a nearby unit under the other treatment, and it counts only where the identification assumptions hold.

Set a prediction tree, a causal tree and a causal forest beside each other and they look like one object drawn at three sizes. Each is optimizing something the others are not. That difference decides what you are allowed to read off the output.

A prediction tree cuts wherever the cut makes the outcome easier to predict. Its leaves therefore collect people with similar outcomes, which is not the same population as people who respond similarly. A causal tree cuts where the estimated effect differs across the cut, and pays for that with the honest division Athey and Imbens costed at 7% to 22% in mean squared error. A causal forest abandons the readable diagram altogether and returns adaptive weights instead. You trade a picture you could show a stakeholder for a local estimate at every point.

Read one of them through another's objective and you will end up describing a group with high outcomes as a group that treatment helps.

The sorted heterogeneity plot is the most persuasive object a causal forest produces and among the least informative. It is a smooth curve of predicted effects with the dramatic tail on the right. It looks identical whether that tail rests on a great many treated-control comparisons or on a handful — or, as in the mindsets data, on 76 schools counted as though they were 10,391 independent units.

The audit has a published form. Chernozhukov and three co-authors set it out in Econometrica in 2025. Three named objects do the work: the Best Linear Predictor (BLP) of the CATE on the machine-learning proxy, Sorted Group Average Treatment Effects (GATES), and Classification Analysis (CLAN) of the most and least affected units.

Split-dependence they answer by repeating the split rather than arguing about it. From the abstract: “Estimation and inference are based on repeated data splitting to avoid overfitting and achieve validity. We use quantile aggregation of the results across many potential splits, in particular taking medians of p-values and medians and other quantiles of confidence intervals.” Their inference algorithm fixes the number of splits at N_S = 250 with significance level alpha = 0.05. It reports medians of estimates, of p-values and of confidence-interval endpoints across those splits.

This is not a small-data thought experiment. The application is the Haryana immunization RCT: seven districts, 140 primary health centres, 2,360 villages (915 at risk for all treatments) and 295,038 children. At baseline, 39% of children had reportedly received the measles vaccine, and only 19.4% received it before 15 months of age.

Auditing means turning the picture back into a question about evidence. In the region this plot is excited about, what did the weighting actually have to work with, and does the answer survive a different split of the data? That question breaks into a short list of checks, and the section that follows is the list. Run them in order. The last one usually settles the matter, because it asks the forest to survive its own randomness.

A variable-importance chart is not enough, and neither is the sorted curve. A policy decision needs four things: evidence that the ranking means something, that the numbers are roughly the right size, that the extremes rest on real comparisons, and that none of it moves when the forest is grown again.

On the mindsets data the output of exactly these checks splits cleanly in two, and Athey and Wager published it. The average treatment effect was large and clear: a 95% confidence interval for the ATE of 0.247 +/- 0.04. The heterogeneity the forest claimed did not survive its own calibration test. Their own reading is worth keeping: “Although our omnibus tests did not find strong evidence of treatment heterogeneity, this does not mean there is no heterogeneity present.”

The stages run in a fixed order. Cross-fit the nuisance models, grow honest trees, aggregate them into weights, estimate the local moments, then validate the structure that comes out.

Each stage has an owner in the work this lesson has been quoting. Cross-fitting keeps the nuisance estimates from being fit and used on the same rows. Honesty does the same job one level up, inside each tree. It is the stage Athey and Imbens priced at 90% coverage against 74% to 84%. Aggregation converts trees into the adaptive neighborhood Wager and Athey measured at 0.02 mean squared error, against 0.21 to 0.33 for 10-nearest-neighbour matching. The moment step is where the parameter finally gets defined. That is the step generalized random forests extended to quantile regression, conditional average partial effects and instrumental-variables treatment effects alike.

Validation is the only stage that can tell you whether any of it held. It is also the stage that quietly disappears once the plot already looks convincing. And it is the stage that moved the mindsets result from t = 4.7789 to t = 0.058 without touching a single earlier step. Trace the map and you can see which decision each assumption was attached to.

Three conditions are worth insisting on before a forest prediction is allowed to move a budget. It has to improve policy value on held-out data — not fit better, but produce better decisions on rows the forest never saw. The region it is confident about has to be supported, with both treatment alternatives genuinely present there. And it has to hold still for at least as long as the policy will run. A targeting rule that rearranges itself on the next refit is a rule about the sample rather than about the people.

The first condition is a theorem. Athey and Wager proved it in Econometrica in 2021: “Given a doubly robust estimator of the causal effect of assigning everyone to treatment, we develop an algorithm for choosing whom to treat, and establish strong guarantees for the asymptotic utilitarian regret of the resulting policy.” Maximise a doubly robust estimate of policy value over a constrained class Pi and you get a policy whose regret is bounded on the order of the square root of VC(Pi)/n. Regret here is the gap between the utility of the learned policy and the best achievable within Pi. VC(Pi) is the Vapnik-Chervonenkis dimension of the policy class, and n is the sample size.

That bound is also the argument for coarsening, because it is explicitly a function of how complicated the rule is allowed to be. When the estimates wobble but the broad ranking does not, shrink the class. Turn a continuous CATE surface into two or three wide groups, or into a conservative rule applied above a threshold. That lowers VC(Pi) on the same n and therefore tightens the guarantee. It discards precision the forest never really had and keeps the part that replicates.

What not to do is take the most dramatic number and act on it. On the mindsets data that number carried p = 1.786e-06 until the evaluation respected the 76 schools. After that the same quantity read 0.321932 with p = 0.294. Without enough treated-control comparisons behind it, and without an independent check that the effect exists at all, the number is a description of the seed.