Double/Debiased Machine Learning

The estimate came out far from the truth on data that had nothing wrong with it. That is how the 2018 paper opens — with the failure rather than the cure. It draws the failure twice.

Figure 1 is a partially linear simulation. It has n = 500 observations and p = 20 covariates. A random forest holds everything nobody could hold in their head, and a naive, non-orthogonal plug-in estimator reads the treatment effect off it. The true nuisance function in that design is smooth and low-dimensional. That is the easy case, the one where flexible machinery should have nothing to fear. The estimate still lands far off the true value. Nothing about the data was adversarial. The regularization that made the forest fit well was enough on its own to shift the single number the study exists to produce.

Figure 2 changes one thing at a time on the same problem. The score is now the orthogonal one, and the nuisance functions are fit on the full sample. The estimator is still badly biased. Then the identical data is run again with two-fold cross-fitting, so that every prediction entering the score comes from a model fit outside the fold it is being scored on. The bias goes, and it goes at roughly the same spread. Of that panel the authors write, plainly: “Here, we see that the use of sample-splitting has completely eliminated the bias induced by overfitting.”

Two figures, two diseases. The first is regularization in the nuisance models leaking into the target. The second is those models grading their own homework. Neither figure touches a third thing. A variable neither model contains, pushing both who got treated and how the outcome turned out, survives every fold discipline in the paper. What you hold then is an honest measurement of the wrong quantity.

Those are the two diseases in Figures 1 and 2, and double/debiased machine learning is a cure for exactly those two and no others. The paper's summary states the whole construction in one sentence: “We show that the impact of regularization bias and overfitting on estimation of the parameter of interest θ0 can be removed by using two simple, yet critical, ingredients: (1) using Neyman-orthogonal moments/scores that have reduced sensitivity with respect to nuisance parameters to estimate θ0, and (2) making use of cross-fitting which provides an efficient form of data-splitting.”

Two ingredients, and it is worth seeing why either alone is not enough. The first is the choice of score, the equation whose solution is the estimate. Many equations recover the target when the nuisance models are perfect. DML insists on one whose sensitivity to nuisance error is reduced. Nudge the outcome model or the treatment model slightly off and, to first order, the target does not move with it. Errors in the parts nobody reports stop travelling straight into the part everybody quotes. That is the answer to Figure 1.

That property holds around the truth. It says nothing about a nuisance model that is wrong because it memorised the rows it is now being scored on. That is why the orthogonal estimator in Figure 2 is still biased when the nuisances are fit on the full sample, and it is the second ingredient's job. Cross-fitting fits the nuisances outside a fold and evaluates the score inside it. Every prediction entering the estimate is then a prediction about data the fitting never saw.

Separating fitting from evaluating is not free, and the price has been measured. An honest rule buys coverage and pays for it in precision. Athey and Imbens reported both halves in PNAS in 2016: “Through a simulation study, we show that for our preferred method honest estimation results in nominal coverage for 90% confidence intervals, whereas coverage ranges between 74% and 84% for nonhonest approaches.” Honesty in their sense costs between 7% and 22% in mean squared error of the treatment effects. That is the trade in numbers. You give up some precision on the point estimate. The interval you get back means what it says.

What the two ingredients ask in return is a short list worth reciting before any run. The causal question has to be identified from the data you actually have. Treated and untreated units have to overlap enough to be compared at all. The score has to be the right score for the target you are claiming. The nuisance estimates have to converge fast enough for the argument to hold. Miss any of these and the machinery still runs, still prints an interval, and still means nothing. Handing a model every available column does not turn those columns into valid controls.

A workflow is just that list in working order. Define the score, which means naming the target and writing the Neyman-orthogonal moment equation for it. Partition the folds, which means respecting clusters, time, and whatever counts here as an independent unit. Fit the nuisances outside each fold. Evaluate the moments from out-of-fold residuals and score components. Then aggregate and report the target, its standard error, and some evidence of how much either depends on how you happened to draw the folds.

Nothing in the argument so far was specific to a single effect on a single outcome. Wherever you can write a Neyman-orthogonal score for a quantity, you can run the same discipline of fitting outside and evaluating inside around it. The same list of prerequisites comes along unchanged.

It helps to keep two columns in mind, because the paper's own application has trouble in each and only one of them gets solved.

On one side sits everything about estimation. How the nuisance functions were fit, whether their errors leak into the target, whether the model that made a prediction had already seen the row it is predicting. The paper's 401(k) application shows how much that side can move. The data is the 1991 Survey of Income and Program Participation, the treatment is 401(k) eligibility, the outcome is net financial assets. With no controls at all the effect is enormous: “Turning to the results, it is first worth noting that the estimated ATE of 401(k) eligibility on net financial assets is $19,559 with an estimated standard error of 1413 when no control variables are used.” The DML2 estimates in Table 2 all land between $6,830 and $9,247. Those were computed over 100 sample partitions, with 2-fold and 5-fold cross-fitting, across six different nuisance learners: lasso, regression tree, forest, boosting, neural net and ensemble. Twelve combinations of learner and fold count, one narrow band. That is what a solved estimation problem looks like.

On the other side sits everything about design. Whether treatment is as good as randomly assigned given the covariates you have. Whether the groups overlap enough to compare. Whether the quantity you named is the quantity a decision actually needs. None of that moved across those twelve runs, and none of it can, because it is a claim about how the data came to exist and the estimator only ever sees the data. Worse, the reflex that seems to help it hurts something else. A 2021 paper in the Journal of Econometrics opens on exactly that point: “Researchers often argue that unconfoundedness is more plausible when more covariates are included in the analysis. Less discussed is the fact that covariate overlap is more difficult to satisfy in this setting.” Overlap is a restriction that tightens as covariates are added. D'Amour and his four co-authors then make it checkable rather than rhetorical. Under strict overlap with bound η, no classifier of treated against control covariate distributions can have accuracy above 1 - η. Train that classifier and you have an upper-bound test on the assumption the whole estimate rests on.

The method solves an estimation problem sitting inside a larger causal design. The trap is mistaking the noise it removes for the doubt it leaves standing.

An audit goes wrong the moment an implementation is handed over as a single function call, because a bundle can only pass or fail as a bundle. Ask for it in pieces, and ask in this order.

The score first — which target does it estimate, and is the moment equation the orthogonal one for that target or a convenient approximation of it. The nuisances next — what was fit, on which columns, and how well it predicted out of fold. Then the folds — what defines a unit here, whether the split respects that definition, and whether time order was preserved where it matters. Last the causal assumptions, written out as sentences somebody is willing to defend, not buried in an appendix.

That is not an idiosyncratic checklist. It is the shape of a serious implementation. DoubleML, the reference package for R, states its anatomy in one line: “The double machine learning framework consists of three key ingredients: Neyman orthogonality, high-quality machine learning estimation and sample splitting.” The package exposes them as separate arguments — score, learners, dml_procedure, and n_folds together with n_rep — and defaults to n_folds = 5. Its authors, writing in the Journal of Statistical Software in 2024, report that K = 4 or K = 5 works better than K = 2, and they recommend repeating the estimation over multiple sample splits. That is where the last line of the workflow comes from. n_rep is the published way to show how much the answer depends on the draw.

Kept apart, each piece can fail where you can see it. Bundled, a failure in the fourth hides comfortably behind good numbers from the first three.

Reach for DML when the nuisance functions really are complicated — many covariates, relationships nobody would have guessed — while the thing you must report stays a single number, and when that number needs an interval somebody is going to act on. Those conditions together make the case.

That second condition deserves more suspicion than it usually gets, because flexibility and honest intervals are not the same achievement. A competition run at the 2016 Atlantic Causal Inference Conference tested exactly this. Thirty methods were submitted across two tracks, 15 do-it-yourself and 15 black-box. The black-box submissions, together with the post-competition methods, were evaluated on 7,700 datasets: 77 scenarios by 100 replications, 58 covariates each. The results appeared in Statistical Science in 2019. On overall performance the verdict was clear, and it favours flexibility: “The most consistent conclusion was that methods that flexibly model the response surface perform better overall than methods that fail to do so.” On intervals it was not. Every interval was intended to have 95% coverage. The last nine methods ranked by coverage all fell below 75%. Point accuracy and interval honesty were separate results in the same table.

So test the case instead of assuming it. A DML estimate should beat a transparent baseline. It should also hold still when you swap the learner library for another reasonable one — the way the 401(k) estimates held inside one band across six learners.

Where the design already did the work — a clean randomized comparison, a handful of covariates, a model you can write on a single line — a simpler estimator is easier to defend and frequently loses nothing in efficiency. The machinery exists to absorb nuisance complexity. There is nothing in it that repairs a design.