Out-of-Bag Evaluation and Forest Interpretation

Each training row is absent from some bootstrap samples, and it can be predicted using only the trees that did not train on it. How many rows is that? Not a vague "roughly some rows". The figure sits on Breiman and Cutler's own Random Forests page: “About one-third of the cases are left out of the bootstrap sample and not used in the construction of the kth tree.” Liaw and Wiener put the same figure at around 36% in R News in 2002. Two independent write-ups, one number.

That one-third is the whole mechanism. Bootstrap each tree and about a third of the rows are omitted from that resample. Record which examples are out of bag for that tree. Predict each row by aggregating only the trees that excluded it. Compute out-of-bag metrics from those predictions, and you have performance evidence without carving a separate internal holdout out of the training data.

Preserve a final, untouched, deployment-like test set anyway. The out-of-bag estimate is built from the same rows the forest was fitted on. The next section shows what that costs when the rows are not the independent draws the resampling assumes.

Out-of-bag estimates can support tree-count monitoring, feature experiments, and model comparison. What they assume is that the rows are ordinary and independent — that leaving a row out of a bootstrap sample leaves out everything that makes that row easy to predict. They become less trustworthy when groups, time, space, preprocessing leakage, repeated tuning, or nonuniform weights violate that logic.

There is a number for the gap. A random forest trained on a central-African forest inventory of 11.8 million trees reported R² = 0.53 and an RMSPE of 56.5 Mg ha⁻¹ under a random 10-fold cross-validation. The same fitted model under a spatial 44-fold cross-validation reported R² = 0.14 and an RMSPE of 77.5 Mg ha⁻¹. A null model with no predictors at all had an RMSPE of 82 Mg ha⁻¹. Ploton and colleagues published that in Nature Communications in 2020, and their abstract states the consequence plainly: “A standard nonspatial validation method suggests that the model predicts more than half of the forest biomass variation, while spatial validation methods accounting for SAC reveal quasi-null predictive power.” Row-random resampling said half the variation. Respecting the deployment boundary said the model was barely beating a model with no predictors at all.

The direction and rough size of that bias replicate. Andy Stock, choosing blocks for spatial cross-validation in a marine remote sensing case study in 2025, reports that “Depending on the model and error measure, 10-fold cross-validation underestimated prediction errors by 5% (RMSE of MBR) to 54% (APD of RFXY).”

Out-of-bag resampling is row-random by construction, so it inherits exactly this failure. Use group-aware, temporal, or spatial evaluation when those boundaries define deployment. Read the OOB number as an internal diagnostic, not as the claim you ship.

No importance score reveals a feature's causal contribution. Not by itself. Each of the four methods below is a different experiment on the fitted model, and three of the four have a documented failure mode with a source behind it.

Impurity importance adds split improvements attributed to each feature across trees. It is fast, it is built in, and it is a training-distribution quantity. It is also biased by the number of categories and the scale of measurement of a predictor, which Strobl and colleagues showed in 2007. Permutation importance measures score loss after disrupting one feature on a chosen evaluation set. It reflects model reliance for one metric. It can understate correlated substitutes, which Strobl and colleagues showed again in 2008. And the permutation itself can manufacture rows the model has never seen, which Hooker and colleagues showed in 2021. Drop-column comparison refits the pipeline without a feature or a group. It is expensive, it is informative, and it lets substitutes reorganize. It is the shape of experiment those same authors recommend — still not a causal intervention. Local attribution allocates one prediction among encoded features under an explanation method. Useful for case review, dependent on the background distribution, sensitive to feature dependence. It explains the fitted model, not the world.

A forest can score 0.97 and report, on its own chart, that none of its features matter. Scikit-learn ships the demonstration as a worked example on the Breast cancer Wisconsin (diagnostic) data. A RandomForestClassifier reaches a “Baseline accuracy on test data: 0.97”. The permutation importances, computed on the training set, then report that “permuting a feature drops the accuracy by at most 0.012, which would suggest that none of the features are important”. A model scoring that well appears, by its own importance chart, to be using nothing.

The machinery behind that result has a name — two, in fact. Strobl and colleagues set them out in BMC Bioinformatics in 2008: “We identify two mechanisms responsible for this finding: (i) A preference for the selection of correlated predictors in the tree building process and (ii) an additional advantage for correlated predictor variables induced by the unconditional permutation scheme that is employed in the computation of the variable importance measure.”

A partial dependence curve replaces one feature with selected values and averages predictions over other observed columns. It can summarize a global model pattern. It can also evaluate combinations that never occur. Hooker and colleagues stated the mechanism in Statistics and Computing in 2021: “In particular, we describe how breaking dependencies between features in hold-out data places undue emphasis on sparse regions of the feature space by forcing the original model to extrapolate to regions where there is little to no data.” Scikit-learn's own user guide says the same thing about the plots it draws — “in the case of correlated features, we will create absurd data points to compute the PDP/ICE”.

Individual conditional expectation curves reveal heterogeneity that the average can hide. That is not a folk observation. It is why the method exists. Goldstein and colleagues introduced ICE plots in 2015 with the reason in their abstract: “In the presence of substantial interaction effects, the partial response relationship can be heterogeneous. Thus, an average curve, such as the PDP, can obfuscate the complexity of the modeled relationship.” Scikit-learn restates it: “While the PDPs are good at showing the average effect of the target features, they can obscure a heterogeneous relationship created by interactions. When interactions are present the ICE plot will provide many more insights.”

ICE fixes the averaging problem and inherits the support problem. Both curves are drawn by substituting values into rows, so both can leave the observed data manifold, and the scikit-learn sentence above covers PDP and ICE alike. Restrict the plotted range to combinations that actually occur, or say in the caption that the curve extends past them.

Use several views and ask whether they tell a coherent, support-aware story. Establish held-out behavior first: metrics, calibration, slices, and support. Choose the resampling boundary that matches deployment, because the difference between R² = 0.53 and R² = 0.14 in Ploton and colleagues' forest was the boundary, not the model. Group correlated features into concepts that should be perturbed or removed together, or the chart will report 0.012 for columns the model depends on. Compare importance methods and read their disagreements as information about dependence rather than as noise. Inspect local cases: decision paths, neighbors in leaf space, attribution stability. Plot dependence curves only over plausible ranges and combinations, since breaking dependencies forces the model to extrapolate. Then test interventions separately. Do not translate predictive reliance into causal policy without evidence built for that question.

Two rows can be considered similar when they land in the same leaves across many trees. That similarity has a published definition rather than an intuition. Liaw and Wiener state it in R News: “The (i, j) element of the proximity matrix produced by randomForest is the fraction of trees in which elements i and j fall in the same terminal node.” Breiman and Cutler give the arithmetic on their Random Forests page: “If two cases occupy the same terminal node, their proximity is increased by one. At the end of the run, the proximities are normalized by dividing by the number of trees.” Count shared leaves, divide by the number of trees. That is the distance.

Its own authors claim it for missing-data handling, outlier detection, and low-dimensional views of the data, and it can support diagnostics, retrieval, or anomaly analysis on that basis.

Every one of those counts comes from the forest's own splits. So the proximity inherits the forest's biases, including the split-selection bias Strobl and colleagues described. It is not an independent ground-truth distance.

Performance asks whether predictions help, and which resampling boundary was used to ask. Ploton and colleagues' forest scored R² = 0.53 or R² = 0.14 depending on that single choice. Reliance asks which inputs the fitted model uses, and that answer moves with the perturbation: the same forest can report 0.97 accuracy and a 0.012 maximum permutation drop. Effect asks what would happen if the world changed. Neither of the first two answers it.

Keep them in separate sections so an importance chart cannot silently become a policy claim. The cost of not doing so is on the record. A cost-based risk score with real predictive performance. A share of Black patients receiving additional help that a corrected target would move from 17.7% to 46.5%. A letter from two New York State regulators to the vendor's chief executive, dated 25 October 2019.