Towards Scalable Meta-Learning of near-optimal Interpretable Models via Synthetic Model Generations

AI summary

Overview

• Research area: Meta-learning for interpretable machine learning, specifically synthetic data generation to pre-train transformer models that output near-optimal decision trees for tabular data. The paper appears in the "Generative AI in Finance" workshop track.
• Technical level: Intermediate. Familiarity with decision trees, pre-training, and terms like class imbalance and label noise helps, but the core ideas are explained in plain terms.
• Scope: The paper presents a four-step pipeline that uses Structural Causal Models (SCMs) to synthesize tabular datasets together with their near-optimal decision trees, then shows that a MetaTree transformer pre-trained only on this synthetic data matches the accuracy of the original MetaTree pre-trained on hand-curated real-world data.

What This Paper Is About

Training transformer models to produce interpretable decision trees requires large amounts of high-quality training data, and generating the optimal trees for that data is extremely expensive. The authors build a synthetic data pipeline that creates both the datasets and their near-optimal trees at the same time, so that meta-learning of decision trees can scale without depending on curated real-world datasets or costly optimal tree solvers.

Key Contributions

1. A Structural Causal Model (SCM)-based synthetic data generation workflow that produces labeled tabular datasets with causal feature-label relationships for pre-training MetaTree.
2. A label reassignment and noising scheme: CART trees are fit to the synthetic data, original labels are replaced by the CART tree predictions, and 5% label noise is added to create datasets intrinsically aligned with near-optimal trees without running an optimal tree solver.
3. Quality filters — a normalized class imbalance metric (from the Penn Machine Learning Benchmarks) and a CART accuracy filter — that remove datasets unsuited to decision tree construction.
4. Empirical evidence that MetaTree pre-trained purely on synthetic data performs comparably to the original MetaTree pre-trained on hand-curated real-world data when benchmarked against CART and GOSDT.

Main Findings

• Constant-time target generation: GOSDT's training time rises to nearly 200 seconds at a tree depth of 6, while the proposed method stays constant at under 1 second regardless of depth. GOSDT also grows exponentially with the number of binary features, whereas the proposed method remains consistently low.
• Synthetic MetaTree is competitive: Benchmarked on 91 datasets from the original MetaTree paper over 10 trials per dataset, MetaTree trained on synthetic data reached 0.6956 ± 0.0061 accuracy with 30 trees, close to the original MetaTree's 0.7047 ± 0.0059.
• CART leads at higher tree counts: CART achieved the highest accuracy in the benchmark, 0.7053 ± 0.0060 with 30 trees. GOSDT performed well with a single tree (0.6524 ± 0.0072 versus 0.6508 ± 0.0068 for original MetaTree) but fell behind as the number of trees increased.
• Data scaling improves accuracy: Pre-training datasets ranged from 100,000 to 20 million data points. Accuracy rose with dataset size; the 20-million-point model achieved the highest accuracy at approximately 0.675, followed by the 10 million, 5 million, and 1 million models. The 100,000-point model degraded after roughly 40,000 training steps, suggesting overfitting, while models trained on 5 million points and above stayed stable.
• Moderate label noise helps, excessive noise hurts: Noise levels tested were 10%, 20%, 30%, 40%, and 50%. Models at 10%, 20%, and 30% peaked at approximately 0.68 within the first 25,000 steps. The 40% model peaked at about 0.66 after around 40,000 steps, and the 50% model failed to learn effectively, fluctuating around 0.56 across 100,000 training steps.
• A model-complexity sweet spot exists: With an identical 100,000-example pre-training set, models from 1 to 12 attention layers were compared. The 1-layer model performed worst, the 2-layer model was best and most stable at approximately 0.665 peak accuracy, and models with 4, 8, or 12 layers showed diminishing returns. The 12-layer model has 6 times the depth of the optimal 2-layer model, and the authors link the result to the Chinchilla scaling law.
• More uniform data distribution: Synthetic datasets show a more uniform class imbalance distribution over the [0, 0.3] range than the hand-curated MetaTree benchmarks, with class counts declining systematically as the number of classes grows, a consequence of the quality filters.

Methodology in Plain English

The authors avoid the two usual bottlenecks in this line of work: scarce curated data, and expensive solvers for certifiably optimal trees. Their pipeline has four steps.

1. Sample synthetic features and labels from a Structural Causal Model, so features and labels have genuine causal relationships rather than arbitrary correlations.
2. Fit a CART tree to each synthetic dataset to establish a decision tree baseline.
3. Apply quality filters: discard datasets that are badly imbalanced or poorly separable. The stated thresholds are a normalized class imbalance metric (I = K * sum_i (n_i/N - 1/K)^2, normalized by the worst case k - 1) kept below 0.3, which targets no majority class above 75% of samples, and a CART accuracy requirement above 70%. The paper also gives an example that datasets with over 90% of samples in one class are discarded.
4. Relabel the data using the CART tree's predictions and add 5% label noise, producing datasets that are naturally aligned with near-optimal trees.

MetaTree is then pre-trained on these synthetic datasets with the corresponding trees as targets, and at inference time it predicts a tree for an unseen real-world dataset. Pre-training used 20 million examples — twice the 10 million used in the original MetaTree paper — on a single server node with eight NVIDIA A100 GPUs on the GenAI ultra cluster, completing 100,000 epochs in approximately 32 hours. SCM data generation used the TabPFN v1 codebase with example scripts, run on a node with 16 CPU cores and 16 GB of RAM, with multiple parallel processes seeded uniquely. Evaluation used the 91 holdout datasets from the MetaTree paper, sampled N times to give 91 × N splits, with identical training and test splits and matched hyperparameters such as maximum tree depth and number of trees across all compared models.

Why This Matters

The work shows that a transformer can learn to produce interpretable, near-optimal decision trees without ever seeing curated real-world training data and without paying the combinatorial cost of optimal tree solvers. That removes a practical barrier for building interpretable models in settings where explanations are mandatory.

Real-world applications:

• Credit scoring and loan decisioning, the paper's own example ("Give A Loan?"), where decision trees are already standard and regulators expect transparency.
• Other high-stakes financial decisions such as risk assessment and compliance-driven model review, the domain the authors explicitly target.
• Healthcare decision support, cited among the fields where opaque deep learning models are hard to adopt.
• Rapid architecture and hyperparameter iteration for tabular machine learning teams, since synthetic data removes data-curation cost and data variability as confounding factors.

For industry, the appeal is economic: constant-time data generation under 1 second regardless of depth or feature count makes large-scale pre-training of interpretable models feasible on modest compute relative to optimal tree solvers.

Future Directions

• Scaling the training set further: the authors note that effectively training their 12-layer architecture would require a proportionally larger dataset than the 100,000 examples used, following the Chinchilla-style scaling argument.
• Revisiting the noise schedule: the pipeline uses 5% label noise, while the noise study found the best results at 10%, 20%, and 30% and degradation at 40% and 50%. Whether the pipeline's 5% is optimal is left open.
• Extending beyond the 91 holdout datasets and binary-style tabular benchmarks toward more classes, regression targets, or non-tabular structured problems.
• Applying the same synthetic pre-training idea to other interpretable model families beyond decision trees, and validating on live high-stakes financial or clinical decision tasks.

Target Audience

Researchers and practitioners in interpretable machine learning, meta-learning, and tabular deep learning who care about generating training data at scale; applied scientists and model risk teams in finance and healthcare who need transparent models but cannot rely on hand-curated datasets or expensive optimal tree solvers; and machine learning engineers interested in scaling laws for pre-training on synthetic, causally grounded data.

Authors’ abstract

Decision trees are widely used in high-stakes fields like finance and healthcare due to their interpretability. This work introduces an efficient, scalable method for generating synthetic pre-training data to enable meta-learning of decision trees. Our approach samples near-optimal decision trees synthetically, creating large-scale, realistic datasets. Using the MetaTree transformer architecture, we demonstrate that this method achieves performance comparable to pre-training on real-world data or with computationally expensive optimal decision trees. This strategy significantly reduces computational costs, enhances data generation flexibility, and paves the way for scalable and efficient meta-learning of interpretable decision tree models.