iLTM: Integrated Large Tabular Model

AI summary

Overview

Research area: Machine learning for tabular data — large-scale pre-training, meta-learning, and hybrid neural/tree model architectures.

Technical level: Intermediate. The paper assumes familiarity with gradient-boosted decision trees (GBDTs), multilayer perceptrons, and the basic idea of foundation models, but its core argument is accessible to anyone who follows applied ML.

Scope: A single paper introducing iLTM, a pre-trained tabular foundation model that fuses GBDT-derived embeddings, a meta-trained hypernetwork, retrieval-augmented prediction, and MLPs into one architecture, then evaluates it on classification and regression benchmarks.

What This Paper Is About

Deep learning has transformed text, vision, and speech, but tabular data remains dominated by gradient-boosted decision trees because no single neural approach wins consistently across the huge variety of tabular tasks. The authors ask whether a single pre-trained model can combine the inductive biases of tree ensembles with the flexibility of neural networks, so that new tabular tasks require little or no task-specific tuning.

Key Contributions

1. A novel neural-tree hybrid architecture. iLTM integrates GBDT leaf embeddings, a dimensionality-agnostic representation pipeline, a meta-trained hypernetwork that generates the main network's weights, and a retrieval module into one model (Sections 3.1–3.3).

2. Large-scale meta-training on real tabular data. The hypernetwork is pre-trained on 1,806 real-world classification datasets drawn from roughly 5,000 OpenML candidates after strict de-duplication against evaluation benchmarks (Section 4.1).

3. Cross-task transfer from classification to regression. The same meta-trained weights, pre-trained only on classification, transfer to regression targets after light fine-tuning, achieving the best average rank on 18 public regression datasets (Section 5.3, Figure 4).

4. Strong, tuning-light performance. iLTM matches or beats well-tuned GBDTs and leading deep tabular models on the TabZilla Hard benchmark, high-dimensional biomedical datasets, and regression, while requiring less per-task hyperparameter search. Code and weights are released open-source.

Main Findings

• Best mean AUC ranking on TabZilla Hard. Across 36 deliberately challenging datasets, iLTM tops the critical difference diagram, forming a statistically indistinguishable top group with TabPFNv2, TabM, Mitra, and XGBoost (Figure 3).

• Advantage on high-dimensional data. On a curated biomedical collection with up to 19,993 features, iLTM averages 0.9191 AUC versus XGBoost's 0.9067, with the largest gains on SMK-CAN-187 (0.7336 vs 0.6890) and TOX-171 (0.9750 vs 0.9377) (Table 1).

• Classification pre-training transfers to regression. After short fine-tuning, iLTM achieves the lowest average rank across 18 regression benchmarks, outperforming CatBoost, XGBoost, TabM, and TabPFNv2 (Figure 4).

• GBDT embeddings are a key ingredient. Using boosted trees rather than non-boosted ensembles (e.g., random forests) for the embedding stage produced a larger performance gain; the GBDT itself is intentionally left under-tuned since its role is representation, not prediction.

• Dimensionality-agnostic representation decouples capacity from input size. Random features (dimension 2¹⁵) followed by PCA down to 512 dimensions let the hypernetwork and main network operate on a fixed-size embedding regardless of raw feature count, mitigating the curse of dimensionality.

• Retrieval improves similarity-based prediction. A parameter-free soft k-NN mechanism interpolates main-network logits with neighbor-weighted scores, controlled by a temperature τ and a mixing weight α, and works at inference regardless of training configuration.

• Practical resource footprint. The whole system trains and runs on a single 24 GB A5500 GPU, converging in roughly 400,000 meta-training steps with gradient accumulation size 40.

Methodology in Plain English

The authors start from the observation that trees and neural networks have complementary strengths, and build a pipeline that uses both.

First, each dataset is transformed two ways: a GBDT is fitted and every data point is recorded by which leaf it lands in across all trees, producing a sparse binary "fingerprint" that carries the tree's inductive biases; separately, a robust preprocessing pipeline one-hot encodes categoricals, imputes missing values, and scales features. These representations can be used alone or concatenated.

Second, because datasets have wildly different numbers of features, the authors project everything into a fixed-size space using random features followed by PCA and normalization. This means the model never sees the raw high-dimensional input directly.

Third, a hypernetwork — a meta-model trained across thousands of datasets — reads a small "generation subset" of labeled examples from a new task and outputs the weights of a main MLP specialized to that task. This is the core of the pre-training: the hypernetwork learns how to produce good weights for unfamiliar datasets.

Fourth, at prediction time, the main network's output can be blended with a retrieval score that looks up similar training examples in the learned embedding space and averages their labels, weighted by cosine similarity.

The model is meta-trained on classification tasks via cross-entropy loss, then evaluated on held-out TabZilla and high-dimensional benchmarks. For regression, the architecture is kept identical but per-class operations (like class-mean conditioning and one-hot labels) are replaced with dataset-level versions and standardized targets.

Why This Matters

The paper argues that tabular data deserves a foundation model, and that such a model should not abandon trees but absorb them. If the approach generalizes, practitioners could stop re-running expensive hyperparameter sweeps for every new table and instead deploy a single pre-trained model that adapts on the fly.

Real-world applications:

• Healthcare and biomedical discovery. The high-dimensional experiments use gene-expression datasets (lung, prostate, CLL-SUB-111, TOX-171); a model that handles thousands of features with few samples fits clinical and omics workflows.

• Finance and risk modeling. Tabular credit scoring, fraud detection, and actuarial tasks routinely involve mixed categorical/numerical columns, missing values, and heavy per-task tuning costs that iLTM aims to remove.

• Logistics and operations. Demand forecasting, routing, and supply-chain prediction are tabular and heterogeneous; a pre-trained adapter reduces deployment friction across many subproblems.

• Public services and administration. Government and civic datasets are famously messy and varied; a model robust to feature scale, outliers, and missingness lowers the barrier to applying modern ML.

Industry relevance: The combination of competitive accuracy, less task-specific tuning, an open-source release, and a 24 GB GPU footprint makes iLTM practical for organizations that want foundation-model-style efficiency without re-training from scratch per dataset — a direct challenge to the default "just use XGBoost" workflow.

Future Directions

• Closing the gap to dedicated estimators. iLTM ties but does not dominate XGBoost on TabZilla Hard; understanding which dataset properties favor each remains open.

• Scaling the pre-training corpus and embedding dimensions. The authors note 1024-dimensional embeddings were computationally prohibitive and 256 degraded performance, suggesting headroom if hardware allows.

• Extending beyond classification and regression. Whether the same meta-trained backbone transfers to clustering, anomaly detection, or survival analysis is untested.

• Improving retrieval at scale. Prior work shows retrieval methods can be sensitive to index quality on large or diverse tables; making iLTM's retrieval robust in those regimes is an unresolved engineering question.

• Separating the contribution of each component. The paper integrates four ideas; ablations isolating GBDT embeddings versus hypernetwork versus retrieval would clarify where the gains actually come from.

Target Audience

Applied machine learning engineers and data scientists who work with tabular data and want to reduce per-task tuning; tabular ML researchers tracking the emerging "large tabular model" literature (TabPFN, TabICL, Mitra, HyperFast); and graduate students or methodologists interested in hybrid tree-neural architectures, hypernetworks, or cross-task transfer. Readers without any background in gradient boosting or meta-learning will find Sections 3.1–3.3 dense, but the benchmark results and motivation are readable by a general technical audience.

Authors’ abstract

Tabular data underpins decisions across science, industry, and public services. Despite rapid progress, advances in deep learning have not fully carried over to the tabular domain, where gradient-boosted decision trees (GBDTs) remain a default choice in practice. We present iLTM, an integrated Large Tabular Model that unifies tree-derived embeddings, dimensionality-agnostic representations, a meta-trained hypernetwork, multilayer perceptrons (MLPs), and retrieval within a single architecture. Pre-trained on more than 1,800 heterogeneous classification datasets, iLTM achieves consistently superior performance across tabular classification and regression tasks, from small datasets to large and high-dimensional tasks. After light fine-tuning, the meta-trained hypernetwork transfers to regression targets, matching or surpassing strong baselines. Extensive experiments show that iLTM outperforms well-tuned GBDTs and leading deep tabular models while requiring less task-specific tuning. By bridging the gap between tree-based and neural methods, iLTM offers a new framework for tabular foundation models for robust, adaptable, and scalable tabular learning.