TabICLv2: A better, faster, scalable, and open tabular foundation model

AI summary

Overview

• Research area: Machine learning — tabular data, specifically tabular foundation models (TFMs) built on prior-data fitted networks (PFNs) and in-context learning.
• Technical level: Advanced. The paper assumes familiarity with Transformer attention, softmax temperature scaling, optimizers (AdamW, Muon), synthetic priors, and benchmark methodology such as cross-validation and improvability.
• Scope in one sentence: The paper introduces TabICLv2, an open tabular foundation model with a new synthetic data generator, architectural changes (repeated feature grouping, target-aware embedding, query-aware scalable softmax, mixed-radix ensembling, quantile regression), and improved pretraining (Muon optimizer), evaluated on the TabArena and TALENT benchmarks.

What This Paper Is About

Gradient-boosted trees long dominated tabular prediction, but tabular foundation models such as TabPFNv2 and TabICL now top predictive benchmarks by performing in-context learning in a single forward pass rather than training per dataset. The problem this paper targets is that the strongest TFMs (notably RealTabPFN-2.5) rely on hyperparameter tuning, ensembling, and fine-tuning on real data, while remaining slow and memory-hungry on large tables. TabICLv2 aims to beat that state of the art without any tuning, while making the model faster, scalable to million-scale tables within modest memory, and fully open, with code released for inference, pretraining, and synthetic data generation.

Key Contributions

1. A new synthetic data generation engine. The prior retains the structural causal model framework but adds a "random Cauchy graph" mechanism (replacing tree-structured graph sampling), eight types of random functions (MLP, Tree Ensemble, Discretize, GP, Linear, Quadratic, EM, Product), correlated scalar sampling via shared Beta-distribution parameters, and dataset/graph filtering.
2. Architectural innovations at roughly constant model size. Repeated feature grouping via circular shifts, target-aware embedding injected before row compression, query-aware scalable softmax (QASSMax) applied to TF_col inducing-point aggregation and TF_icl, mixed-radix ensembling for more than 10 classes, and 999-quantile regression trained with pinball loss.
3. Optimized pretraining protocols. A three-stage curriculum with batch size 64 over roughly 35M datasets, learning rates of 8e-4, 1e-4, and 2e-5 for the three stages, gradient clipping raised from 1 to 10 in stages 1 and 2, cosine learning rate scheduling, and replacement of AdamW with the Muon optimizer based on the Schaipp (2025) implementation.
4. Extensive evaluation and ablation, plus open releases. Benchmarks on TabArena and TALENT, runtime comparisons, scalability tests up to roughly 600K samples, a component-wise ablation study, and released code at https://github.com/soda-inria/tabicl plus a self-contained short implementation at https://github.com/soda-inria/nanotabicl.

Main Findings

• State of the art on both benchmarks without tuning. On TabArena and TALENT, TabICLv2 without any tuning surpasses RealTabPFN-2.5, which is described as hyperparameter-tuned, ensembled, and fine-tuned on real data. A statistical test confirms the win rate versus RealTabPFN-2.5 on TALENT is greater than 50%.
• Better Pareto front. TabICLv2 dominates the improvability-versus-runtime Pareto fronts on both benchmarks and substantially outperforms heavily tuned CatBoost and XGBoost with orders of magnitude less training time.
• Consistently faster than TabPFN-2.5. With 100 features and 8 estimators, TabICLv2 is faster across all hardware tested, reaching 10.6x speedup on an H100 GPU at 50K samples and 11.8x on CPU at just 10K samples.
• Scalability to large tables. The model maintains top rankings across dataset sizes from 10^3 to 10^5, outperforming RealTabPFN-2.5 on larger datasets (more than 20K samples). On roughly 600K-sample datasets from the TALENT extension, it still performs strongly, whereas TabPFN-2.5 produced out-of-memory errors. Disk offloading reduces requirements to under 24 GB CPU and 50 GB GPU to process a table with 1M samples and 500 features within 450 seconds.
• Many-class classification. On 12 TALENT datasets with more than 10 classes, TabICLv2 with the ECOC wrapper from TabPFNv2 or its native mixed-radix ensembling substantially outperforms all baselines; the ECOC wrapper is slightly better but 3x slower than the native handling.
• QASSMax defeats attention fading in a controlled test. In a needle-in-haystack task with four negative clusters and one anchor cluster, QASSMax maintained 100% accuracy with consistently low attention entropy even with 15K negatives, while the no-SSMax variant lost accuracy and collapsed at 15K negatives, and SSMax degraded substantially at extreme scales.
• Prior is the largest single factor in the ablation. Pretraining TabICLv2 with the TabICL prior substantially lowers performance and validation loss degrades in the second half of pretraining; pretraining the TabICL architecture with the TabICLv2 prior only matches TabICL. Three components give comparable significant gains (approximately 100 Elo, 64% win rate): early target inclusion, Muon instead of AdamW, and QASSMax. Repeated feature grouping and prior filtering yield smaller gains.
• Efficient convergence and data filtering. TabICLv2 matches RealTabPFN-2.5 in log-loss after approximately 200K steps, and in fewer than 100K steps on normalized accuracy. In pretraining stage 1, roughly 35% of classification and 25% of regression datasets are filtered out.
• Moderate pretraining cost. Totaling 24.5 GPU-days per model on H100 80GB GPUs (about 20 for stage 1, 2.5 for stage 2, 2 for stage 3), lower than TabICL's 60 A100-days, with around 35M datasets versus TabICL's approximately 83M and TabPFNv2's approximately 130M.
• Few-shot and robustness behavior. With 16 training samples, which are not included in its pretraining, TabICLv2 comes second to RealTabPFN-2.5 while outperforming untuned classical baselines and the LLMs Claude Opus 4.6 and Qwen3.5-112B. It is also quite robust to added label noise compared with tree-based models, and unlike TabICL(v1) it does not benefit from TabPFNv2's random forest extension on larger datasets.

Methodology in Plain English

The team builds a Transformer that, given a training set and a test row in one forward pass, predicts the test target without any gradient updates on the user's data. Instead of training on real tables, the model is pretrained on millions of entirely synthetic datasets produced by a data-generating "prior": random causal graphs are sampled, random functions are applied at each node, and columns are extracted from randomly assigned nodes, with filtering to discard degenerate datasets.

To keep computation manageable, the architecture follows TabICL's two-stage design: a lightweight column-wise stage embeds each feature, a row-wise stage compresses each row into a single vector through [CLS] tokens, and a dataset-wise stage performs in-context learning over row embeddings, giving a runtime complexity of O(n^2 + nm^2) for n rows and m columns. Three modifications sharpen this: each column is placed into multiple circular-shift groups so features do not collapse into identical representations; target values are added directly to every training token early so the model can judge feature relevance before compression; and a query-aware temperature scaling (QASSMax) counteracts the flattening of attention over long contexts.

For classes beyond the 10 seen in pretraining, labels are decomposed into mixed-radix digits and column embeddings are averaged across digits. For regression, the model predicts 999 quantiles with pinball loss, averaging them for point estimates and otherwise building a monotone distribution with closed-form PDF, CDF, and moments. Pretraining proceeds in three stages of increasing dataset size, using Muon instead of AdamW and a higher learning rate. Evaluation uses the TabArena (51 datasets) and TALENT (300 datasets) benchmarks, with metrics chosen per task (ROC AUC, log-loss, RMSE) and primarily reported as improvability, the average relative error gap to the best method per dataset.

Why This Matters

The work matters because it shows that a fully open model, without tuning or real-data fine-tuning, can lead the tabular prediction leaderboard while also being fast and memory-efficient enough for large tables — addressing the concern that top-tier TFM performance had become locked behind closed recipes. The ablation also isolates which ingredients drive gains (prior quality, early target injection, Muon, QASSMax), which is useful guidance for others building TFMs.

Real-world applications, based on domains the paper associates with tabular data:

• Healthcare tabular records and prediction tasks.
• Credit card fraud detection.
• Downstream adaptations built on TFMs such as causal inference, generative modeling, joint predictive distributions, and simulation-based inference.
• Use as a basis for future foundation models for time-series, causality, and relational databases, as suggested in the conclusion.

Industry relevance: TabICLv2 offers out-of-the-box accuracy with orders of magnitude less training time than tuned gradient-boosted trees, plus CPU speedups (11.8x at 10K samples) that make deployment practical outside GPU-rich environments. Its released inference, pretraining, and synthetic data generation code lowers the barrier to reproducing and extending state-of-the-art tabular modeling.

Future Directions

• Text and semantics. TabICLv2 does not natively use semantic information from column names or textual features, which prior work has shown to be valuable; combining it with text embedding models is suggested as feasible given its scalability to many features.
• Truly massive data and new tasks. Datasets with millions of samples remain challenging, and extensions such as multi-output regression and handling distribution shifts are left to future work.
• Missing values. Missing entries are currently imputed with the mean; adding missing indicators or introducing missingness during pretraining may improve handling, but is unexplored.
• Distributional regression and tuning. Capabilities in distributional regression are not evaluated beyond toy datasets because of a lack of established benchmarks, and hyperparameter tuning or fine-tuning, which could further improve performance at increased runtime cost, is not explored.

Target Audience

This paper is most useful to machine learning researchers and practitioners working on tabular data, foundation models, and in-context learning, especially those interested in synthetic prior design, long-context attention scaling, or efficient large-table inference. It also suits engineers who need strong out-of-the-box tabular predictions under compute and memory constraints, and readers who want an ablation-backed account of which design choices actually drive TFM performance. Some background in Transformer attention and benchmark methodology is required to follow the details.

Authors’ abstract

Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in attention improving generalization to larger datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the Muon optimizer. On the TabArena and TALENT benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale datasets under 50 GB GPU memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and foster open research by releasing code for inference, pretraining, and synthetic data generation at https://github.com/soda-inria/tabicl.