Do Tabular Foundation Models Know Physics? Contamination, Units, and the Deterministic Limit

AI summary

Overview

Research area: Machine learning — specifically tabular foundation models (TFMs), benchmark evaluation, and the intersection of ML with the physical sciences (physics-informed learning, symbolic regression, out-of-distribution generalization).

Technical level: Intermediate. The experimental protocol is straightforward to follow, but interpreting the uncertainty-calibration argument and the Buckingham-π dimensional analysis benefits from some background in Bayesian predictive distributions and dimensional analysis.

Scope in one sentence: The paper audits four tabular foundation models for contamination and then probes what their learned prior actually contains, finding that they interpolate physical data well but cannot represent a noiseless mechanism or exploit physical units.

What This Paper Is About

Tabular foundation models fill in tables the way language models fill in text: entries are masked and recovered from context in a single forward pass, with no gradient step. Since most physical measurement arrives as tabular data, the natural question is whether pretraining on tables taught these models any physics. The authors test that directly by evaluating four TFMs against six trained baselines on data sampled from 316 physical equations, both inside and outside the sampled domain, and then interrogating the two properties every physical law has — noiseless determinism and dimensional structure — that no random structural causal model has.

Key Contributions

1. A contamination audit of the two real-data-pretrained TFMs. The authors enumerate TabDPT's pretraining corpus (122 OpenML datasets — 93 classification and 28 regression — referred to as 123 datasets in Section 3) and Real-TabPFN-2.5's (43 datasets), showing that neither contains a Feynman equation or any regression target generated by an analytic function. Because name and content-hash filtering cannot catch a table resampled from the same equation under a new name, they also measure exposure directly: comparing each model on datasets from its own corpus against comparable unseen datasets, read relative to models that saw neither.

2. A physics-specific benchmark spanning 316 equations. Data from Feynman (120 equations, standardized in SRBench), LSR-Transform (111) and LSR-Synth (85) is regenerated from the equations rather than loaded, sweeping n_samples ∈ {50, 200, 2000}, multiplicative noise σ ∈ {0, 0.01, 0.1}, and split scale k ∈ {1, 2} (in-domain versus extrapolation), over 3 seeds with a fixed 2000-point test set, scored by NMSE and reported as mean rank per stratum.

3. Evidence that the TFM prior cannot represent a noiseless mechanism. On noiseless data the correct posterior predictive is a delta function, yet every TFM reports a nonzero predictive width that plateaus as context grows while its error keeps falling — a failure specific to the σ = 0 limit, since the same models are correctly calibrated at σ = 0.1.

4. Evidence that the TFM prior does not exploit dimensional structure. Using n_π = n_vars − rank(D) with D the SI dimension-exponent matrix from the Feynman units table, the authors show that error tracks raw column count for every model, and that the strongest TFM (TabPFN-3) is significantly worse on tasks with more dimensionless groups, while RealMLP and CatBoost are significantly better.

Main Findings

• No detectable contamination. Neither audited corpus contains a physics equation or analytic regression target; physics appears only as detector classification (MAGIC, HIGGS, MiniBooNE on OpenML). Real-TabPFN-2.5's corpus contains no real-world regression table at all, and its one function-generated entry, fried, links to OpenML 901 — the binarised variant of Friedman #1 with a two-class label thresholded at the mean. In the exposure comparison (TabDPT: 18 in-corpus regression datasets against 37 matched datasets from TabArena and OpenML-CTR23; Real-TabPFN-2.5: fried against the unexposed friedman2), both candidates sit inside the unexposed band at every context size, and models with no pretraining at all show larger seen-versus-unseen gaps. The authors bound what they could have detected: any contamination effect below roughly one rank position would be invisible at this pool size.

• TFMs dominate on physics data, in and out of domain. Under default configurations the ordering reproduces BeyondArena's: TFMs lead the trained baselines across every stratum, and TabPFN-3 leads throughout.

• Tuning does not close the gap. TabPFN-3 still leads under all three pipelines in every stratum, the sole exception being the Gaussian process at the largest context size and only against the TFM's cheapest setting. Mean rank over 314 tasks: 8.47 for TabPFN-3 versus 9.85 for the tuned and ensembled GP, the best baseline. The clearest reshuffling comes from extrapolation rather than tuning: out of domain the field compresses and models converge on a common failure.

• A predictive-uncertainty floor at zero noise. At σ = 0 every TFM plateaus in predictive interval width while its RMSE keeps falling, so reported uncertainty stops tracking error; the Gaussian process interval is exactly zero-width in-domain. On the damped harmonic oscillator with n = 200 noiseless samples, the three distributional TFMs report normalized widths of 0.010 (TabICLv2) and 0.019–0.020 (the two TabPFN variants). At σ = 0.1 the same models sit at the expected calibration width, so the deficiency is specific to the noiseless limit rather than a broken estimator.

• No phase-locked extrapolation of dynamics. All four TFMs fail to continue the damped harmonic oscillator past the sampled range, relaxing to a constant within roughly half a period. TabPFN-3 and TabICLv2 recover the decaying envelope; TabICLv2 shows wave-like banding and TabDPT continues roughly a quarter period before flattening. A phase cross-correlation over 201 lags spanning one period shows this structure is not phase-locked: the correct unshifted curve ranks 98th of 201 for TabICLv2 and 157th for TabPFN-3. TabDPT emits no predictive density and cannot be tested this way.

• Dimensional structure penalises the strongest TFM. Error tracks the raw column count for every model, and for most models dimensionless dimensionality adds nothing further. TabPFN-3 degrades as dimensional reducibility increases, while RealMLP and CatBoost — which can turn that structure to their advantage — improve.

• Ensembling contributes almost nothing. Post-hoc greedy selection measurably improves only RealMLP; for TabM, CatBoost, GP and Ridge the median change is indistinguishable from zero. Allowing 25, 50, 100 or 200 rounds yields results identical to four decimal places, saturating at four to ten distinct configurations. The authors attribute this to noiseless deterministic targets making differently-tuned fits converge on the same function.

• TabDPT is an outlier among the TFMs. It performs worse than its synthetic-prior counterparts specifically on LSR-Synth and on low-dimensional tasks. The authors record this without explanation and note it is not evidence of contamination, since the effect runs in the wrong direction — LSR-Synth is the cleanest of the three sets.

Methodology in Plain English

The authors do not reuse any published benchmark table. For each of the 316 equations they regenerate data directly from the equation, so sample count, noise level and sampling domain stay under their control. Training points are drawn from the equation's sampling box; test points are drawn from a shell at scale k outside it, with rejection of anything falling back inside the k = 1 box — without that rejection most of an out-of-distribution draw would land in-domain and the split would silently measure interpolation. Targets carry multiplicative noise y′ = y(1 + ε), matching the relative error of physical instruments, and the same setup is run across 3 sample sizes, 3 noise levels, 2 split scales and 3 seeds (a fresh draw per seed replaces cross-validation, and the median is taken).

All 26 model–pipeline entities — the four TFMs at two ensemble sizes, plus the six trained baselines at library defaults, tuned with 25 random-search configurations, and tuned with Caruana-style greedy post-hoc ensembling — are ranked jointly inside every (task, regime) cell, then averaged per stratum. Because raw error spans orders of magnitude across equations, ranks are reported rather than raw errors. The authors disclose rather than remove two confounds: trained baselines carve out 20% of n_samples for model selection under the tuned pipelines while TFMs condition on all of it, and each model's internal preprocessing (TabPFN applies its own quantile and power transforms) is left active but recorded. Runs that run out of memory, time out, or return NaN are recorded as failures and imputed at NMSE = 1 rather than dropped; 12 of 362,231 runs failed permanently, all TabDPT, all tracing to one numerical edge case on a single task at n_samples = 2000. The full grid cost approximately 1300 GPU-hours (H100 and A100) and approximately 4200 CPU-hours.

Why This Matters

The paper reframes the "do foundation models know physics" question from capability benchmarking to prior characterization. TFMs are Bayesian by construction — pretraining on samples from a prior under a proper scoring rule makes the forward pass an approximation to the posterior predictive — so the useful question is what that prior contains, and data sampled from known physical laws is the cleanest instrument for answering it. The result is a precise diagnosis rather than a score: TFMs are excellent amortised interpolators whose prior assigns noise to every mechanism and carries no notion of units, which is exactly why they cannot yet act as physical models. The contamination audit is also methodologically useful on its own, since it demonstrates how to bound a detection floor instead of asserting cleanliness.

Applications implied by the framing:

• Scientific measurement pipelines. Physical measurement arrives overwhelmingly in tabular form, so a model that interpolates accurately but reports an irreducible uncertainty floor cannot be used where a noiseless instrument reading is the expected answer.
• Astronomical and survey catalogues. TFMs are already applied in astrophysics, almost exclusively in interpolation; this work delimits where that interpolation stops being trustworthy.
• Symbolic regression and equation discovery. The 316-equation setup and the SRBench-standardized Feynman data connect the TFM results directly to the equation-discovery literature that has historically used the same equations.
• Experimental design and domain coverage. The shell-based extrapolation protocol is a practical template for testing any tabular pipeline on the region just outside where it was trained.

Industry relevance: any workflow that treats a tabular foundation model as a drop-in replacement for a physics-based simulator should read the uncertainty-floor result first, since a model that never reports zero uncertainty on deterministic data misstates its own reliability. The dimensional-analysis finding also matters for feature engineering: a model that is column-permutation-invariant and unit-blind cannot exploit the fact that a five-variable law may depend on only two dimensionless groups.

Future Directions

• Pretrain on continuous physical targets. This is the authors' own stated next step: no current TFM has been pretrained on a physical law with units or a continuous target, and that absence is the natural thing to fix.
• Build a physics-aware prior. A physical model would need a prior that contains physics — one that can assign mass to the noiseless region and that can represent or recover dimensional structure rather than being blind to it.
• Recover phase, not just envelope. The banding in TabICLv2 and the quarter-period continuation in TabDPT are suggestive but not phase-locked; whether any pretraining recipe can produce genuine periodic extrapolation is open.
• Explain TabDPT's divergence. TabDPT performs worse on LSR-Synth and on low-dimensional tasks than the synthetic-prior TFMs, and the authors explicitly leave this without an explanation.

Target Audience

Researchers building or benchmarking tabular foundation models, especially those working on out-of-distribution generalization and uncertainty calibration. Also valuable for physicists and domain scientists who use TFMs in scientific pipelines, for benchmark designers concerned with contamination and detection limits, and for symbolic-regression researchers who work with the Feynman and SRBench equation suites. Readers need not be experts in physics, but the paper's central argument lands hardest for those who already care about the difference between interpolating a function and modelling a law.

Authors’ abstract

Tabular foundation models (TFMs) learn to fill in tables the way language models fill in text, and tables are arguably the format in which most physical measurement arrives. Did they learn any physics in the process? They are Bayesian by construction, so the question is what their prior contains. We probe it directly, evaluating four of them (TabPFN-3, TabICLv2, TabDPT and Real-TabPFN-2.5) against six baselines on datasets sampled from 316 physical equations, in and out of domain. TFMs dominate, out of the box and after tuning. But we show that their prior can represent neither a noiseless mechanism nor physical units, which is why they interpolate physics without yet being able to act as physical models.