Speedrunning Relational Foundation Model Pretraining

AI summary

Overview

Research area: Relational deep learning — specifically the pretraining of Relational Foundation Models (models that learn directly from multi-table databases linked by primary–foreign keys), combined with benchmark and leaderboard design for training-efficiency research.

Technical level: Advanced. The framing (an open speedrun leaderboard) is accessible, but the recipe involves Relational Transformer internals, a typed attention operator, the Muon optimizer, and a Gaussianity regularizer (SIGReg).

Scope: The paper introduces modded-nanoRT, the first open speedrun benchmark for pretraining a nano-scale Relational Transformer on a single GPU, fixes its protocol by empirical calibration of the held-out database, and opens the leaderboard with a record that reaches the quality target in 305.63 seconds on one NVIDIA H200 — 8.11× faster than the reference implementation.

What This Paper Is About

Progress in pretraining foundation models is limited by who can afford to run experiments: recipe knowledge is developed inside well-resourced labs, reported under private protocols, and does not accumulate into shared knowledge. Community "speedruns" solve this for language and tabular models by fixing the problem (data, hardware, target) and ranking submissions by wall-clock time to reach that target, so many contributors on one GPU each can advance the same recipe. This paper builds the equivalent artifact for relational foundation models, where the format does not transfer mechanically because pretraining mixes several databases with different schemas and the quantity of interest is zero-shot transfer to a database the model has never seen — which means both the training mixture and an entire held-out database must be fixed in advance.

Key Contributions

1. modded-nanoRT, the first relational speedrun leaderboard. A fixed protocol (Table 1): RelBench v1 binary classification tasks minus rel-Avito and rel-Event, temperature-based mixture sampling with α = 0.5, rel-Avito held out, pooled AUROC as the metric, target 0.81621, a 12-block / width-256 / 8-head Relational Transformer with 19.3M trainable parameters as reference, one NVIDIA H200, and a 45-minute budget. Architecture, optimizer, regularization, context sampling, precision, and systems implementation stay open.

2. A calibration procedure for choosing the held-out database. Because a database that saturates early, stalls at the noise floor, or is dominated by seed variance cannot rank entries, the authors run a 45-minute leave-one-out calibration for each of six candidate databases (rel-Amazon, rel-HM, rel-Stack, rel-Avito, rel-Trial, rel-F1) over multiple seeds, and rank them by score(d) = ΔAUROC(d) · s(d) / σ(d), where s is the regression slope of the AUROC curve over the final 20% of wall-clock time and σ is the standard deviation across seeds. rel-Avito ranked first.

3. Typed trilinear attention. The relational operator that conditions attention scores and values on the foreign key traversed, using a frozen text embedding of the foreign-key description as a third multiplicative factor in both the score and the message, so a new schema adds no parameters.

4. A record recipe and its evidence. Typed trilinear attention plus Muon plus SIGReg, reaching the target in 305.63 seconds (8.11× faster than reference), with a matched three-seed attribution study and a leave-one-database-out transfer study (13 of 15 pairs won outside the development database).

Main Findings

• Record time: The archived record logs its first record-eligible verification at 305.63 seconds (5:05.6) on one NVIDIA H200, 8.11× faster than the reference implementation.
• Record AUROC: That verification is the first scheduled evaluation, after 100 updates, and returns 0.871392 pooled AUROC on rel-Avito against a target of 0.81621 — above every value the reference implementation reaches inside its full 45-minute budget.
• Reference baseline cost: The reference needs 10,500 of its 11,621 updates to reach the target and is evaluated every 500 updates, so both reported times are upper bounds set by their own evaluation grids.
• Verification dominates: Verification consumes 182.96 seconds (59.9%) of the record against 122.67 seconds of training (40.1%).
• Record set inside warmup: The 100 updates fall inside the learning-rate warmup, so the record is set before either optimizer reaches its peak rate; the schedules were chosen for a 45-minute horizon and are mis-specified for a five-minute run.
• Record lineage (chronological, not controlled): baseline 41:20 → + typed attention 12:27 (3.32×) → + SIGReg 7:26 (5.56×) → + Muon 5:05.6 (8.11×).
• Muon is the attributable gain: Across three paired seeds, Muon improves RT over AdamW in all three pairs, by +2.56 ± 3.13 points on average, and by +6.60 ± 3.53 points in the presence of typed attention (3/3 wins).
• Typed attention depends on the optimizer: It costs −1.21 ± 5.79 points under AdamW (1/3 wins) but gains +3.56 ± 4.79 points under Muon + SIGReg (2/3 wins). The authors offer two readings: the operator is simply slow, so a fixed budget measures convergence speed rather than compatibility; or the three-factor score needs orthogonalized updates that a coordinate-wise optimizer cannot supply.
• SIGReg shows no final-quality effect: Contrasts straddle zero (−0.47 ± 0.66 points before typed attention, 0/3 wins; +0.26 ± 0.42 points with typed attention, 3/3 wins). It is kept in the record recipe on the time-to-target evidence, and the authors claim no effect for it on attainable quality.
• Transfer across held-out databases: Over six leave-one-database-out holdouts the final recipe wins 15 of 18 matched database–seed pairs, with mean and median gains of 2.14 and 2.53 AUROC points. Excluding development on rel-Avito, it wins 13 of 15 pairs and averages +1.98 points.
• Per-database transfer: rel-F1 +5.04 ± 1.69 (3/3), rel-Avito +2.94 ± 6.24 (2/3), rel-HM +2.37 ± 0.51 (3/3), rel-Stack +1.88 ± 1.85 (3/3), rel-Trial +0.88 ± 2.91 (2/3), rel-Amazon −0.29 ± 12.86 (2/3). Overall 67.93 → 70.06 (15/18). rel-Amazon's 12.86-point standard deviation exceeds every effect measured.
• The relation vector need not be semantic: In a seven-seed ablation, shuffled FK embeddings did not reduce either reported mean (0.8371 ± 0.0381 best AUROC, 0.7794 ± 0.0395 over steps 100–900), while fixed random per type/layer vectors gave the highest means (0.8397 ± 0.0156; 0.7830 ± 0.0254) and reached the target in 7/7 seeds with less than half the record recipe's best-AUROC standard deviation. Removing type identity (random per edge) reached the target in only 3/7 seeds. No pair separates at α = 0.05 (min p_Holm = 0.441), so this ordering is descriptive, but it supports dropping the text encoder.
• Target provenance: 0.81621 is the best pooled AUROC the rel-Avito calibration run reached inside the 45-minute budget, at 41:20 of 45:23; the validation artifact was frozen with seeds capped at 50,000 and a fixed hop radius and text embeddings.

Methodology in Plain English

The authors take an existing, small Relational Transformer (RT) and freeze everything that defines the problem while leaving everything that defines the solution open. The frozen part is: which databases feed pretraining, how they are mixed, which database is held out for zero-shot evaluation, the metric, the target score, the evaluation schedule, the hardware, and the timing boundary. The open part is: architecture, optimizer, regularization, context sampling, numerical precision, and systems engineering. Anyone with one H200 can then submit a recipe and be ranked on the same clock.

Two design decisions specific to the relational setting had to be settled before the clock meant anything. First, the database mixture: seed counts differ by orders of magnitude across RelBench, so sampling proportionally would let the largest database dominate and turn multi-database pretraining into single-database training, while uniform sampling would over-weight the small schemas. They temper the mixture with w(d) ∝ n(d)^0.5 — halfway between proportional (α = 1) and uniform (α = 0) — implemented by subsampling each database's seed list until its share of seeds matches its weight. Second, the held-out database: they run each of six candidates as a 45-minute leave-one-out experiment over multiple seeds and score it by how much AUROC it gains, how steeply it is still climbing at the end, and how much seed noise it carries — a sharp, still-improving, low-variance curve ranks best. rel-Avito won.

For the record recipe they make three changes. Typed trilinear attention gives each foreign-key type its own multiplicative factor inside the attention head dimension (and a matching factor on the message), derived from a frozen text embedding of the foreign-key description, so no new parameters are needed for an unseen schema; the relation vector is held static across the 12 blocks while the projections around it train. Muon replaces the momentum update of each matrix-valued parameter with the nearest semi-orthogonal matrix, computed by a five-step Newton–Schulz iteration in bfloat16, at peak learning rate 5×10⁻³, while AdamW handles encoders, decoders, normalization, scales, biases, and the SIGReg projector at 10⁻³ with weight decay 0.1. SIGReg pushes the hidden states leaving the first block toward isotropy by matching random one-dimensional projections of each (table, column) group to a standard Gaussian via an Epps–Pulley statistic, using a trainable 256 → 1024 → 64 projector, K = 64 random unit vectors, groups of at least eight valid cells, and weight λ = 10⁻³ in the objective.

Evaluation always concatenates predictions and labels across the two rel-Avito tasks (user-visits and user-clicks) into one pooled AUROC, so that no single task's ranking dominates. Attribution uses three paired seeds per configuration and scores the best scheduled checkpoint within the budget; transfer is checked by repeating the leave-one-out procedure across all six databases. Runs that miss the 45-minute target stay in the analysis as right-censored failures.

Why This Matters

Impact on research. Relational pretraining recipes have so far been developed inside individual labs against private baselines, which means improvements do not compose and cannot be independently reproduced. modded-nanoRT changes the unit of progress: a fixed problem, an open stack, a public leaderboard, and a record that fits on one GPU. The calibration methodology — choosing the evaluation database by how sharply its learning curve separates signal from seed noise, rather than by convention — is a reusable idea for any benchmark whose held-out split is a design variable.

Real-world applications.

• Enterprise data platforms: most organizational data lives in relational databases; a model that learns from schema and foreign keys directly could replace the repeated, task-specific flattening pipelines that currently have to be rebuilt for every new database and prediction problem.
• Cold-start prediction on new databases: because the relation typing comes from a frozen text embedding rather than per-relation parameters, a new schema can be handled without retraining — relevant for customers onboarding a database the model has never seen.
• Low-resource research groups and universities: the whole protocol runs on a single NVIDIA H200 with a 19.3M-parameter model and a 45-minute budget, so participation does not require a cluster.
• Reproducibility auditing: frozen training and validation artifacts, a declared random seed, and a fixed timing boundary make published efficiency claims checkable rather than asserted.

Industry relevance. Speedrun leaderboards are a cheap way for a community to do R&D that no single organization would fund, and the results here — that Muon accounts for the attributable gain and that typed attention only pays off in combination with it — are directly actionable for anyone pretraining on multi-table data. The finding that 59.9% of the record is evaluation overhead is also a practical warning: at this scale, the measurement apparatus, not the model, is the bottleneck.

Future Directions

• Attack the evaluation path first. Verification is 59.9% of the record and the first scheduled evaluation clears the target by 5.5 points, so the benchmark currently resolves time-to-target only to the width of its own evaluation grid. Shortening the evaluation path and refining the evaluation schedule are the highest-leverage next changes.
• Refit the learning-rate schedules. The record ends after 100 updates, inside the warmup, using schedules designed for a 45-minute horizon; schedules matched to a five-minute run are an obvious open gain.
• Resolve the typed-attention question. The operator loses 1.21 points under AdamW and gains 3.56 under Muon + SIGReg; whether it is simply slow or genuinely needs orthogonalized updates is unresolved, and testing it under other optimizers would settle it.
• Drop the text encoder. The seven-seed ablation found fixed random per-type/per-layer vectors matched or beat semantic foreign-key embeddings, with lower variance and 7/7 target reach, but no pair separated at α = 0.05 — a higher-powered replication would confirm whether semantic relation content matters at all, and whether the encoder can be removed from the stack.
• Widen the evidence beyond best-checkpoint selection. The reported contrasts select the best scheduled checkpoint over three seeds and rank rather than test; rel-Amazon's 12.86-point standard deviation exceeds every measured effect, so more seeds and formal tests are needed for the transfer claims.

Target Audience

Machine learning researchers and engineers working on relational deep learning, tabular foundation models, or schema-aware architectures who want a reproducible, single-GPU setting for testing architectural and optimizer ideas. It is equally aimed at practitioners in industry who pretrain on multi-table enterprise data and need to know which recipe changes actually buy training efficiency, and at benchmark designers interested in how to calibrate a held-out split so that a leaderboard can resolve real improvements. Readers without background in Relational Transformers, Muon, or distribution-matching regularizers will need to consult the cited prior work.

Authors’ abstract

Progress on foundation model pretraining is gated by who can afford a trial, and relational pretraining is no exception. Recipes are developed inside individual labs, reported under private protocols, and do not accumulate. Speedrun leaderboards remove that barrier by decentralizing the search rather than the training run. The problem is fixed, the stack is open, entries are ranked by wall-clock time to a quality target, and one GPU is enough to hold the record. We introduce modded-nanoRT, the first such leaderboard for Relational Transformers. Which database to hold out decides whether the benchmark can resolve improvements at all, so we fix it by calibration, ranking six candidates by how sharply a 45-minute learning curve separates improvement from seed noise. We open the leaderboard with typed trilinear attention, a relational operator that conditions attention scores and values on the foreign key traversed, together with Muon and SIGReg. The recipe reaches the target in 305.63 seconds on one NVIDIA H200, 8.11x faster than the reference implementation, and wins 13 of 15 pairs outside the development database. We will release the benchmark, its frozen artifacts, and the record runs. Our code is available at https://github.com/kunumi/nanoRT.