RelBench v2: A Large-Scale Benchmark and Repository for Relational Data

AI summary

Overview

Research area: Machine learning benchmarks for relational deep learning (RDL) — learning directly on multi-table relational databases rather than on flattened, single-table feature sets.

Technical level: Intermediate. The paper assumes familiarity with graph neural networks, temporal train/validation/test splitting, and standard supervised metrics, but the core ideas (multi-table schemas, autocomplete versus forecasting tasks, leakage prevention) are explained in accessible terms.

Scope (one sentence): This paper introduces RelBench v2, an expansion of the RelBench benchmark with four new relational datasets, a new class of "autocomplete" predictive tasks, and integrations of external benchmarks for evaluating models that learn on relational databases.

Note that the provided paper content is truncated partway through Section 6.3 (4DBInfer); consequently, the paper's conclusion, limitations, and any results beyond that point are not reported in the available text.

What This Paper Is About

Relational databases store structured data across many linked tables, and the RDL paradigm treats those databases as heterogeneous graphs so models can learn from entities and their relationships directly, instead of relying on hand-engineered flattening of the schema into a single table. Existing benchmarks for this setting were limited in scale, domain coverage, and task variety, which constrains systematic evaluation as the field moves toward larger models and relational foundation models. RelBench v2 addresses this by adding large-scale datasets across new domains, introducing tasks that predict existing column values under temporal constraints, and integrating external benchmarks and evaluation frameworks into a single repository.

Key Contributions

1. Four new large-scale relational datasets. rel-arxiv (scholarly publication data), rel-salt (enterprise resource planning), rel-ratebeer (a consumer platform), and rel-mimic (clinical records) collectively contribute over 22 million rows (22,615,013) across 29 tables and 327 columns, bringing the benchmark to 11 datasets.

2. A new task paradigm: autocomplete tasks. Unlike RelBench v1 forecasting tasks, whose targets are constructed via SQL queries, autocomplete tasks ask models to infer missing values of existing columns in relational tables at a given seed time while respecting temporal constraints. RelBench v2 introduces 23 such tasks, spanning binary classification, multiclass classification, and regression.

3. Thirteen new forecasting tasks for rel-arxiv, rel-ratebeer, rel-mimic, and rel-f1, covering entity binary classification, entity multiclass classification (the benchmark's first entity multiclass task, rel-arxiv's author-category), entity regression, and recommendation (link prediction).

4. Integration of external benchmarks and frameworks. Event streams from the Temporal Graph Benchmark (TGB) are translated into relational schemas, ReDeLEx provides uniform access to 70+ real-world databases suitable for pretraining, and 4DBInfer datasets and tasks are incorporated to broaden multi-table evaluation coverage.

Main Findings

• RDL models beat single-table baselines across task types. The paper states that RelBench v2's experimental results demonstrate RDL models consistently outperform single-table baselines across autocomplete, forecasting, and recommendation tasks, which the authors present as evidence for modeling relational structure explicitly.

• Autocomplete binary classification: The heterogeneous GraphSAGE-based model outperformed the LightGBM baseline on every reported binary task. Averages were 70.84 (validation) and 70.52 (test) for the GNN versus 60.71 and 50.34 for LightGBM. The authors exclude random choice and majority class as baselines here because both correspond to AUROC values of approximately 50.00.

• Autocomplete multiclass classification: The GNN reached an average accuracy of 71.66 (validation) and 69.14 (test), compared with 36.55 and 28.22 for LightGBM, 37.83 and 35.79 for the majority baseline, and 20.78 and 19.51 for random. Individual tasks such as rel-salt's item-plant reached 99.70 validation and 99.46 test accuracy.

• Autocomplete regression: RDL achieved higher R² values on most tasks, with notable test results including 0.736 on rel-hm's transactions-price and 0.394 on rel-ratebeer's beer_ratings-total_score. The paper states that RDL outperforms both feature-based and aggregation baselines "across most tasks."

• Entity binary classification forecasting: Averages were 81.68 (validation) and 78.19 (test) for the GNN versus 74.18 and 71.79 for LightGBM, with the largest reported margins on rel-ratebeer's user-churn (96.85 validation, 94.27 test).

• Entity multiclass classification forecasting: On rel-arxiv's author-category task, the GNN achieved 52.63 validation and 50.74 test accuracy, versus 1.95 and 2.01 for LightGBM, 8.83 and 9.09 for the majority baseline, and 1.75 and 1.77 for random. The authors attribute this to benefits from coauthorship, citation patterns, and publication context.

• Recommendation: ID-GNN with four layers performed best, averaging 23.43 (validation) and 23.35 (test) MAP, compared with 16.09 and 13.96 for LightGBM, 14.48 and 7.45 for Past Visit, and 11.96 and 10.61 for Global Popularity. The authors note that ID-GNN excels when predictions are highly entity-specific, while the plain GNN performs better when such specificity is less critical.

• Deeper models help but are memory-limited. Increasing RDL models from two to four layers tended to improve performance for both GNN-based models, though the four-layer GraphSAGE model encountered CUDA memory errors on an 80GB Nvidia A100.

• Class imbalance and instability appear in some tasks. On rel-salt's sales-office, the majority baseline already achieves 99.91 validation and 99.88 test accuracy; the GNN matches or slightly improves on it, whereas LightGBM shows unstable behavior (59.93 test) despite high validation accuracy (99.90).

• TGB translation covers diverse scales. Table 11 lists 13 translated TGB datasets, ranging from tgbl-wiki-v2 (3 tables, 166,701 rows, 7 columns) to tgbn-token (5 tables, 81,663,534 rows, 14 columns) and thgl-software (18 tables, 2,171,733 rows, 74 columns).

Methodology in Plain English

The authors extend an existing benchmark rather than proposing a new model architecture. Their approach has four parts:

1. Data curation. Four real-world relational databases are packaged as parquet tables plus schema metadata, each with fixed validation and test timestamp cutoffs. Models train on data up to the validation cutoff, validate on rows between the validation and test cutoffs, and are tested on rows after the test cutoff. Data beyond the test cutoff is hidden during inference to prevent test-time leakage, using the temporal neighbor sampling strategy of Fey et al. (2024). rel-mimic additionally applies patient-level date shifting for privacy.

2. Task construction. Two task families are defined. Forecasting tasks, as in RelBench v1, use SQL queries to build target columns, and each task specifies an entity ID, a seed time, and target labels via a training table. Autocomplete tasks instead target existing columns: for each task a fixed set of observed columns and one target column are defined, and models must infer the target using only relational and temporal context available up to the seed time. Because some columns are highly correlated, the authors manually identify and remove columns that would leak the target — for example, dropping the "review_text" column in the rel-amazon review-rating task.

3. Model implementation. Raw row-level data is encoded into initial node embeddings using PyTorch Frame, specifically with the ResNet tabular model. Temporal-aware subgraph sampling is performed around each entity node at the seed time, and embeddings are passed into a heterogeneous GraphSAGE model with sum-based neighbor aggregation to iteratively update node embeddings. Task-specific prediction heads turn output embeddings into predictions. For recommendation tasks, GraphSAGE computes pairwise scores via inner product and is trained with Bayesian Personalized Ranking loss, while ID-GNN routes target embeddings through a source-specific MLP head and is trained with cross-entropy loss.

4. Evaluation. Autocomplete and forecasting classification use ROC-AUC for binary tasks and accuracy for multiclass tasks; regression uses R² and also reports MAE; recommendation uses MAP@K with K set per task. Baselines include LightGBM over raw entity table features, global zero, global mean/median, entity mean/median, random, majority class, Past Visit, and Global Popularity. The paper reports averages alongside per-task numbers and refers to appendix tables for standard deviations and MAE results.

Why This Matters

Impact on research. RelBench v2 provides a larger, more diverse evaluation surface for a paradigm the authors describe as moving toward larger models and relational foundation models. Sharing validation and test cutoffs across all tasks within a dataset enables multi-task learning and pre-training across predictive tasks on the same relational database. The integrations with TGB, ReDeLEx, and 4DBInfer allow comparisons across temporal GNN baselines, relational deep learning baselines, and a wider set of relational databases, which the authors frame as a step toward unified relational-temporal evaluation.

Real-world applications (as described or directly implied in the paper):

• Predicting customer churn and estimating sales, which the paper identifies as central to real-world decision making.
• Product recommendation, including next-item and next-link prediction such as whether users will purchase a product.
• Clinical outcome prediction from electronic health records, represented by rel-mimic's patient ICU length-of-stay task.
• Enterprise order fulfillment and supply chain operations, represented by the SAP-derived rel-salt multiclass tasks over sales offices, shipping points, and payment terms.
• Scholarly network analysis, including citation prediction and inferring an author's research area from publication context.
• Attribute completion inside database interfaces, motivated by the SAP S/4HANA Sales Order user interface, where a payment category is recommended based on other filled-in fields.

Industry relevance. The dataset mix — enterprise resource planning data from SAP AI Research, a consumer review platform, corporate-style sales order workflows, and hospital records — reflects settings where relational databases are already the primary storage abstraction. The autocomplete framing is especially relevant to enterprise software, since it targets the everyday problem of filling in missing fields rather than a separate offline prediction pipeline. Because rel-mimic requires PhysioNet credentials and supports date-shifted, de-identified data, the benchmark also illustrates how sensitive industrial and clinical data can be distributed for research under access controls.

Future Directions

• Relational foundation models. The paper motivates benchmarks explicitly by the emergence of foundation models for relational databases and notes that relational databases are a "natural next frontier" for pretraining; scaling pretraining across the benchmark's shared task cutoffs is the implied next step.
• Large-scale pretraining data. ReDeLEx's integration of 70+ real-world relational databases is presented as suitable for pretraining, raising the question of how well models pretrained on that breadth transfer to RelBench's forecasting and autocomplete tasks.
• Scaling model depth and memory. The four-layer GraphSAGE configuration encountered CUDA memory errors on an 80GB Nvidia A100 on recommendation tasks, so deeper relational architectures will require more memory-efficient sampling or training.
• **Leakage-safe

Authors’ abstract

Relational deep learning (RDL) has emerged as a powerful paradigm for learning directly on relational databases by modeling entities and their relationships across multiple interconnected tables. As this paradigm evolves toward larger models and relational foundation models, scalable and realistic benchmarks are essential for enabling systematic evaluation and progress. In this paper, we introduce RelBench v2, a major expansion of the RelBench benchmark for RDL. RelBench v2 adds four large-scale relational datasets spanning scholarly publications, enterprise resource planning, consumer platforms, and clinical records, increasing the benchmark to 11 datasets comprising over 22 million rows across 29 tables. We further introduce autocomplete tasks, a new class of predictive objectives that require models to infer missing attribute values directly within relational tables while respecting temporal constraints, expanding beyond traditional forecasting tasks constructed via SQL queries. In addition, RelBench v2 expands beyond its native datasets by integrating external benchmarks and evaluation frameworks: we translate event streams from the Temporal Graph Benchmark into relational schemas for unified relational-temporal evaluation, interface with ReDeLEx to provide uniform access to 70+ real-world databases suitable for pretraining, and incorporate 4DBInfer datasets and tasks to broaden multi-table prediction coverage. Experimental results demonstrate that RDL models consistently outperform single-table baselines across autocomplete, forecasting, and recommendation tasks, highlighting the importance of modeling relational structure explicitly.