GraphBench: Next-generation graph learning benchmarking

AI summary

Overview

Research area: Graph machine learning, specifically benchmarking and evaluation methodology for graph neural networks and graph foundation models.

Technical level: Intermediate. The paper is readable without deep architectural knowledge, but evaluating it fully requires familiarity with message-passing neural networks, graph transformers, graph-level and node-level prediction tasks, and out-of-distribution evaluation.

Scope: The paper introduces GraphBench, an open-source benchmark suite of 38 datasets across node-, edge-, graph-level, and generative tasks, with standardized splits, domain-specific metrics, hyperparameter-tuning scripts, and out-of-distribution (OOD) evaluation, along with baselines for message-passing neural networks (MPNNs) and graph transformers (GTs).

What This Paper Is About

Machine learning on graphs has advanced in areas such as molecular property prediction and chip design, but evaluation practices are fragmented: datasets tend to be narrow and task-specific, protocols are inconsistent, and metrics are often misaligned, which limits reproducibility and fair comparison. This is especially problematic for graph foundation models, which need broad, realistic, and difficult benchmarks to be meaningfully evaluated. GraphBench is the authors' response: a unified benchmark suite spanning multiple real-world domains and learning regimes, with standardized evaluation and tuning procedures, plus baseline results for modern MPNNs and GTs.

Key Contributions

1. A unified benchmark suite. GraphBench spans four domains divided into seven subdomains, yielding 38 datasets in total, covering node-level, edge-level, graph-level prediction, and generative tasks. The domains include social networks, electronic circuits, chip design, combinatorial optimization, SAT solver prediction, algorithmic reasoning, and weather forecasting.

2. Standardized protocols and tuning. For each dataset the suite provides a clearly defined prediction or generation task, domain-specific splits, standardized input features and labels, evaluation scripts with task-specific metrics such as RMSE, ROC-AUC, and closed gap, and hyperparameter-tuning scripts.

3. Explicit out-of-distribution evaluation. Selected tasks test OOD generalization through domain-specific temporal shifts or problem-size shifts rather than random train–test splits, which the authors argue is necessary for evaluating graph foundation models.

4. Documented data-generation pipelines and baselines. The full dataset generation process for all domains is documented in the appendix to support replication and extension, and the paper benchmarks MLP and DeepSet baselines (which ignore connectivity), GIN, additional GNNs for electronic circuit datasets, recent GNNPlus models, a graph transformer, and a specialized weather graph transformer inspired by GraphCast.

Main Findings

• Social network engagement prediction favors graph-aware models. On the Bluesky-based task of predicting aggregated future engagement, results show a consistent ranking from DeepSet to MLP to GNNs across all interaction types (quotes, replies, reposts), with GNNs achieving lower MAE and higher R² and Spearman correlation. On quotes, GatedGCN+ reaches MAE 0.734 ± 0.010, GIN+ 0.737 ± 0.014, versus MLP 0.784 ± 0.001 and DeepSet 0.810 ± 0.005. On replies, GIN+ reaches MAE 0.669 ± 0.017 versus MLP 0.725 ± 0.004 and DeepSet 0.789 ± 0.033. On reposts, GIN+ reaches 0.810 ± 0.015 versus MLP 0.874 ± 0.003 and DeepSet 0.918 ± 0.013. Overall R² and Spearman values remain modest, which the authors say leaves room for stronger task-specific GNNs. GatedGCN+ ran out of memory in all tested configurations on the replies and reposts datasets.

• Electronic circuit prediction degrades with circuit size. On RSE for power conversion efficiency, results are 0.07 ± 0.03 (graph transformer, GT), 0.09 ± 0.06 (GIN) and about 0.04 for GNNPlus models at 5 components, rising to 0.37 ± 0.02 (GT), 0.44 ± 0.13 (GIN), and 0.49 ± 0.07 (GIN+) at 10 components. The GT consistently achieves higher accuracy than MPNN baselines for seven and ten components, while GNNPlus baselines improve performance on small circuits, suggesting an advantage for local message passing. The authors attribute the RSE increase for larger circuits to reduced training data from costly high-fidelity simulation and combinatorial growth of the circuit space.

• Combinatorial optimization: GIN strongest, graph transformer weak. For supervised maximum independent set prediction (MAE, lower is better), GIN achieves the best results on most datasets, which the authors attribute to MPNNs' inductive bias aligning well with graph-structured CO problems. For example, GIN scores 0.491 ± 0.099 on small RB graphs and 0.292 ± 0.041 on small BA graphs, while the GT scores 4.112 ± 2.353 on small RB graphs and 3.481 ± 3.446 on small BA graphs. GNNPlus models are far more accurate, e.g., GIN+ at 0.0006 ± 0.0002 on small BA graphs and 0.001 ± 0.0003 on large BA graphs. DeepSet generally outperforms the MLP baseline except on ER graphs, suggesting global information aggregation helps this task.

• SAT solving receives the largest algorithm-selection dataset the authors are aware of. The three SAT datasets cover a wide range of instance sizes, each with two tasks: performance prediction (regression of solver runtime) and algorithm selection (multi-class classification of the best solver). Dataset-specific challenges include noisy targets from stochastic solver runtimes and large graphs requiring both local and global formula structure.

• Algorithmic reasoning: solid baselines, imperfect extrapolation. The paper contributes 21 large-scale datasets covering seven classic graph algorithms at three difficulty levels, each including a shift in the graph data distribution and larger graphs in test sets. Across experiments, GNN and GT models show strong baseline performance, but extrapolation on size and data distribution is not perfectly achieved.

• Weather forecasting residual prediction. The task predicts the 12-hour future change (residual) in atmospheric state from an initial snapshot, using a spatially and variable-weighted MSE inspired by GraphCast's objective; an unweighted MSE is reported per variable at evaluation. Data comes from ERA5 reanalysis preprocessed via the WeatherBench2 pipeline, downsampled to a 64 × 32 equiangular grid with conservative area-preserving interpolation, a 6-hour temporal resolution (0h, 6h, 12h, 18h), and 62 physical and derived variables, of which 15 are defined across 13 pressure levels and 47 at the surface.

Methodology in Plain English

The authors start from a critique of existing benchmarks: TUDatasets offers many graph-level tasks but most datasets are small and molecular without unified metrics; the Open Graph Benchmark and its extensions provide larger datasets but focus mainly on molecules and citation graphs, are partially saturated, support limited model comparison, and lack generative tasks; other benchmarks target narrow prediction regimes with small scale and limited OOD evaluation; and graph generation benchmarks are particularly underdeveloped. GraphBench is assembled to fill those gaps.

Concretely, the team collects or constructs datasets across four application domains and seven subdomains, and organizes them by learning regime: node-level, edge-level, graph-level prediction, and generation. For each dataset they fix a task definition, realistic splits, input features, labels, and evaluation metrics, and they add hyperparameter-tuning scripts so that comparisons happen under realistic conditions. Where OOD generalization matters, they design temporal shifts (for example, in the Bluesky social data, edges are built only from interactions observed before a cutoff, and predictions concern later posts) or problem-size shifts. Finally, they run a common set of baselines across the suite: MLP and DeepSet models that ignore graph connectivity, GIN, additional GNNs for circuits, GNNPlus models, and graph transformers. All of this is packaged as an open-source Python library with data loaders, PyTorch Geometric data objects, predefined splits, and tuning scripts.

Why This Matters

GraphBench targets reproducibility and comparability in a field where results are hard to compare across papers because datasets, splits, and metrics differ. Its inclusion of OOD splits and non-standard domains pushes evaluation beyond random splits and molecule-centric tasks, which matters increasingly as graph foundation models claim broad capability.

Real-world applications represented in the benchmark:

• Social platforms: predicting future post engagement supports user ranking, trend detection, influencer marketing, and proactive moderation.
• Hardware and chip design: analog power converter surrogates aid topology generation and parameter tuning, while Boolean circuit generation supports logic synthesis where minimal circuits are NP-hard to find.
• Combinatorial optimization: vehicle routing, scheduling, and resource allocation, with a supervised setting that predicts optimal objective values and an unsupervised setting that decodes solutions directly when ground truth is unavailable or expensive.
• SAT-based workflows and earth systems: SAT solvers underpin hardware and software verification, automated planning, and operations research; weather forecasting matters for agriculture, energy, and public safety.

Industry relevance: the benchmark deliberately covers modern application domains such as large-scale chip design, weather forecasting, and combinatorial optimization rather than small citation or molecular graphs. The released Python package, standardized loaders, and tuning scripts lower the engineering cost for teams comparing models or building graph foundation models.

Future Directions

• Sliding-window and other temporal variants for the social network task. The paper states that the current design accumulates edges from the start of the record up to each split cutoff, making validation and test graphs denser than the training graph, and explicitly leaves sliding-window variants for future work.
• Stronger task-specific models for engagement prediction. Because R² and Spearman values remain modest across interaction types, the authors note there is room for stronger task-specific GNNs.
• More expressive architectures for combinatorial optimization. The paper notes that solving these CO problems has been shown to require more than 1-WL expressivity, indicating a need for more expressive architectures, and that the unsupervised surrogate-loss setting poses additional training challenges for MPNNs.
• Better extrapolation in algorithmic reasoning and extension of the suite. Size and distribution extrapolation is not perfectly achieved by GNN and GT baselines, and the modular framework is designed to be extended to new domains, datasets, and baselines.

Target Audience

Researchers and practitioners who build or evaluate graph neural networks, graph transformers, and graph foundation models; benchmark designers and reproducibility-focused methodologists; and industry teams working on chip design, electronic design automation, circuit synthesis, weather forecasting, social platform analytics, SAT-based verification, and combinatorial optimization who need realistic, standardized evaluation rather than narrow dataset-specific comparisons. The open-source Python package and PyTorch Geometric integration also make it useful for engineers prototyping new graph models under a unified protocol.

Authors’ abstract

Machine learning on graphs has made substantial progress across domains such as molecular property prediction and chip design. Yet benchmarking practices remain fragmented, often relying on narrow, task-specific datasets and inconsistent evaluation protocols, hindering reproducibility and broader progress. With the recent popularity of graph foundation models, these weaknesses have become apparent, as existing benchmarks are insufficient for thorough evaluation. To address these challenges, we introduce GraphBench, a comprehensive benchmark suite spanning diverse real-world domains and task settings, including node-level, edge-level, graph-level, and generative tasks. GraphBench provides standardized evaluation protocols, including consistent dataset splits and metrics for assessing out-of-distribution generalization across selected tasks, as well as a unified hyperparameter-tuning framework. We further evaluate GraphBench with recent message-passing neural networks and graph transformer models, establishing principled baselines for future research. See www.graphbench.io for further details.