GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation

AI summary

Overview

• Research area: Machine learning — graph neural networks (GNNs), large language model (LLM) code generation, and benchmark design.
• Technical level: Advanced. The paper assumes familiarity with graph learning tasks (node, edge, and graph-level prediction), GNN training pipelines, and LLM prompting/agent protocols.
• Scope in one sentence: The paper introduces GNN-CB, a competition-based benchmark of 18 human-curated GNN coding competitions evaluated through a unified hidden-test pipeline, used to compare human participants against nine LLMs under frozen zero-shot prompting.

What This Paper Is About

Existing coding benchmarks such as HumanEval, CodeContests, SWE-bench, and BigCodeBench test general programming and software engineering, while graph benchmarks such as OGB and LRGB test the predictive accuracy of GNN models rather than the ability to implement them. No benchmark previously evaluated whether LLMs can autonomously solve end-to-end GNN coding tasks from a repository specification. GNN-CB fills this gap by staging 18 GNN competitions in which both human experts and LLMs must build complete graph-learning pipelines and submit predictions scored on hidden test sets.

Key Contributions

1. A new benchmark: GNN-CB is presented as the first competition-based benchmark for evaluating both humans and LLMs on GNN coding tasks.
2. 18 curated competitions: The benchmark spans node classification, edge prediction, graph classification, and graph regression across seven graph categories (homogeneous, heterogeneous, temporal, spatial, hierarchical, molecular, generalization), multiple application domains, and three difficulty tiers (Easy, Medium, Hard), all under unified hidden-test evaluation.
3. A controlled LLM evaluation framework: A frozen prompt template with slot filling, a plan-then-code generation paradigm, and bounded execute-and-repair loops (k ≤ 5) are used, supporting both non-agent (API-only) and autonomous agent-based execution.
4. A comparative human–LLM study: Human participants and nine LLMs are evaluated under identical scoring, enabling analysis of performance variability, task difficulty, and implementation failure modes.

Main Findings

• LLMs rarely reach Human Top: Across 9 models and 18 competitions (162 model–task pairs), LLMs tie or surpass the Human Top score in only 19 cases; in more than 88% of model–task pairs, the evaluated LLM does not reach Human Top performance.
• Average human performance stays ahead: The average LLM performance exceeds the average human performance in only 6 of 16 competitions (37.5%).
• No single model dominates: A few competitions are won by LLMs, but humans hold the top score on most tasks. Among models reaching Human Top-level performance there is no clear model-family advantage, with counts roughly proportional to the number of models in each family.
• Open-source models show larger gaps: Open-source reasoning and chat models exhibit larger average gaps and greater cross-task variability than proprietary counterparts under the evaluated protocol. DeepSeek-v4 Pro underperforms proprietary reasoning models in mean gap, and Llama-3.3 70B and Qwen2.5-Coder 32B show larger gaps than Claude Opus 4.6 and Gemini-3 Flash; Kimi-k2.6 is noted as approaching the consistency of proprietary chat models.
• No reasoning-versus-chat advantage: Within each tier, reasoning LLMs show no consistent benchmark-wide advantage over chat LLMs.
• Gaps widen with difficulty: LLM performance gaps are larger on Hard competitions than on Easy and Medium competitions.
• Failure modes are structural, not cosmetic: Defining severe underperformance as scoring at least 30 percentage points below the best LLM on the same task, manual inspection found 20 cases, with Llama-3.3 70B and Qwen2.5-Coder 32B accounting for six and five cases respectively. Causes were incorrect graph construction (8/20), feature processing or utilization (7/20), indexing or data-reading errors (3/20), and unsuitable model selection (2/20).
• Human variability tracks difficulty: Easy tasks such as C03 reached an average score of 0.945 with standard deviation 0.053, while harder tasks such as C02 averaged 0.383 with standard deviation 0.086 and C04 averaged 0.389; medium tasks such as C09 showed notable variability.
• Regression tasks are harder to stabilize: The two regression competitions (C14 BrainAge, MAE; C17 Mix2Print, NMAE) show larger relative performance gaps and less stable convergence than most classification settings.
• Winning-by-a-little cases exist: Claude Opus 4.7 is recorded as tying or improving on Human Top in C08 (99.8 vs 99.6) and C18 (57.2 vs 56.5), while GPT-5.4 Codex improves on Human Top in C13 (83.5 vs 79.9), and Gemini-3 Flash ties in C11 (93.3).

Methodology in Plain English

The authors organized the benchmark around three protocols.

Competition design. Each of the 18 competitions ships with a dedicated repository specifying the task, submission format, evaluation metric, and leaderboard configuration. Submissions are scored only against hidden test sets through automated GitHub-based pipelines that validate formats, run scoring scripts, and update leaderboards without manual intervention. Encrypted prediction files, decrypted only at evaluation time, are used in some configurations to prevent information leakage.

Human evaluation. Participants solve each competition using GNN models only, without external solutions, under controlled constraints (GNN only, 3 CPU hours, seed = 25, at most 5 runs, one final encrypted predictions.csv). Each task had 13–17 participants, drawn from 18 universities across 12 countries. LLM use was restricted to low-level utility support.

LLM evaluation. A single frozen prompt template is instantiated by filling predefined slots with repository-derived information (README, repository tree, dataset summary, submission specification, dependency requirements). The model produces a structured <plan> followed by executable <code>; the code is run locally under CPU-only constraints, and if it fails, the traceback is fed back for repair, bounded at k ≤ 5 iterations. Models are decoded at temperature = 0, with API-only runs using max_output_tokens = 10K and agent-mode runs following each agent's native session budget. The same hidden-test pipeline and task-specific metrics (Macro-F1, accuracy, AUC for classification; MAE and NMAE for regression) score both tracks.

Models evaluated. API-only mode: Gemini-3 Flash, Gemini 2.5 Pro, Llama-3.3 70B, Claude Opus 4.6, Qwen2.5-Coder 32B, Kimi k2.6, and DeepSeek v4 Pro. Agent mode: Claude Opus 4.7 and GPT-5.4 Codex.

Why This Matters

Impact on research. GNN-CB redirects benchmark attention from "can a model write code" to "can a model build a working graph-learning pipeline from a repository description." It also provides a discriminative, competition-style measure and exposes distinct failure modes — especially incorrect graph construction — that aggregate scores hide. The paper is released as a living benchmark with automated evaluation infrastructure, dynamic leaderboards, and reproducible pipelines.

Real-world applications (as reflected in the benchmark's domains):

• Drug and molecular discovery: molecular graph classification for BACE-1 inhibition (C05).
• Clinical and neuroimaging diagnosis: diabetes prediction (C06), diabetic retinopathy (C07), Parkinson's detection (C11), brain-connectivity classification for autism (C16), preeclampsia classification from cfRNA (C04), and brain-age regression from connectome data (C14).
• Security and integrity: source-code graph classification for vulnerability detection (C01) and fake-news graph classification (C03).
• Finance and infrastructure: temporal Bitcoin transaction node classification (C09), citation link prediction (C10), street-network graph classification (C02), GitHub social network node classification (C15), 3D bioprinting material interactions (C17), and cell-type classification (C18).

Industry relevance. The paper argues that its findings show the risk of deploying LLM-generated GNN solutions without expert oversight, and suggests that task-aware model routing or ensembling may be more effective than scaling a single model.

Future Directions

• Extend beyond zero-shot: measure scaling through few-shot prompting, retrieval-augmented generation, and task-specific fine-tuning, and characterize stochastic variability via repeated sampling, confidence intervals, and pass@k-style metrics rather than the single greedy run reported.
• Deepen error analysis: build a fine-grained error-mode taxonomy (incorrect graph construction, invalid message-passing architectures, preprocessing data leakage, mishandled class imbalance, malformed submissions) to explain where and why LLMs break down.
• Analyze cost and agentic depth: use the logged execution and repair traces for token-economy and cost–latency analysis, contrasting the per-call budget of API-only runs with agent session budgets, and for paired significance testing and bootstrap confidence intervals.
• Broaden the benchmark: add regression competitions (only two exist, limiting general conclusions about graph regression) and extend to graph generation, quantum graph modelling, and self-supervised or reinforcement-learning paradigms, using the living benchmark longitudinally to track whether the capability–consistency gap narrows.

Target Audience

Researchers and practitioners in graph machine learning and LLM evaluation who need a reproducible, competition-style testbed for GNN implementation ability; benchmark designers interested in hidden-test pipelines and agentic versus non-agent protocols; and engineering or applied-science teams assessing whether LLM-generated GNN pipelines can be trusted without expert review. The paper is most useful to readers already comfortable with GNN architectures and training workflows, since it does not teach graph learning from scratch.

Authors’ abstract

Large language models (LLMs) have demonstrated strong performance on coding and reasoning benchmarks; however, their ability to solve graph-structured machine learning problems remains largely unexplored. In particular, no benchmark currently evaluates whether LLMs can autonomously solve end-to-end Graph Neural Network (GNN) coding tasks under realistic competition settings. To address this gap, this paper introduces GNN-CB, the first competition-based benchmark for evaluating both humans and LLMs on GNN coding tasks. GNN-CB consists of 18 curated competitions spanning node-, edge-, and graph-level prediction across diverse graph categories, domains, and difficulty tiers. All submissions are evaluated through a unified automated pipeline with hidden test sets and standardized scoring. Human participants solve tasks under controlled competition constraints, while LLMs are evaluated using a frozen zero-shot prompting protocol based on a plan-then-code paradigm with bounded execute-and-repair loops. The benchmark additionally supports both non-agent and autonomous agent-based evaluation within the same protocol. Under our evaluated protocol, LLMs rarely match Human Top performance and show less stable performance across competitions. No single model dominates: a few competitions are won by LLMs, yet humans still hold the top score on most tasks. We release GNN-CB as a living benchmark with automated evaluation infrastructure, dynamic leaderboards, and reproducible execution pipelines. Beyond benchmarking, GNN-CB provides a practice-oriented resource for studying GNN implementation across progressively diverse graph-learning tasks. The benchmark and evaluation framework are publicly available at https://basiralab.github.io/GNN-CB/.