OpenMAG: A Comprehensive Benchmark for Multimodal-Attributed Graph

AI summary

Overview

• Research area: Multimodal-Attributed Graph (MAG) learning — graphs whose nodes carry attributes from more than one modality (in this benchmark, text and visual).
• Technical level: Intermediate. Readers need some familiarity with graph neural networks, multimodal representation learning, and standard evaluation metrics, but the paper is written as a benchmarking and comparison study rather than a derivation-driven methods paper.
• Scope (1 sentence): The paper introduces OpenMAG, a unified, open-source benchmark for MAG learning built from 19 datasets across 6 domains, 16 modality encoders, 24 standardized models, 8 downstream tasks, and a 5-dimensional evaluation protocol, from which the authors derive 14 stated insights.

What This Paper Is About

MAG research has produced many new models, but the existing benchmarks used to compare them cover only a few application domains, rely on frozen (non-trainable) feature encoders, include a narrow set of model families, and mostly evaluate supervised graph-based tasks. OpenMAG's goal is to fix that by assembling a much broader dataset repository, supporting both frozen and trainable encoders, standardizing a large library of models, and evaluating them across both graph-based and modality-based (including generative) tasks under one protocol.

Key Contributions

1. A comprehensive benchmark. OpenMAG integrates 19 datasets across 6 domains, supports 16 encoders ranging from frozen extractors to trainable backbones, and standardizes 24 MAG learning models and 8 downstream tasks in one framework.
2. A five-perspective evaluation protocol. The paper defines and runs systematic experiments on Necessity, Data Quality, Effectiveness, Robustness, and Efficiency, organized around 10 research questions (Q1–Q10).
3. Fourteen distilled insights (C1–C14). Findings are condensed into actionable conclusions about modality preference, fine-tuning, encoder choice, model paradigms, robustness, and complexity trade-offs.
4. An open-source benchmark library. The authors release OpenMAG as a user-friendly library with documentation and modular interfaces, hosted at the anonymous repository link given in the paper, so researchers can add custom methods or datasets.

Main Findings

• Multimodality beats unimodality, but the preferred modality depends on the domain (C1, C2). Across all datasets, multimodal settings outperform unimodal baselines. Textual features excel on e-commerce datasets (product descriptions), while visual features dominate in social scenarios (driving user interaction).
• Graph structure helps multimodal generation, up to a point (C3, C4). On the G2Image task, adding a moderate number of aggregated neighbors generally improved scores: on SemArt, CLIP-Score rose from 68.58 (0 neighbors) to 70.21 at 4 neighbors (+1.63) and DINO-Score from 50.86 to 55.29 at 4 neighbors (+4.43); on Toys, CLIP-Score rose from 57.83 to a peak of 60.19 at 6 neighbors (+2.36) and DINO-Score from 28.13 to 30.83 at 6 neighbors (+2.70). With 8 neighbors, SemArt CLIP-Score fell to 67.10 (below the 0-neighbor baseline) and SemArt DINO-Score fell to 50.57, which the authors attribute to semantic noise from over-dense neighborhoods.
• Fine-tuning beats frozen encoders, at a cost (C5). On node classification with an MLP baseline, strategies involving fine-tuning consistently outperform purely frozen baselines, which the authors attribute to domain shift in pre-trained models. This gain comes with increased training time and memory use.
• Unified encoders align better than independent ones (C6). On modality retrieval with the Flickr30k dataset, unified encoders jointly pre-trained on image–text pairs (e.g., CLIP, Qwen2.5) achieved better cross-modal alignment than stitching together separate unimodal models.
• Graph-enhanced models dominate most graph-based scenarios (C7). In Table 3, DGF achieved the best results on Node Classification for Movies (Accuracy 53.89, F1-score 41.45), Link Prediction on DY (MRR 77.28, Hits@3 92.51) and Bili_Dance (MRR 42.55, Hits@3 58.25), and Node Clustering on Toys (NMI 51.29, ARI 36.24) and RedditS (NMI 84.89, ARI 78.07). LGMRec achieved the best Link Prediction F1-score on Movies (46.26).
• Simple MLPs can beat complex models on retrieval (C8). In modality retrieval on Toys, MLP reached T2I MRR 99.46 and I2T MRR 99.45, higher than the other reported models, suggesting structural aggregation can introduce noise. Multimodal-enhanced models excel elsewhere: LGMRec led G2Image (CLIP-Score 68.47, DINO-Score 52.73), and DGF led G2Text (BLEU-4 6.83, CIDEr 44.28).
• MLLM-enhanced models underperform in discriminative settings (C9), which the authors attribute to a gap between generative pre-training objectives and structural reasoning.
• LoRA is the best parameter-efficient fine-tuning strategy tested (C10). On G2Text with Flickr30k, LoRA improved OPT to BLEU-4 6.63, ROUGE-L 30.00, CIDEr 46.23 (versus 6.01/29.36/41.51 with no fine-tuning), and improved LLaMA's CIDEr to 47.76 (versus 44.58 with none). Adapters improved some lexical matching but degraded semantic grounding in several settings (e.g., OPT CIDEr fell to 34.95).
• Robustness patterns differ by paradigm (C11, C12). On the Toys dataset, shuffling text or image features and flipping labels produced distinct vulnerability profiles: under text noise, DMGC collapsed catastrophically, while graph-enhanced models were more robust to label noise than standard GNN baselines. Under added spurious edges, GAT declined sharply, whereas SAGE and multimodal models stayed stable.
• Complexity separates the model families (C13, C14). As reported in Table 6, graph-enhanced models generally scale linearly with edges, though structure-learning variants such as DMGC and GSMN introduce quadratic O(|V|²) terms; multimodal-enhanced models are governed mainly by the number of modalities M and attention constants; MLLM-enhanced models incur quadratic O(S²) costs in sequence length plus large Θ memory. Empirically, traditional GNNs showed the lowest latency, graph-enhanced methods incurred substantial time overhead, MMGCN's multi-view graph construction caused severe memory bottlenecks, and LGMRec kept memory low relative to standard GNNs.

Methodology in Plain English

The authors first define the standard MAG pipeline in four stages: encode each modality into embeddings, fuse the modalities into one node representation, propagate information across graph neighbors, and project to a task output. They then build each piece of the benchmark around that pipeline. Datasets are drawn from six domains — e-commerce products, social media, art networks, video recommendation, book recommendation, and image networks — with topology constructed using domain-specific rules (for example, co-viewing relationships in social media and co-purchase records in e-commerce). Encoders are split into frozen ones (text: Sentence BERT, T5, Llama3.2; vision: ViT, DINOv2, Swinv2; vision-language: CLIP, Qwen-VL) and trainable ones (BERT and ViT fine-tuned end-to-end). Models are organized into a taxonomy of graph-enhanced, multimodal-enhanced, and MLLM-enhanced paradigms, with traditional unimodal GNNs as baselines. Evaluation runs 3 graph-based tasks (Node Classification, Link Prediction, Node Clustering) and 5 modality-based tasks (Modality Matching, Modality Retrieval, Modality Alignment, Graph-to-Text Generation, Graph-to-Image Generation), using metrics such as Accuracy, F1-score, MRR, Hits@K, NMI, ARI, BLEU-4, ROUGE-L, CIDEr, CLIP-Score and DINO-Score. The five evaluation dimensions are then applied as controlled experiments: comparing unimodal vs. multimodal and varying neighbor counts, comparing frozen vs. fine-tuned and unified vs. independent encoders, benchmarking all models across tasks, injecting modality, label, and structural noise, and measuring theoretical and empirical cost.

Why This Matters

• Impact on research: Prior comparisons were fragmented. The paper contrasts OpenMAG with MAGB (5 datasets, 2 domains, frozen encoders only, 4 unimodal GNNs, supervised graph-based tasks, evaluated on Effectiveness + Efficiency + Modality Preference) and MM-GRAPH (7 datasets, 3 domains, frozen encoders only, 7 models, supervised graph-based tasks, evaluated on Effectiveness + Quality + Necessity). OpenMAG covers 19 datasets, 6 domains, frozen and trainable encoders, 24 models, both unsupervised and supervised tasks, graph-based and modality-based tasks, and all five evaluation dimensions. This gives the field a single controlled setting in which claims about new MAG models can be checked.
• Real-world applications (the domains represented in the benchmark):
  • E-commerce product networks, where edges follow co-purchase records and text features matter most.
  • Social media platforms, where co-viewing relationships and visual features dominate.
  • Art and cultural heritage networks, used here for graph-to-image generation experiments.
  • Recommendation settings spanning video and book catalogs, plus image networks.
• Industry relevance: The efficiency and robustness experiments speak directly to deployment. Findings that multimodal-enhanced models offer the best accuracy–efficiency trade-off, that explicit multi-view graph construction in MMGCN causes severe memory bottlenecks, and that models relying on intrinsic node attributes resist spurious edges all inform practical architecture selection under resource and data-quality constraints.

Future Directions

1. Unifying multimodal and graph structures (from C1–C4). The authors argue that neither unimodal semantics nor pure topology suffices, and that current fusion mechanisms treat modalities and structure as separate entities. They call for unified architectures that inherently blend them.
2. Flexible and aligned modality encoding (from C5–C6). The paper identifies flexible fine-tuning and precise cross-modal alignment as open challenges; the truncated conclusion names this as the second future direction, building on the finding that fine-tuning helps but costs compute and that unified encoders align better than independent ones.
3. Separating robustness from edge reliability (from C11–C12). Since attention-based models overfit spurious connections while attribute-driven and self-supervised models stay stable, a natural next step is architectures that deliberately decouple representation learning from strict reliance on local edge reliability.
4. Reducing complexity in graph-enhanced and MLLM-enhanced models (from C13–C14). The authors note that these families are constrained by quadratic cost in structure learning or sequence processing, and state that graph-enhanced models require optimization to mitigate latency and memory overheads in practice.

Target Audience

Researchers and graduate students working on graph neural networks, multimodal learning, or graph foundation models who need a fair comparison point for new MAG methods; benchmark and evaluation researchers interested in protocol design across necessity, data quality, effectiveness, robustness, and efficiency; and applied practitioners in recommendation, social network, and e-commerce settings who need guidance on which model paradigm and encoder configuration to choose given their accuracy, robustness, and compute constraints.

Authors’ abstract

Multimodal-Attributed Graph (MAG) learning has achieved remarkable success in modeling complex real-world systems by integrating graph topology with rich attributes from multiple modalities. With the rapid proliferation of novel MAG models capable of handling intricate cross-modal semantics and structural dependencies, establishing a rigorous and unified evaluation standard has become imperative. Although existing benchmarks have facilitated initial progress, they exhibit critical limitations in domain coverage, encoder flexibility, model diversity, and task scope, presenting significant challenges to fair evaluation. To bridge this gap, we present OpenMAG, a comprehensive benchmark that integrates 19 datasets across 6 domains and incorporates 16 encoders to support both static and trainable feature encoding. OpenMAG further implements a standardized library of 24 state-of-the-art models and supports 8 downstream tasks, enabling fair comparisons within a unified framework. Through systematic assessment of necessity, data quality, effectiveness, robustness, and efficiency, we derive 14 fundamental insights into MAG learning to guide future advancements. Our code is available at https://github.com/YUKI-N810/OpenMAG.