MyGram: Modality-aware Graph Transformer with Global Distribution for Multi-modal Entity Alignment

AI summary

Overview

Research area: Artificial Intelligence / Knowledge Graph Fusion — specifically multi-modal entity alignment (MMEA), which links equivalent entities across multi-modal knowledge graphs.

Technical level: Advanced. The paper assumes familiarity with graph neural networks, attention mechanisms, contrastive learning objectives, and linear algebra concepts such as Gram matrices and determinants.

Scope: This paper proposes MyGram, a framework that combines modality-aware graph convolutional diffusion with a Transformer and a Gram-based geometric regularization loss to align entities across knowledge graphs that contain structural, relational, attribute, and visual modalities.

What This Paper Is About

Multi-modal knowledge graphs built from different sources often describe the same real-world entity in different ways, so systems need to automatically identify which entities across two graphs refer to the same thing. Existing methods tend to rely on point-wise contrastive learning and shallow modality features, which makes them vulnerable to confusion between visually or attributively similar but semantically distinct entities. MyGram's goal is to capture deeper structural context within each modality and to enforce global consistency across modalities so that equivalent entities are matched more reliably.

Key Contributions

1. A modality-aware graph diffusion Transformer framework (MyGram) that integrates graph diffusion with a Transformer architecture to obtain structurally contextualized, modality-specific features for entity alignment.

2. A Gram-based global alignment strategy that minimizes the volume of a 4-dimensional parallelotope formed by multi-modal embeddings, enforcing global distribution alignment rather than only point-wise feature matching.

3. A modality-aware graph convolutional diffusion module (MGD) that propagates multi-hop neighborhood information separately within each modality and then fuses modalities through a multi-head attention mechanism to support finer-grained fusion.

4. Extensive empirical validation on cross-knowledge-graph datasets (FB15K-DB15K, FB15K-YG15K) and the bilingual DBP15K dataset, reporting maximum Hits@1 improvements of 4.8% on FBDB15K, 9.9% on FBYG15K, and 4.3% on DBP15K over the second-best models.

Main Findings

• State-of-the-art alignment accuracy: MyGram achieves the best result on all metrics across the reported benchmark experiments. At 80% seed on FB15K-DB15K it reaches MRR .879, Hits@1 .842, Hits@10 .948; on FB15K-YG15K it reaches MRR .836, Hits@1 .783, Hits@10 .938. On DBP15K it reaches MRR .876 / Hits@1 .833 / Hits@10 .960 (ZH-EN), .879 / .836 / .964 (JA-EN), and .908 / .869 / .979 (FR-EN).

• Improvements over the second-best models: The paper reports maximum Hits@1 gains of 4.8% on FBDB15K, 9.9% on FBYG15K, and 4.3% on DBP15K. MyGram is reported to outperform state-of-the-art methods including SimDiff and IBMEA.

• All modalities help, and relations matter most: In the 80%-seed modality ablation, removing any single modality degrades performance. Removing the relational modality causes the most significant drop on both datasets, followed by attributes, then images.

• Both core components contribute: The component ablation shows significant degradation when the modality-aware graph convolutional diffusion module is removed, and a performance effect when the Gram-based loss is removed, supporting the value of both structural propagation and cross-modal semantic coherence.

• Robustness under low-resource conditions: When seed proportions are varied from 5% to 30%, all models degrade as seeds shrink, but MyGram maintains its performance advantage over the compared models.

• Case-level behavior: In a case study on the entity Shang Hai from FB15K-DB15K, where Shang Hai and Hong Kong share similar modality features, MyGram correctly identifies the match while MEAformer and PMF assign a lower rank to the correct entity.

Methodology in Plain English

MyGram processes each entity along four channels: structure (graph neighborhoods), relations, attributes, and image features. Structure is encoded with a relational reflection graph attention network (RRGAT); relation and attribute features are turned into vector representations from bag-of-words features; images are encoded with a pretrained image encoder, specifically VGG-16.

Each modality is then passed separately through a diffusion module that repeatedly propagates information along the (normalized, self-looped) graph adjacency, blending each propagation step with the original features. This gives each modality a version of the entity that reflects its multi-hop neighborhood rather than just its own surface features, which is how the method addresses interference from entities that look or read similarly.

The enriched modality features are then fed into a Transformer-style multi-head self-attention mechanism so the modalities can exchange information and be weighted dynamically. The resulting per-modality weights are combined with the structural embedding to produce a joint representation.

For training, the paper departs from purely point-wise contrastive objectives. For each entity pair, the source entity's structural feature and the target entity's visual, attribute, and relation features are stacked into a 4 × 4 Gram matrix. The determinant of this matrix gives the squared volume of the 4-dimensional parallelotope those vectors span; a smaller volume means the vectors lie in a tighter subspace, which the authors treat as greater cross-modal coherence. The Gram loss is defined over these volumes in a softmax-style objective with a temperature coefficient, and the total loss combines it with an InfoNCE contrastive loss weighted by a factor λ.

Experimental settings: hidden layer size fixed at 300 dimensions across network components, 1000 training epochs, initial learning rate 5e-3, visual embedding dimension 4096, Transformer intermediate dimension 400, and 5 self-attention heads. Cross-KG experiments use 20%, 50%, and 80% of alignment pairs as training seeds; the bilingual DBP15K setting uses 30%. Entities without images receive randomly initialized visual vectors. Evaluation uses Hits@1, Hits@10, and MRR.

Why This Matters

Research impact. The paper reframes cross-modal alignment as a global geometric distribution-matching problem rather than a set of pairwise distance constraints. The Gram-volume formulation offers a concrete alternative regularization signal for multi-modal representation learning, and the modality-specific diffusion design shows that per-modality structural context can be captured before fusion instead of being averaged away during it. The paper reports comparisons against 14 named baselines, which gives the community a broad reference point.

Real-world applications:

• Knowledge graph fusion and integration across heterogeneous sources such as Wikidata, DBpedia, Freebase, and YAGO.
• Intelligent question answering and search, where cross-source entity linking determines whether retrieved facts refer to the same object.
• Recommendation systems that combine text and images with structured knowledge, where entity resolution across catalogs is a prerequisite.
• Cross-lingual knowledge services, since the DBP15K experiments cover ZH-EN, JA-EN, and FR-EN alignment.

Industry relevance. Any organization maintaining multiple knowledge bases, product catalogs, or content repositories faces duplicate-entity problems. A method that works with limited pre-aligned seed pairs (the paper tests 5% to 30%) is directly relevant to settings where labeled alignment data is scarce and manual curation is expensive. The code is released at https://github.com/HubuKG/MyGram.

Future Directions

• Integration with large language models. The conclusion states that the authors plan to explore integrating LLMs to further enhance MMEA.
• Extending the Gram formulation beyond four modalities. The loss is built on a 4-dimensional parallelotope over structure, visual, attribute, and relation features; whether the approach generalizes to additional modalities (for example audio, as the introduction mentions multi-modal knowledge graphs containing it) is left open.
• Understanding the component trade-offs. The paper reports that removing relational information hurts most, but does not report a detailed analysis of how the diffusion depth, the λ weighting term, or the number of attention heads interact with that finding.
• Scaling and stress-testing in unsupervised settings. The low-resource study covers 5% to 30% seed ratios; behavior closer to fully unsupervised alignment is not reported.

Target Audience

Researchers and practitioners working on knowledge graph fusion, entity alignment, multi-modal representation learning, and graph neural networks. It is also relevant to engineers building cross-source knowledge integration pipelines who need robustness under limited supervision. Readers without a background in graph convolution, attention mechanisms, and contrastive objectives will find the methodology sections demanding, particularly the Gram matrix and determinant-based loss derivation.

Authors’ abstract

Multi-modal entity alignment aims to identify equivalent entities between two multi-modal Knowledge graphs by integrating multi-modal data, such as images and text, to enrich the semantic representations of entities. However, existing methods may overlook the structural contextual information within each modality, making them vulnerable to interference from shallow features. To address these challenges, we propose MyGram, a modality-aware graph transformer with global distribution for multi-modal entity alignment. Specifically, we develop a modality diffusion learning module to capture deep structural contextual information within modalities and enable fine-grained multi-modal fusion. In addition, we introduce a Gram Loss that acts as a regularization constraint by minimizing the volume of a 4-dimensional parallelotope formed by multi-modal features, thereby achieving global distribution consistency across modalities. We conduct experiments on five public datasets. Results show that MyGram outperforms baseline models, achieving a maximum improvement of 4.8% in Hits@1 on FBDB15K, 9.9% on FBYG15K, and 4.3% on DBP15K.