Global-Lens Transformers: Adaptive Token Mixing for Dynamic Link Prediction

AI summary

Overview

• Research area: Dynamic graph learning, specifically continuous-time dynamic link prediction using Transformer-style architectures.
• Technical level: Advanced (assumes familiarity with self-attention, temporal graph neural networks, and complexity analysis).
• Scope: The paper proposes GLFormer, an attention-free Transformer-style framework built on an adaptive token mixer and a hierarchical aggregation module, and evaluates it against vanilla Transformers, pooling, and MLP token mixers across five dynamic-graph backbones and six benchmarks.

What This Paper Is About

Transformer-based models are the dominant approach for dynamic graph link prediction because self-attention captures long-range temporal dependencies, but that mechanism scales quadratically with sequence length, which is expensive on large or high-frequency graphs. The paper asks whether self-attention is actually necessary in this setting, motivated by prior findings that Transformers' success owes more to their overall architecture than to attention itself. The goal is to design a lightweight, attention-free Transformer-style model for dynamic graphs that matches or beats attention-based baselines while being substantially faster at inference.

Key Contributions

1. GLFormer: an attention-free Transformer-style framework for dynamic graph learning. It replaces self-attention with an adaptive token mixer that uses interaction order and time intervals to aggregate neighbors, aiming for both efficiency and robustness on evolving neighbor sequences.
2. A hierarchical aggregation mechanism. By stacking local token mixers across layers with progressively enlarged offset ranges, the model expands its temporal receptive field and captures long-range dependencies without the cost of global attention.
3. Controlled evidence that attention is not essential in this domain. The authors replace self-attention with pooling or MLP modules across five Transformer baselines (TGN, TCL, TGAT, CAWN, DyGFormer) and show in Figure 2 that attention-free variants often match the original Transformer on four datasets.
4. Comprehensive evaluation on six benchmarks. Experiments on Wikipedia, Reddit, MOOC, LastFM, SocialEvo, and Enron report AP and AUC-ROC, along with inference-time and ablation analyses, with the code released at https://github.com/Hope-Rita/GLFormer.

Main Findings

• Attention-free variants are competitive: Replacing self-attention with pooling or MLP modules still often matches the original Transformer's performance across four datasets (Figure 2), supporting the paper's premise that the architecture, not attention, drives much of the gain.
• GLFormer achieves the best average rank on most backbones: Under AP, GLFormer's average rank is 1.67 (TGN), 1.17 (TCL), 1.17 (TGAT), 2.5 (CAWN), and 1 (DyGFormer), versus the highest vanilla self-attention rank shown at 3.33 (TCL AP).
• Best results on DyGFormer: With the DyGFormer backbone, GLFormer ranks 1 on both AP and AUC-ROC, reaching AP of 99.03 on Wikipedia, 99.24 on Reddit, 87.87 on MOOC, 93.34 on LastFM, 94.76 on SocialEvo, and 92.62 on Enron.
• Larger gains on some backbones: Under TGN with AP, GLFormer improves LastFM from 77.07 to 84.96 and Enron from 87.50 to 88.48 over the vanilla Transformer. Under TGAT with AP, Enron improves from 71.12 to 75.48.
• Not uniformly superior: For the CAWN backbone, GLFormer ranks 2.5 for both AP and AUC-ROC, and on AP for Wikipedia (98.67 vs 98.76), MOOC (81.37 vs 80.15 vanilla), and SocialEvo (84.96 vs 84.96) it does not lead the vanilla Transformer on every dataset.
• Layer depth depends on the backbone: Using offsets s¹, s², s³ set to 2, 4, and 8, the authors find two or three GLFormer layers are generally sufficient. TGAT works well with two layers, while sequence-based models like TCL and DyGFormer benefit from three layers to capture long-range dependencies.
• Inference efficiency: GLFormer's per-layer cost is O(N·K_l·d) ≈ O(N·K_l), with O(K_l) kernel parameters per layer and O(Σ N·K_l) total over L layers, compared with O(N²) for full self-attention. Figure 5 reports log-scale inference (evaluation) time across four datasets, and the authors describe GLFormer as offering a superior balance between efficiency and performance.
• Ablation results: Removing learnable parameters (w/o LP), removing relative times (w/o RT), swapping GELU for ReLU (w/ ReLU), removing the residual network (w/o ResNet), and removing channel mixers (w/o CM) all affect performance. The paper reports that LP captures temporal order and structure while RT models event timing, and that GELU, channel mixers, and residual connections are all important for robustness and training stability.

Methodology in Plain English

The model starts from embeddings produced by an existing dynamic graph method (the "backbone"), such as a memory network, temporal graph attention, random walks, or a sequential model. For a given node, these embeddings form a sequence of neighbor tokens.

Instead of letting every token attend to every other token, GLFormer mixes each token only with its M most recent neighbors. Each neighbor's weight combines two things: a learnable per-offset parameter that encodes the importance of interaction order, and a time-based weight computed by a softmax over negative relative time gaps that encodes the importance of timing. A learnable parameter β balances the two. Positions near the start of the sequence that have fewer than M past neighbors simply sum over the valid indices, and causally invalid indices are masked.

On top of this token mixer, the model keeps a standard channel-wise feed-forward block with layer normalization and residual connections, mirroring the non-attention parts of a Transformer. To reach further back in time, later layers use wider offset ranges defined by R_l = {p : s^(l−1) ≤ p ≤ s^l}, with kernel size K_l = s_l − s_{l−1} + 1 determining how many recent neighbors are used per layer. Stacking these layers enlarges the receptive field without pairwise comparisons inside the window.

Node representations are produced by mean-aggregating the layer outputs, and link prediction uses an MLP over the concatenation of the two endpoint representations, trained with binary cross-entropy and 1:1 negative sampling. Inference time is reported instead of training time to avoid confounding differences in training strategies across methods.

Why This Matters

Impact on research. The paper adds to a body of work (Metaformer, Informer, pyramid attention) suggesting that the Transformer block matters more than attention itself, and it is among the first to test this directly for dynamic graphs. It also provides a concrete efficiency argument: a per-layer cost of O(N·K_l) versus O(N²) for self-attention.

Real-world applications (as named or implied by the paper):

• Traffic systems, where interactions are continuously generated and inference must keep up with the event stream.
• Social networks, where the paper frames evolving relationships as temporally ordered events.
• Recommendation platforms, which the paper cites as a domain where dynamic graph learning plays a central role.
• The paper also mentions fraud detection as a task that depends on modeling these dynamics.

Industry relevance. The authors argue the accuracy/efficiency tradeoff makes GLFormer well-suited to real-time applications where both speed and accuracy matter. The model is released as code and is designed as a drop-in replacement for the attention module in existing backbones (TGN, TCL, TGAT, CAWN, DyGFormer), so it can be adopted without redesigning an existing pipeline.

Future Directions

• Inductive deployment. The paper evaluates only the transductive setting, where future interactions are predicted between nodes seen during training. Whether the adaptive token mixer transfers to inductive settings is not reported.
• Scaling to larger or higher-frequency graphs. The complexity argument favors GLFormer, but the benchmarks used (Wikipedia, Reddit, MOOC, LastFM, SocialEvo, Enron) are from one public collection, and the paper does not report results on web-scale graphs.
• Tuning the hierarchical schedule. Only s¹, s², s³ = 2, 4, 8 with 1–3 layers is explored. Other growth schedules, kernel sizes, or deeper stacks are an open question the paper does not resolve.
• Closing the CAWN gap. GLFormer ranks 2.5 for both metrics on the CAWN backbone and does not beat the vanilla Transformer on every CAWN dataset, leaving room to understand when attention remains preferable.
• Generalization beyond link prediction. The paper states the architecture is designed specifically for link prediction; whether the same mixer helps other dynamic-graph tasks such as recommendation or fraud detection is not tested.

Target Audience

Researchers and practitioners working on temporal or dynamic graph neural networks, especially those deploying link prediction on high-frequency event streams where inference cost matters. It is also relevant to readers interested in efficient Transformer variants and the broader debate about whether self-attention is the source of Transformer performance. Familiarity with self-attention, temporal graph models, and asymptotic complexity is assumed, so the paper is best suited to readers at an intermediate-to-advanced level.

Authors’ abstract

Dynamic graph learning plays a pivotal role in modeling evolving relationships over time, especially for temporal link prediction tasks in domains such as traffic systems, social networks, and recommendation platforms. While Transformer-based models have demonstrated strong performance by capturing long-range temporal dependencies, their reliance on self-attention results in quadratic complexity with respect to sequence length, limiting scalability on high-frequency or large-scale graphs. In this work, we revisit the necessity of self-attention in dynamic graph modeling. Inspired by recent findings that attribute the success of Transformers more to their architectural design than attention itself, we propose GLFormer, a novel attention-free Transformer-style framework for dynamic graphs. GLFormer introduces an adaptive token mixer that performs context-aware local aggregation based on interaction order and time intervals. To capture long-term dependencies, we further design a hierarchical aggregation module that expands the temporal receptive field by stacking local token mixers across layers. Experiments on six widely-used dynamic graph benchmarks show that GLFormer achieves SOTA performance, which reveals that attention-free architectures can match or surpass Transformer baselines in dynamic graph settings with significantly improved efficiency.