InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillation

AI summary

Overview

Research area: Graph machine learning, specifically knowledge distillation from Graph Neural Networks (GNNs) into Multi-Layer Perceptrons (MLPs), with a focus on node classification on graphs.

Technical level: Intermediate. The paper assumes familiarity with message passing in GNNs, knowledge distillation, and semi-supervised node classification, but its central idea (weighting training by how structurally important a node is) is stated plainly.

Scope in one sentence: The paper proposes InfGraND, a distillation framework that ranks nodes by their structural influence over the graph and uses that ranking to weight what an MLP student learns from a GNN teacher, evaluated on seven graph benchmarks in both transductive and inductive settings.

What This Paper Is About

GNNs are accurate on graph data but expensive, because every prediction requires aggregating information from a node's neighbors (message passing). Plain MLPs are fast but ignore graph structure, so they are much weaker. Existing methods that distill a GNN teacher into an MLP student either make the student bigger/more complex or decide which nodes matter most using prediction confidence (entropy) rather than the graph's structure. InfGraND asks a different question instead: how structurally influential is a node within the graph, and can that influence guide the distillation?

Key Contributions

1. A taxonomy of GNN-to-MLP distillation methods. The authors categorize prior work into three groups and argue that existing approaches either add computational complexity (larger students, positional encoders, structure-aware tokenizers) or use graph-agnostic discrimination based on prediction uncertainty.
2. InfGraND, an influence-guided distillation framework. To the authors' knowledge, this is the first method to perform node-level discrimination in GNN-to-MLP distillation by computing node influence from the graph structure itself. The influence metric is parameter-free.
3. One-time multi-hop feature pre-computation. The student MLP receives pooled multi-hop neighborhood features computed offline, embedding structural awareness at the input level without adding inference-time overhead.
4. Rigorous evaluation across seven real-world homophilic graph benchmarks in both transductive and inductive settings, plus label-limited scenarios, ablations, and sensitivity analyses, showing InfGraND consistently outperforms prior GNN-to-MLP distillation methods.

Main Findings

• Structure-based influence identifies the nodes that matter for GNN generalization. When the training set was split into two class-balanced subsets of 25% of labeled nodes each (top 25% per class by influence vs. bottom 25%), models trained on high-influence nodes consistently outperformed those trained on low-influence nodes across Cora, Citeseer, and PubMed, and across GCN, GAT, and GraphSAGE teachers.
• InfGraND beats competing distillation baselines. The paper reports average gains of 0.9% over KRD in the transductive setting and 3.0% in the inductive setting, and 0.6% over HGMD. Comparisons with HGMD were restricted to the transductive setting with a GCN teacher because the official HGMD implementation available at the time supported only that setup.
• Large gains over a vanilla MLP. The paper reports an average improvement of 12.6% (transductive) and 9.3% (inductive) over the non-distilled MLP, averaged across the three teacher architectures.
• Distilled MLPs often surpass their own GNN teachers. In nearly all cases the distilled student exceeded the teacher's accuracy, which the authors present as a challenge to the assumption that expressive GNNs are always superior on graph data.
• Teacher accuracy does not determine student accuracy. On Amazon-Photo (transductive), GCN scored 90.7% vs. GAT's 87.6%, yet the GAT-distilled student reached 94.5% vs. the GCN-distilled student's 94.2%.
• Better accuracy at much lower inference cost. On the Citeseer dataset, InfGraND improved over GraphSAGE-L3 by 4.3%, over GCN-L2 by 3.8%, and over GAT-L2 by 4.8%, while being 8.56x, 6.84x, and 13.89x faster respectively. AdaGMLP required more time than InfGraND and GLNN because it relies on an ensemble of models for prediction. The authors do not report timing comparisons against FF-G2M, KRD, and HGMD because those share InfGraND's architecture and therefore have identical inference times.
• The method scales to large graphs. On OGBN-Arxiv with a GraphSAGE teacher, InfGraND outperformed the KRD baseline in both the transductive and inductive settings.
• Ablation results. When only influence guidance was used ("w/Influence", raw features instead of propagated features), accuracy was 83.4% on Cora, 74.6% on Citeseer, 81.1% on Pubmed, 92.1% on Amazon-Photo, 93.2% on Coauthor-CS, and 93.8% on Coauthor-Phy. When only feature propagation was used ("w/Propagation", influence loss weights set to zero), accuracy was 83.6%, 75.0%, 81.0%, 93.0%, 93.0%, and 94.3% respectively. The full model scored 84.0%, 75.2%, 81.3%, and 94.2% on Cora, Citeseer, Pubmed, and Amazon-Photo in the transductive setting. The inductive ablation rows are not included in the provided paper content.
• No benefit was observed from simply enlarging the student MLP. Contrary to prior reporting that wider GLNNs achieve higher accuracy at longer inference time, the authors did not observe this effect; even a small number of hidden dimensions was sufficient to reach 100% training accuracy, so they kept the hidden dimensions and layers of the distilled MLP methods fixed.

Methodology in Plain English

1. Train a teacher GNN normally. A standard supervised GNN (GCN, GAT, or GraphSAGE) is trained on the labeled nodes, then frozen.
2. Score every node by structural influence. The paper defines influence as how much a perturbation to one node's features changes another node's representation after several rounds of message passing — formally the L1 norm of an expected Jacobian, normalized so influences sum to one, then aggregated into a single Global Influence Score (GIS) per node scaled to [0,1]. Because the exact computation is intractable, the authors strip out nonlinear activations and weight matrices (following the simplification used in Simplifying Graph Convolutional Networks) and propagate features with the normalized adjacency matrix for k = 2 iterations. They then use cosine similarity between a node's original features and its 2-hop propagated embedding as an efficient, parameter-free influence indicator, applying MinMaxScaler globally per graph. This is a one-time preprocessing step, not a recurring cost at inference.
3. Pre-compute multi-hop features for the student. Using the same linear propagation, the authors generate feature matrices for multiple propagation steps (p = 0 to P) and average-pool them into a single matrix X̃. Average pooling is chosen instead of concatenation so the student's input dimension and parameter count do not grow.
4. Distill with influence-weighted losses. The student MLP is trained on a combination of two objectives. The supervised loss has a standard cross-entropy term on labeled nodes plus an influence-weighted cross-entropy term (weights δ1 and δ2). The distillation loss follows the homophily principle and uses KL divergence to pull the student's prediction for a node toward the teacher's predictions for that node's neighbors, weighting each neighbor by (γ1 + γ2 · GIS) / |N(v_i)|. The γ1 term keeps a baseline gradient from all neighbors while γ2 amplifies influential ones. The two losses are combined as λL_s + (1 − λ)L_d.
5. Evaluate fairly. Experiments used a fixed set of five random seeds with average performance over five runs, hyperparameters tuned with the WandB platform using validation accuracy, and random hyperparameter searches run until validation performance saturated for both baselines and InfGraND. Baselines were reproduced using their official public implementations.

Why This Matters

Impact on research. The paper reframes node prioritization in distillation from a graph-agnostic question ("how confident is the teacher?") to a graph-centric one ("how influential is this node in the structure?"). It also provides empirical evidence that high-influence nodes drive GNN generalization, and that distilled MLPs can outperform their own GNN teachers — both findings that invite reconsideration of assumptions about GNN superiority on graph data.

Real-world applications (from the paper's framing):

• Social networks modeling user interactions to identify misinformation within communities.
• E-commerce graphs capturing user-product relationships for personalized recommendations that increase revenue.
• Retrieval-Augmented Generation (RAG) systems that use graphs as external knowledge for Large Language Models, enabling factual grounding and contextual reasoning.
• Latency-critical or resource-constrained deployments, where per-prediction message passing creates training and inference bottlenecks.

Industry relevance. The paper explicitly motivates the work by noting that GNNs' high computational and memory demands have limited their adoption in production, while industry often falls back to MLPs. InfGraND targets that exact trade-off: the feature propagation step is described as an offline, pre-computed operation analogous to pre-computed embedding tables managed by a parameter server, so the deployed model remains a plain MLP with no inference-time overhead.

Future Directions

The paper states that Section 6 (Conclusion) covers a summary of contributions and a discussion of future research directions, but the provided content is truncated before those details, so the specific directions proposed by the authors are not reported here. The work logically raises the following open questions:

• Extending beyond homophilic graphs. All seven benchmark datasets are described as homophilic, and the distillation loss relies explicitly on the homophily principle. Whether influence-guided distillation transfers to heterophilic graphs is not reported.
• Scaling the HGMD comparison. The comparison against HGMD was limited to the transductive setting with a GCN teacher because the public implementation available at the time supported only that configuration; the full comparison remains open.
• Generality of the influence surrogate. The influence score is approximated by removing nonlinearities and weight matrices and using cosine similarity after 2-hop propagation, with a sensitivity analysis over the number of propagation steps reported separately in the paper's appendix. Whether richer or learned influence estimators would improve results is not reported.
• Interaction between student capacity and distillation gain. The authors found that enlarging the student MLP did not consistently help in their experiments, leaving open what determines the ceiling of distillation gains when the student is already able to reach 100% training accuracy.

Target Audience

Researchers and practitioners working on graph machine learning, model compression, and efficient inference — particularly those interested in knowledge distillation, GNN deployment under latency or resource constraints, and semi-supervised node classification. The paper is most useful to readers who already understand message passing in GNNs and basic distillation objectives, and to engineers looking for a distillation recipe that leaves a plain MLP at serving time.

Authors’ abstract

Graph Neural Networks (GNNs) are the go-to model for graph data analysis. However, GNNs rely on two key operations - aggregation and update, which can pose challenges for low-latency inference tasks or resource-constrained scenarios. Simple Multi-Layer Perceptrons (MLPs) offer a computationally efficient alternative. Yet, training an MLP in a supervised setting often leads to suboptimal performance. Knowledge Distillation (KD) from a GNN teacher to an MLP student has emerged to bridge this gap. However, most KD methods either transfer knowledge uniformly across all nodes or rely on graph-agnostic indicators such as prediction uncertainty. We argue this overlooks a more fundamental, graph-centric inquiry: "How important is a node to the structure of the graph?" We introduce a framework, InfGraND, an Influence-guided Graph KNowledge Distillation from GNN to MLP that addresses this by identifying and prioritizing structurally influential nodes to guide the distillation process, ensuring that the MLP learns from the most critical parts of the graph. Additionally, InfGraND embeds structural awareness in MLPs through one-time multi-hop neighborhood feature pre-computation, which enriches the student MLP's input and thus avoids inference-time overhead. Our rigorous evaluation in transductive and inductive settings across seven homophilic graph benchmark datasets shows InfGraND consistently outperforms prior GNN to MLP KD methods, demonstrating its practicality for numerous latency-critical applications in real-world settings.