If You Want to Be Robust, Be Wary of Initialization

AI summary

Overview

Research area: Adversarial robustness of Graph Neural Networks (GNNs), with an extension to general Deep Neural Networks, sitting at the intersection of graph representation learning, optimization theory, and adversarial machine learning.

Technical level: Intermediate. The paper's core message is stated plainly (initialization and training length shape robustness), but the supporting argument rests on Lipschitz, smoothness, and convergence assumptions and on several derived upper bounds, so the theoretical sections assume some familiarity with optimization notation.

Scope in one sentence: The paper derives upper bounds that connect a model's adversarial risk to the norms of its initial weights and its number of training epochs, then tests those bounds with adversarial attacks on real graph datasets.

What This Paper Is About

Most defenses against adversarial attacks on GNNs change the input graph (pre-processing, edge pruning, low-rank approximation) or change how messages are passed between nodes. This paper looks instead at something almost nobody has examined: whether the random values chosen to initialize the network's weights, and how long the network is trained, determine how vulnerable the final model is.

The authors build a theory that ties adversarial risk to the size of the initial weights and the number of training epochs, then confirm the theory experimentally by attacking GCNs, GINs, and ordinary DNNs under several initialization schemes.

Key Contributions

1. A theoretical analysis linking weight initialization strategies to adversarial robustness in GNNs, including an upper bound that depends on the norms of the initial weights and on the number of training epochs (Theorem 2 for node-feature attacks on GCNs, Theorem 3 for structural attacks).

2. A characterization of how the parameters of a Gaussian initialization distribution affect expected adversarial risk (Lemma 4), plus an analogous bound for GIN-based classifiers (Theorem 5) under a bounded input feature assumption.

3. A generalization of the analysis beyond graphs: a general upper bound applicable to any Deep Neural Network (Theorem 6), with a sharper bound when the loss is strongly convex and smooth (Lemma 7).

4. Extensive experiments across models and real-world datasets under several adversarial attacks, showing that certain initialization strategies improve defense without degrading clean accuracy, with observed gaps of up to 50% compared to alternative initialization approaches.

Main Findings

• Smaller initial weight norms mean greater robustness. The derived upper bounds grow with the norm of the initial weight matrices, so a lower initial norm corresponds to a tighter (smaller) bound and thus a more robust model. The authors note that setting all initial weights to zero would give the smallest bound but would harm learning, since it restricts back-propagation and leads to poor local minima.

• More training epochs means more vulnerability. The bounds scale with terms involving the number of epochs (expressed as 2^t, sharpenable to (1 + ηL)^t), so longer training loosens the bound and increases adversarial vulnerability. The authors describe this as evidence of the commonly discussed trade-off between clean accuracy and attacked accuracy.

• The epoch effect shows up as an inflection point in practice. On Cora and CiteSeer, clean accuracy rises and plateaus while attacked accuracy rises to an inflection point and then declines. The authors suggest stopping training at that inflection point.

• Higher Gaussian variance hurts robustness. With Gaussian initialization and the mean held constant, higher variance (σ) produced higher Success Rates, meaning more vulnerability, for both PGD and Mettack on Cora and CiteSeer. For certain budgets, including 30% and 40%, the observed gap ranges between 5% and 15%.

• Scaling parameters behave as predicted. For uniform initialization U(−β, β) and for a scaled orthogonal initialization, higher values of the scaling parameter β produced higher upper bounds and weaker empirical robustness on Cora and CiteSeer under PGD and Mettack.

• Structural attacks are affected more than feature attacks. The authors state that the effect of initialization is greater for structural perturbations, because the perturbed adjacency matrix acts at every propagation step, whereas node features enter directly only in the first propagation.

• The theory transfers to GINs and DNNs. Different initialization schemes (Orthogonal, Uniform, Kaiming, Xavier) produced similar clean accuracy at ε = 0 on the GIN (Cora) and the DNN (MNIST), but the gaps in attacked accuracy grew with the attack budget. For the DNN at ε = 0.1, the accuracy gap between the best and worst initialization methods is around 60%.

• Strong convexity changes the picture. When the loss is μ-strongly convex and L-smooth, the bound depends on (1 − μ/L)^t times the initial weight norm plus a term involving the final weights. Since μ ≤ L, increasing epochs diminishes the influence of initialization, leaving a bound that depends solely on the final weights.

• Zero initialization is not the answer. Although zero (or constant) initialization gives the smallest upper bound and therefore the best theoretical robustness, the paper states that it constrains weight behavior during propagation and leads to convergence to unsatisfactory local minima.

Methodology in Plain English

The authors start from the standard definition of adversarial risk for a graph classifier: the expected worst-case change in the model's prediction when the graph is perturbed within a budget ε. A model is called (ε, γ)-robust if that risk is bounded by γ.

They then assume a training setting that is standard but analyzable: activation functions that are 1-Lipschitz (such as tanh), a loss function that is L-smooth, and gradient descent with a learning rate no larger than 1/L. Under these assumptions, they track how the weight matrices evolve from their initial values W₀ toward the optimum W*, and use that trajectory to bound how much a perturbation at the input can change the output.

The result is an expression for γ in terms of the initial weight norms, the final weight norms, and the number of training epochs, multiplied by graph-specific terms. For node-feature attacks on GCNs, that graph term is a sum over nodes of normalized walk counts; for structural attacks, it involves the input feature norm and an additional term; for GINs, it involves a bound B on the feature space and the maximum node degree. The same style of derivation, applied without graph structure, yields the DNN bound.

For the experiments, they train a 2-layer GCN with fixed hyperparameters on the citation networks Cora and CiteSeer, repeat each experiment 10 times using the datasets' provided train/validation/test splits, and evaluate under two gradient-based structural attacks, Mettack (with the Meta-Self strategy) and PGD, at perturbation rates from 10% to 40% of edges (0.1|E| to 0.4|E|). Results for a Dice attack are reported in the appendix. They measure Attacked Accuracy and, where clean accuracy differences could obscure the picture, Success Rate. They then repeat the procedure with variable initialization variance, variable scaling parameters for uniform and orthogonal initialization, and finally with GIN and DNN models (the latter on MNIST) using Orthogonal, Uniform, Kaiming, and Xavier initialization. Code is released at https://github.com/Sennadir/Initialization_effect.

Why This Matters

The paper reframes adversarial robustness as partly a property of training setup rather than only of architecture or defense mechanism. If a robustness improvement can be obtained by changing how weights are drawn and how long training runs, that is a defense that costs essentially nothing at inference time and composes with existing defenses. It also gives a concrete reason to revisit default initialization and epoch choices, which are usually selected for convergence and clean accuracy alone.

Real-world applications named or implied by the paper's framing:

• Drug design, where graph models are used on molecular structures that could be deliberately or accidentally perturbed.
• Protein resistance analysis, where graph-based predictions inform biological conclusions.
• Session-based recommendation, where a manipulated interaction graph could steer recommendations.
• Tabular data modeling, another domain the paper cites for GNN application.
• Node classification on citation networks, the experimental setting used throughout.

Industry relevance: any group deploying GNNs or DNNs in a setting where inputs can be manipulated now has a cheap, testable lever (initialization scale and training length) to check alongside their existing defenses. The finding that the best and worst initializations can differ by large margins in attacked accuracy under identical clean accuracy means that standard clean-accuracy model selection can silently pick a fragile model.

Future Directions

• Designing new initialization schemes that improve adversarial robustness while preserving good generalization, which the authors explicitly name as an open direction.
• Developing new gradient-based weight update rules that enforce robustness during training rather than relying on the initialization alone.
• Tracking robustness metrics alongside the loss throughout training, so that the inflection point the experiments reveal could be detected automatically instead of found by trial and error.
• Identifying the balance point between epochs, stated as theoretically challenging to pin down, and extending the analysis to optimizers other than gradient descent, which the authors argue is possible by following a similar approach.

Target Audience

Researchers and practitioners working on adversarial robustness, graph representation learning, or GNN deployment who want to understand a defense dimension that sits outside the usual pre-processing and message-passing playbook. It is also relevant to machine learning engineers responsible for training recipes, since the practical recommendation concerns initialization scale and epoch count rather than architecture. Readers who only want the actionable takeaway can read the abstract, Section 4's theorem statements, and the experimental figures; readers who want to verify or build on the theory will need the appendix proofs.

Authors’ abstract

Graph Neural Networks (GNNs) have demonstrated remarkable performance across a spectrum of graph-related tasks, however concerns persist regarding their vulnerability to adversarial perturbations. While prevailing defense strategies focus primarily on pre-processing techniques and adaptive message-passing schemes, this study delves into an under-explored dimension: the impact of weight initialization and associated hyper-parameters, such as training epochs, on a model's robustness. We introduce a theoretical framework bridging the connection between initialization strategies and a network's resilience to adversarial perturbations. Our analysis reveals a direct relationship between initial weights, number of training epochs and the model's vulnerability, offering new insights into adversarial robustness beyond conventional defense mechanisms. While our primary focus is on GNNs, we extend our theoretical framework, providing a general upper-bound applicable to Deep Neural Networks. Extensive experiments, spanning diverse models and real-world datasets subjected to various adversarial attacks, validate our findings. We illustrate that selecting appropriate initialization not only ensures performance on clean datasets but also enhances model robustness against adversarial perturbations, with observed gaps of up to 50\% compared to alternative initialization approaches.