Residual Paths and Deep Network Optimization

A plain block must transform its input into the entire desired output. A residual block can preserve the input and learn only a correction.

The identity route also gives gradients a shorter path through many layers. This can improve trainability, but the residual branch can still dominate, vanish, or waste compute.

The shortcut was published in December 2015, and it bought depth immediately. He and colleagues write that they “evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity”, and that “An ensemble of these residual nets achieves 3.57% error on the ImageNet test set”. Depth of that order had not been trainable before. The rest of this lesson is about what the shortcut does and does not buy.

Both routes contribute to the next representation. A tensor enters the shortcut and the residual branch together. Layers on the branch compute a learned correction under normalization and activation. A projection or resizing makes the two routes compatible when shapes differ. The addition combines the identity information with the learned change. Later blocks can preserve, refine, or redirect what comes out.

The second route is interesting only because the first is left alone. Keep the skip connection a clean identity, put the activation after the addition, and “the forward and backward signals can be directly propagated from one block to any other block”. That is the claim the same four authors made in their 2016 follow-up on identity mappings. Every design choice below — projection, normalization order, branch scaling — is a choice about how much of that clean route survives.

Depth remains an engineering choice with measurable costs, and measurable is meant literally. Delete one layer from a trained 15-layer VGG and it collapses. Delete one module from a trained 110-layer residual network and almost nothing happens. Veit and colleagues ran both experiments on CIFAR-10 in 2016: “As expected, deleting any layer in VGG reduces performance to chance levels. Surprisingly, this is not the case for residual networks.” Removing residual modules changed nothing noticeable except at downsampling blocks. Most of the gradient in that 110-layer network came from paths only 10-34 layers deep, and a network trained on those effective paths alone reached 5.96% error against the full model's 6.10%. Six years later three researchers at Google treated the finding as settled: “It is known from prior research (Veit et al., 2016) that residual blocks in a ResNet can be removed without substantially hurting accuracy”. They attribute it to identity skip responses attaining larger values than the residual branches.

Exact behavior depends on the architecture and on how the block is built. The differences have been measured. The measurements do not all point the same way.

Block order. The pre-activation unit moves normalization and activation inside the residual branch and leaves the shortcut a clean identity. Its authors report what that bought: “We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100, and a 200-layer ResNet on ImageNet.” The comparison is against their own 1202-layer post-activation network at 7.93% on the same dataset — nearly twice the error at slightly greater depth. An independent table lists the pre-activation entry too: depth 1001, 10.2M parameters, 4.62 on C10+, with ResNet-110 at 6.61 and 6.41. Post-normalization is not merely riskier at depth. In some architectures it does not train at all. Touvron and colleagues, in 2021: “Note that the original architecture of Vaswani et al. [67] applies the normalization after the block, but in our experiments the DeiT training does not converge with post-normalization.”

Branch scaling. The same paper walks DeiT-S from 12 to 36 blocks and hits a wall: 69.9% top-1, marked training failed, at 64M parameters and 13.8B FLOPs. Adding LayerScale recovers 80.5% at the identical 64M parameters and 13.8B FLOPs. LayerScale is a learnable per-channel diagonal on each residual branch, initialized to 0.1 up to depth 18, 1e-5 at depth 24 and 1e-6 beyond. Nothing about the model changed except the scale on each branch. timm ships it as a standard module, whose LayerScale and LayerScale2d classes default to init_values = 1e-5. Depth alone is not the whole story. In the same paper's Table 1, a 36-block DeiT-S baseline does converge, to 81.9%, once the stochastic-depth drop rate is raised to 0.25. What LayerScale buys is a deep stack that trains without that hand-tuning.

One number per connection is the smaller version of the same idea. Bachlechner and colleagues gate each residual connection with a single parameter initialized to zero, and call it ReZero. Their abstract: “We show that the simplest architecture change of gating each residual connection using a single zero-initialized parameter satisfies initial dynamical isometry and outperforms more complex approaches.” They add: “We apply this technique to language modeling and find that we can easily train 120-layer Transformers. When applied to 12 layer Transformers, it converges 56% faster.”

Two independent evaluations then disagreed about the zero. The same team's Table 1 scores ReZero at 78.3 / 80.1 / 80.8 / 81.6 for depths 12/18/24/36 on ImageNet-1k, against LayerScale's 80.5 / 81.7 / 82.4 / 82.9. The 78.3 at depth 12 sits below the 79.9% DeiT baseline — and their ReZero was an adapted one, with LayerNorm and warmup re-introduced because the original did not converge. Their conclusion: “it is better to initialize alpha to a small value instead of 0, as in LayerScale”. In the DeepNet benchmark, ReZero reaches 26.9 BLEU at 6L-6L on WMT-17 En-De, the weakest entry in the table, and diverges at 18L-18L. Those authors blame instability under half-precision training rather than depth.

That is what checking a scale schedule means in practice. One scheme won on the architecture it was designed for and lost on two others. The only way to know which case you are in is to run the comparison at a matched budget.

Use module-level evidence rather than one global curve. Measure branch ratios, comparing residual-branch norm with shortcut and block-output norms. Trace gradients by depth, checking whether early and late blocks receive usable update ratios. Ablate block groups by removing, bypassing or freezing ranges of depth and re-evaluating. Test scaling choices — initialization, gating, residual multipliers — under matched budgets. Then price the benefit: validation gain per unit of memory, latency and training compute.

Step three says groups, not blocks, and two studies explain why. Layers that survive re-initialization one at a time stop surviving it in groups. The Google study that re-initialized the layers of trained networks in 2022 reports both halves: “In summary, the individually robust layers are generally not jointly robust. But with some clever way of picking out a subset of the layers, joint robustness could still be achieved for up to half of the layers.” ResNet-18 was relatively robust. Deeper ResNets were not jointly robust under the same grouping. The graded version of the effect had been measured six years earlier, on a 54-block network: “Performance only drops slightly up to the removal of 10 residual modules, however, for the removal of 20 modules, we observe a severe drop in performance”.

One-block ablations will therefore tell you that almost every block is expendable. Ten of fifty-four are. Twenty are not.

Initialization, normalization, learning rate, precision, and optimizer state still determine whether residual branches learn useful corrections. Deep stacks can also suffer representation collapse or oversmoothing in specialized architectures.

The first diagnostic in the previous section has a name and a formula. For a block x+f(x), the block's dependency on its residual branch is Var[f(x)]/Var[x+f(x)]. Liu and colleagues defined it that way in 2020 and priced both ends of the ratio: “heavy dependency on its residual branch makes training unstable, since it amplifies small parameter perturbations (e.g., parameter updates) and results in significant disturbances in the model output. Yet we observe that a light dependency limits the model potential and leads to inferior trained models.” Their numbers say the same thing. On WMT'14 En-De, Post-LN failed outright at 12L-12L and at 18L-18L. Pre-LN converged there, but only to 28.26 and 28.38 BLEU. Their Admin initialization reached 28.58 and 29.03. On WMT'14 En-Fr, Admin stabilized a 72-layer model — 60-layer encoder, 12-layer decoder — to 43.80 BLEU where Post-LN failed and Pre-LN reached 43.10. The DeepNet authors reproduced the shape of that result on WMT-17 En-De: vanilla Post-LN diverged at 18L-18L, Admin held to 28.8 there and diverged at 50L-50L, and vanilla Pre-LN converged at every depth but peaked lower. Stability and ceiling are traded against each other, and the ratio is where the trade is visible.

The lever that moves it can be very small. Xiong and colleagues proved in 2020 that for Pre-LN Transformers “the gradients are well-behaved at initialization”, which “motivates us to remove the warm-up stage for the training of Pre-LN Transformers”, and reported comparable results without it. An entire training stage, deleted by a block reordering. Goyal and colleagues had reached for something smaller still in 2017: “the learnable scaling coefficient γ is initialized to be 1, except for each residual block’s last BN where γ is initialized to be 0”. That one line is shipped code. torchvision's ResNet takes zero_init_residual, and the source comment says why: “Zero-initialize the last BN in each residual branch, so that the residual branch starts with zeros, and each residual block behaves like an identity. This improves the model by 0.2~0.3% according to https://arxiv.org/abs/1706.02677”. Treat the shortcut as one part of a training recipe and validate the complete block behavior.