Residual, Dense, and Skip Connectivity

Without a shortcut, a block must learn the full desired mapping from its input. An additive residual block can learn something smaller: a change, measured against an identity or projected path.

That does not make deep networks easy. It creates shorter routes for signals and gradients. The residual branch still needs sensible normalization, initialization, and scale. Each of those three words is load-bearing, and the rest of this lesson attaches a measurement to each. A scaling factor between 0.1 and 0.3 decided whether a network trained at all. A reordering of the normalisation took a 1001-layer network to 4.62% error on CIFAR-10. An initialisation scheme reached 10,000 layers with no normalization layers in the model.

The phrase skip connection hides four distinct operations. Each of the four has a measurement attached to it rather than a slogan.

Residual addition is the parameter-free case. Those ResNets carry no extra parameter compared with their plain counterparts, and that addition alone is what turns 27.94 → 28.54 into 27.88 → 25.03 across the same two depths.

Dense concatenation buys explicit feature reuse and pays for it in channels. DenseNet, published in 2017, wires every layer to every later one, so a block of L layers has L(L+1)/2 direct connections. What the wiring bought is in the authors' own summary: “We also highlight that DenseNet-BC with L = 100 and k = 12 achieves comparable performance (e.g., 4.51% vs 4.62% error on C10+, 22.27% vs 22.71% error on C100+) as the 1001-layer pre-activation ResNet using 90% fewer parameters.” That is 0.8M parameters standing in for 10.2M at the same error. Their largest model, DenseNet-BC with L=190 and k=40 at 25.6M parameters, reaches 3.46% on augmented CIFAR-10 and 17.18% on augmented CIFAR-100. The trade survives outside the paper. The torchvision model documentation lists densenet201 at 20.0M parameters and 76.896 Acc@1 on ImageNet-1k, against resnet101 at 44.5M and 77.374.

Highway or gated paths put parameters on the carry route itself. Highway networks introduced LSTM-inspired adaptive gating units there in 2015. Table 6 of the ResNet paper puts the two designs side by side on CIFAR-10: Highway with 19 layers and 2.3M parameters at 7.54 per cent error, ResNet-110 with 1.7M at 6.43. The same paper states what separates them: “These gates are data-dependent and have parameters, in contrast to our identity shortcuts that are parameter-free. When a gated shortcut is "closed" (approaching zero), the layers in highway networks represent non-residual functions.” A gate that can close is a shortcut that can stop being one.

Encoder–decoder skips are the family a benchmark rewarded outright. U-Net, published in 2015, carries high-resolution encoder features across the bottleneck. As of 6 March 2015 that architecture sat at rank 1 of the ISBI 2012 EM segmentation leaderboard: “The u-net (averaged over 7 rotated versions of the input data) achieves without any further pre- or postprocessing a warping error of 0.0003529 (the new best score, see Table 1) and a rand-error of 0.0382.” The Ciresan sliding-window network scored 0.000420 and 0.0504 on the same board. An independent group later tabulated the same entry — "Image Analysis Lab Freiburg" — at warping error 352×10⁻⁶, Rand error 38×10⁻³ and pixel error 61×10⁻³. The challenge organisers describe what was being scored: 30 consecutive 512×512 ssTEM images, with 13 teams and 86 submissions before the ISBI'12 workshop, rising to 22 teams and 185 submissions. The same architecture won the ISBI cell tracking challenge 2015, with IOU 0.9203 on PhC-U373 and 0.7756 on DIC-HeLa. The second-best 2015 entries scored 0.83 and 0.46. The bottleneck still compresses. What crosses beside it is what the leaderboard rewarded.

Short routes coexist with longer transformed routes, and the short ones carry more of the training than the drawing suggests. In a 110-layer residual network, most of the gradient comes from paths only 10 to 34 layers deep — the first 30.9 per cent of the nominal depth. That window was measured in 2016.

The paths are also individually droppable, which is the strongest evidence that they are paths and not one computation. Drop residual blocks at random during training, bypassing them with the identity, and a 1202-layer residual network reaches 4.91% on CIFAR-10. The identical network trained at constant depth reaches 6.67%.

The correct shortcut depends on what must be preserved. Each choice below has been measured on a named benchmark rather than argued for.

The goal is to preserve useful access without creating an uncontrolled bypass. The fourth step is the one teams skip. The Inception-ResNet failure was found only by watching the last layer before the average pooling produce nothing but zeros after a few tens of thousands of iterations. Fixup removed the normalization layers altogether by setting the branch scale at initialisation instead. Branch magnitude is a quantity you choose. It is not one you inherit from the diagram.

A network may achieve acceptable loss by leaning on a bypass while the intended transformation learns little. That matters in bottlenecks, multimodal fusion, and conditional branches.

Inspect branch-specific activations, gradients, and ablations. End-to-end accuracy alone cannot reveal whether the architecture uses the path for the reason you intended.

The suspicion became an experiment in 2016. Veit and colleagues rewrite residual networks “as an explicit collection of paths” of differing length. Cut individual paths and the network barely notices: “a lesion study reveals that these paths show ensemble-like behavior in the sense that they do not strongly depend on each other”. The routes that matter are also far shorter than the model card: “most of the gradient in a residual network with 110 layers comes from paths that are only 10-34 layers deep”. The long path is still there in the diagram. It is not where the training signal went.