Residual Connections and Stable Depth

A basic residual block computes y=x+F(x). The branch F learns an update, while the shortcut carries the reference representation forward.

If the best local transformation is close to identity, the block can keep F small instead of reconstructing x through several nonlinear layers.

The problem this solves was measured before it was solved. A plain 56-layer network on CIFAR-10 reaches a higher training error than a plain 20-layer one: “the deeper network has higher training error, and thus test error”. That is degradation, not overfitting. The quantity that rises is the training error, and no amount of extra fitting capacity explains a worse fit. Kaiming He and three colleagues reported that in 2016, then rewrote the same stacks with shortcuts, at “exactly the same depth, width, and number of parameters as the plain counterparts”. The ordering reversed. Their 110-layer residual network reaches 6.43% test error on CIFAR-10. Their single 152-layer model reaches 4.49% top-5 validation error on ImageNet.

The headline result does not have to be taken from the paper that claims it. The challenge organisers kept their own scoreboard. On the official ILSVRC2015 results page, MSRA's “Ensemble A for classification and localization” sits first among entries using provided training data, with a classification error of 0.03567 and a localization error of 0.090178. MSRA's Ensemble B (0.03567/0.090801) and Ensemble C (0.0369/0.092108) follow. The best non-MSRA entry listed, Trimps-Soushen, records 0.04649. Results were released on 10 December 2015. So when the paper says “This entry won the 1st place in ILSVRC 2015.”, the 3.57% top-5 figure it reports is the 0.03567 sitting on the scoreboard beside it. The claim and the check are two different documents.

The merge requires compatible shapes and clear semantics.

Only one of the two routes is free. The branch can be given whatever internal structure the designer wants, because whatever it computes is called an update. The shortcut cannot: it either delivers the input unchanged, or it has stopped being an identity and the block should say so.

Residual connections do more than add another feature tensor.

Shortcuts change the shape of the loss surface, and the change can be drawn. Hao Li and four colleagues plotted the surface around a trained minimum in 2018. Their filter-normalization scheme is what makes two networks comparable. In those plots, “as network depth increases, the loss surface of the VGG-like nets spontaneously transitions from (nearly) convex to chaotic”. The same architectures with shortcuts kept a smooth surface, at the same depth. Their stated observation: “skip connections promote flat minimizers and prevent the transition to chaotic behavior, which helps explain why skip connections are necessary for training extremely deep networks.” Depth alone made the landscape unusable. The shortcut kept it flat.

The same difference has been measured outside vision, and there it turns on where the normalization sits relative to the merge. Wang and colleagues reported in 2019 that “we successfully trained a pre-norm Transformer system with a 20-layer encoder on the WMT English-German task, whereas the post-norm Transformer system failed to train for a deeper encoder”. The 20-layer pre-norm encoder came in 1.8 BLEU above the 6-layer baseline. Their table records the 16-layer post-norm Transformer simply as “failed”. Xiong and colleagues supplied the mechanism in 2020. Their mean-field analysis shows that at initialization Post-LN has large expected gradients near the output layer, which is why the learning-rate warm-up stage is needed; Pre-LN gradients are well-behaved and warm-up can be dropped. The shortcut is not what changed between those two systems. Only what sits inside the branch did.

Several details determine whether a residual stack trains cleanly. Each of the five below has a published number attached. The numbers are the reason to care about the detail.

More blocks still increase compute, memory, latency, and opportunities for overfitting, and a residual network can learn redundant or harmful updates.

Depth stops paying, and the authors of ResNet found the point and published it. Their 1202-layer CIFAR-10 network trained without optimization difficulty, down to under 0.1% training error. The shortcuts did exactly what they were built to do. It then scored 7.93% test error against 6.43% for the 110-layer network, on 19.4M parameters against 1.7M. Their verdict names the cause, and it is not optimization: “The testing result of this 1202-layer network is worse than that of our 110-layer network, although both have similar training error. We argue that this is because of overfitting.” Zagoruyko and Komodakis reproduce both rows in their own Table 5 — the original ResNet at 110 layers scoring 6.43 on CIFAR-10 and 25.16 on CIFAR-100, at 1202 layers 7.93 and 27.82. They conclude that very high depth in residual networks shows no regularization effect once parameter count is held fixed against wider networks.

Shortcuts ease optimization under many conditions. They do not prove that the added depth improves the target task. Matched ablations remain necessary. The 1202-layer result is what it looks like when you skip one: a network that trains beautifully and generalizes worse than a much shallower one.

Test a block independently before stacking dozens of copies.

The near-identity check in step 2 is not a thought experiment. It is a documented initialization. Goyal and eight colleagues describe it plainly: “Setting γ = 0 in the last BN of each residual block causes the forward/backward signal initially to propagate through the identity shortcut of ResNets, which we found to ease optimization at the start of training.” Tong He and colleagues at Amazon describe the same effect as making all residual blocks just return their inputs at the start of training. So drive your block into that state on demand and watch the shortcut alone come out the other side. If you cannot, step 2 has just told you the identity route is not clean.