Depth, Composition, and Representation Hierarchies

Many tasks contain reusable intermediate structure. A network can exploit that structure when later layers operate on features constructed earlier.

For vision, that hierarchy is not a teaching metaphor. It was reconstructed, and then it was counted. In 2013 Zeiler and Fergus projected the activations of a trained convolutional network back into pixel space and read off what each layer responds to. Layer 2 fired on “corners and other edge/color conjunctions”. Layer 3 on textures. Layer 4 on class-specific parts: dog faces, bird's legs. Layer 5 on “entire objects with significant pose variation”. Their summary of the figure is one sentence: “The projections from each layer show the hierarchical nature of the features in the network.”

Four years later Bau and four colleagues stopped looking and started measuring. They scored 1,197 segmentation concepts against individual units across every convolutional layer of AlexNet, and reported that the method “reveals detectors for higher-level concepts at higher layers and lower-level concepts at lower layers”. The ladder below is the same argument in other coordinates. An intermediate object is worth building when more than one later computation uses it.

Each layer maps one representation into another. The final task may become simpler in the later coordinate system.

How much a change of coordinates buys has been counted, not argued. In 2014 Montúfar and three co-authors proved a lower bound on it. A rectifier network with n0 inputs and L hidden layers of width n ≥ n0 can compute functions with Ω((n/n0)^((L−1)n0) · n^n0) linear regions. Regions are the pieces on which the network is affine — a direct count of how finely the input space can be carved. Their own reading of the corollary: “Thus we see that the number of linear regions of deep models grows exponentially in L and polynomially in n, which is much faster than that of shallow models with nL hidden units.”

The bound has since been both sharpened and fenced. In 2018 Serra and two colleagues gave tighter upper and lower bounds, exact for one-dimensional inputs. They also attached the condition under which the advantage exists at all: a deep rectifier network beats every shallow counterpart with the same number of neurons only when that number exceeds the input dimension. Composition multiplies regions. Not at every scale, and not for free.

Both can increase capacity. They allocate that computation differently.

The trade is not symmetric, and the asymmetry was proved rather than argued. Telgarsky showed in 2016 that “for any positive integer k, there exist neural networks with Θ(k^3) layers, Θ(1) nodes per layer, and Θ(1) distinct parameters which can not be approximated by networks with O(k) layers unless they are exponentially large — they must possess Ω(2^k) nodes”. The result holds for a class of “semi-algebraic gates” that includes ReLU, maximum, indicator and piecewise polynomial functions. Width can buy back what depth provides. The exchange rate can be exponential.

The exchange has also been attempted in the laboratory, twice, with opposite-looking results. In 2014 Ba and Caruana trained shallow networks to mimic deep ones. On TIMIT and CIFAR-10 the mimics “perform similarly to complex, well-engineered, deeper convolutional models”, in some cases with the same number of parameters. On CIFAR-10 the parity was partial. Their best mimic, SNN-ECNN-MIMIC-30k, itself still contained one convolution and one max-pooling layer, and it reached 14.2% error against 12.0% for its teacher CNN and 11.0% for the four-CNN ensemble — on roughly 70M parameters against ~210k and ~840k.

Two years later a second group re-ran the test on CIFAR-10. Nine authors, Caruana among them, asked in their title whether deep convolutional nets really need to be deep and convolutional, and found that the student models cannot reach comparable accuracy unless they contain multiple layers of convolution. Their abstract answers the title in three words, then explains them: “Yes, they do. This paper provides the first empirical demonstration that deep convolutional models really need to be both deep and convolutional, even when trained with methods such as distillation that allow small or shallow models of high accuracy to be trained.”

A hidden concept may be encoded by a pattern across many coordinates rather than one dedicated neuron. Conversely, one unit may participate in several contexts.

This has been tested rather than assumed. Szegedy and six co-authors reported in 2014 that “there is no distinction between individual high level units and random linear combinations of high level units”. Bau and colleagues re-tested that sentence in 2017 with an experiment easy to state and hard to argue with. Take the 256 units of AlexNet conv5. Apply a random orthogonal rotation drawn from SO(256). Score the units again. The rotated representation had 80% fewer unique detectors — and yet “[e]ach rotated representation has exactly the same discriminative power as the original layer”. Interpretability was destroyed and accuracy was untouched. Whatever the individual units were carrying, it was not the network's performance.

The reason one unit can serve several contexts has a name and a demonstration. Elhage and colleagues at Anthropic and Harvard published Toy Models of Superposition on 14 September 2022: “In this paper, we use toy models — small ReLU networks trained on synthetic data with sparse input features — to investigate how and when models represent more features than they have dimensions. We call this phenomenon superposition.” In 2023 Cunningham and four colleagues recovered such overcomplete feature directions from a real language model, using sparse autoencoders. Their first line names the obstacle they were working against: “Polysemanticity prevents us from identifying concise, human-understandable explanations for what neural networks are doing internally.”

Visualizing one activation can be useful. Assigning a permanent human label to that unit may overstate the evidence, since representations are shaped jointly by architecture and objective — and, as the rotation shows, by a choice of basis the network never committed to.

A narrow hidden layer limits how much information can pass forward in its explicit representation, and that constraint can encourage useful compression, though it may also discard details that are required later.

Deliberately narrow layers are a documented engineering device with numbers attached. GoogLeNet inserted 1×1 convolutions for exactly this purpose: “they are used mainly as dimension reduction modules to remove computational bottlenecks, that would otherwise limit the size of our networks”. The narrowing is what paid for the depth. Its nine authors describe the design as a fixed-budget trade: “we increased the depth and width of the network while keeping the computational budget constant”.

What came out of that budget: a network 22 layers deep, a top-5 error of 6.67% on ILSVRC 2014 validation and test data, and “12 times fewer parameters than the winning architecture of Krizhevsky et al” from 2012. The ImageNet challenge's own 2014 results page lists GoogLeNet first in classification, with an error of 0.06656, ahead of VGG at 0.07325.

Whether a bottleneck helps depends on the task, loss, noise, and downstream use. Smaller dimensionality is not automatically a better representation. It was a better representation there, under that budget, measured against that benchmark.

Depth should answer a modeling need rather than a fashion preference.

Extra layers can make a model worse, and that has been measured with the parameter count held fixed. In 2016 He and three co-authors stated the failure plainly: “When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly.” On ImageNet, under 10-crop testing on the validation set, a 34-layer plain network scored 28.54% top-1 error. An 18-layer plain network scored 27.94%. Sixteen extra layers, worse accuracy.

Adding residual paths — which carry “no extra parameter compared to their plain counterparts” — took the same 34-layer design to 25.03%. Residual nets up to 152 layers then reached 3.57% error and 1st place in the ILSVRC 2015 classification task, where the challenge results list MSRA's classification error as 0.03567. Srivastava and two colleagues had reported the same barrier independently in 2015 — “training becomes more difficult as depth increases, and training of very deep networks remains an open problem” — and removed it with gated shortcut paths. That is what step 3 below is for. Depth that will not train is a training problem before it is a capacity problem.

Matched budgets are what turned scaling from folklore into a stated rule. Tan and Le reported on the balance itself in 2019. Their finding was that “carefully balancing network depth, width, and resolution can lead to better performance”. The resulting EfficientNet-B7 reached state-of-the-art ImageNet accuracy. It did so “while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet”. Depth did not win that comparison. The balance did.