Deep Learning Architectures as Information Routing

A deep network is more than a stack of layers. Its architecture decides which pieces of information can meet, how often they interact, and what must pass through a bottleneck.

A convolution routes evidence through local neighborhoods. Recurrence carries a state forward. Attention creates content-dependent routes. A graph network follows observed relationships. A generative model organizes a path from randomness, or from partial observations, to a structured sample.

Those routes have lengths and costs. Both have been measured and published. The rest of this lesson compares them with the numbers their authors reported, not with adjectives.

Most modern models combine several of these patterns rather than belonging to one box.

The last node, conditional routing, carries the sharpest published numbers. A gate makes the difference between parameters stored and parameters spent visible in a single figure. The Switch Transformer paper put it this way in 2022: “The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost.” It reports up to 7x faster pre-training at equal compute, gains across all 101 languages of mT5, and a 4x speedup over T5-XXL, while scaling to trillion-parameter models.

A shipped product gives the same mechanism a concrete gate. Mixtral 8x7B routes each token, at each layer, to 2 of 8 feed-forward experts. Mistral AI discloses what that costs. Each token has access to 47B parameters and uses only 13B active parameters during inference, over a 32k-token context. Two numbers, one architecture: 47B stored, 13B spent.

Names such as Transformer or U-Net are shorthand. These questions reveal what a design actually assumes. Each of them has an answer someone has already tabulated.

Who can communicate, and across what distance? In 2017 the Transformer's authors did not argue that point. They tabulated it. Maximum path length between any two positions: O(1) for self-attention, O(n) for a recurrent layer, O(log_k(n)) for dilated convolutions, and O(n/r) for self-attention restricted to a neighbourhood of size r. The per-layer costs are O(n^2*d), O(n*d^2) and O(k*n*d^2) respectively. Why put that column in the paper at all? Because “One key factor affecting the ability to learn such dependencies is the length of the paths forward and backward signals have to traverse in the network.” The textbook Dive into Deep Learning derives the same quantities independently, and states the maximum path length as O(n/k) for a k-kernel CNN, O(n) for an RNN and O(1) for self-attention. Connectivity is not a style. It is an exponent.

What is shared, and what does sharing buy? Three stacked 3x3 convolutions have the same 7x7 effective receptive field as one 7x7 layer. Simonyan and Zisserman priced that in 2015. The stack carries 27C^2 weights against the wide layer's 49C^2 — the single wide layer needs 81% more, the stack 45% fewer, for the same reach. Sharing does not by itself make a network small. Their own table still gives 138 million parameters for the 16-layer configuration D and 144 million for the 19-layer E. Szegedy and four colleagues carried the same factorisation further in 2016: “Still the two-layer solution is 33% cheaper for the same number of output filters, if the number of input and output filters is equal.” Their network reached 21.2% top-1 and 5.6% top-5 error on the ILSVRC 2012 validation set, with under 25 million parameters and 5 billion multiply-adds per inference. Better accuracy than VGG's configurations, on a fraction of the parameters.

What persists across a bottleneck, and what shape leaves the network? The U-Net answered both with one topology: “The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization.” The test was the ISBI 2012 EM segmentation challenge. Training data was 30 consecutive 512x512 serial-section TEM images of the Drosophila first instar larva ventral nerve cord, with the test ground truth withheld. The U-Net used no pre- or post-processing. It recorded a warping error of 0.0003529 and a Rand error of 0.0382. That ranked first on the challenge's warping-error leaderboard of 6 March 2015, ahead of the sliding-window ConvNet of IDSIA at 0.000420 and 0.0504. The qualifier belongs in the same breath. The organisers' own official ranking metric was the foreground-restricted Rand F-score, and on that metric two entries using dataset-specific post-processing scored better. On the ISBI cell tracking challenge 2015 the margins are wider: an average IOU of 0.9203 on PhC-U373 against 0.83 for the second best, and 0.7756 on DIC-HeLa against 0.46. Skip connections are what let a dense spatial map come out the far side of a compression path.

A useful design review decomposes the system instead of assigning it one fashionable label. The best-documented composites let you check that decomposition against someone else's scoring.

AlphaFold2's margin at CASP14 was measured by an independent assessor. The official final tertiary-structure group ranking lists group 427, AlphaFold2, assessed on 92 domains, with a summed z-score of 244.0217 and an average z-score of 2.6524. The second-placed group, 473 (BAKER), scored 90.8241 and 0.9872. The 2021 paper in Nature locates the source of that margin: “Underpinning the latest version of AlphaFold is a novel machine learning approach that incorporates physical and biological knowledge about protein structure, leveraging multi-sequence alignments, into the design of the deep learning algorithm.” The routing is nameable piece by piece. An MSA representation and a pairwise-residue representation, exchanged through the Evoformer, then a structure module.

The text-to-image composite is equally decomposable, and its costs are published. One sentence of the latent-diffusion paper of 2022 gives the conditioning mechanism: “By introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes and high-resolution synthesis becomes possible in a convolutional manner.” It describes a 1.45-billion-parameter text-to-image model trained on LAION-400M. On unconditional LSUN-Bedrooms that model scored close to ADM “despite utilizing half its parameters and requiring 4-times less train resources”, in a field where pixel-space diffusion models cost 150-1000 V100 days. The bottleneck is not a detail of the design. It is the reason the system is trainable at all.

Use this pass before you build anything or choose a benchmark. Each step asks for a number that someone will eventually publish about your system: a path length, a parameter count, a FLOP budget, a wall-clock training time, an error on a named benchmark.

Later paths treat vision, language, audio, optimization and generative products in their own contexts. Here the goal is to understand reusable architectural patterns and their failure modes.

That separation matters. A Transformer block is an architecture; prompt engineering is not. A diffusion denoiser is an architecture; a content-moderation policy is a product control. The under 5 hours that Facebook AI Research needed for 28.4 BLEU belong to the training procedure, not to the topology that scored it.