Architecture Motifs: Encoders, Decoders, Bottlenecks, and Towers

Many architectures are combinations of a few reusable patterns rather than entirely new inventions. The way to test that claim is to take one named system and read it as motifs.

AlphaFold2 is a trunk, two representations, and a decoding head. The trunk is 48 Evoformer blocks. They carry an Nseq x Nres MSA array and an Nres x Nres pair array, and they exist so that those two arrays can exchange information. The 2021 Nature paper says it plainly: “The key innovations in the Evoformer block are new mechanisms to exchange information within the MSA and pair representations that enable direct reasoning about the spatial and evolutionary relationships.” A structure module then decodes 3D coordinates. Two towers that talk to each other, one trunk, one decoder. Nothing in that description is new to this lesson.

The vocabulary is not a teaching convenience laid over a result that came from somewhere else. In CASP14 the system reached a median backbone accuracy of 0.96 A r.m.s.d.95. The assessors' final summed z-score table ranks group 427 'AlphaFold2' at 244.0217. The second group, BAKER, scored 90.8241. The motifs below are the parts that table is scoring.

An encoder usually reduces, summarizes, or contextualizes input into an internal representation, and a decoder then turns that representation into outputs with their own spatial, temporal, or symbolic structure.

Compression is not mandatory. Some encoders preserve dense feature maps, and some decoders read all encoder states through attention. The labels describe roles rather than fixed layer types.

What crosses that boundary was measured early. The answer shaped the decade that followed. In 2014 Cho and three co-authors tested their own encoder–decoder translators and found that they “perform relatively well on short sentences, but suffer significantly as the length of the sentences increases”. A conventional phrase-based system trained on the same data did not fall away in the same manner. Their explanation names the crossing: “the fixed-length vector representation does not have enough capacity to encode a long sentence with complicated structure and meaning”. Whatever a decoder cannot reach, it cannot use.

The opposite design removes the crossing rather than widening it. The Transformer is an encoder–decoder that compresses nothing. It has 6 encoder layers and 6 decoder layers, d_model = 512 in the base configuration and 1024 in the 'big' one, joined by cross-attention. The 2017 paper describes what that buys: “This allows every position in the decoder to attend over all positions in the input sequence.” No fixed-length vector stands between the halves, so there is nothing for a long sentence to overflow. The big model scored 28.4 BLEU on WMT 2014 English-German after 300,000 steps, 3.5 days on eight NVIDIA P100 GPUs. A second group then reproduced it. Ott and three co-authors matched that accuracy in under 5 hours on 8 GPUs. They then set a new state of the art of 29.3 BLEU after 85 minutes on 128 GPUs.

Between the two extremes sits a third option: keep the bottleneck and route the discarded detail around it. That is the skip path, and U-Net below is the worked case.

Multi-task systems must decide where features are shared and where they specialize.

Sharing a trunk converts a modeling question into a tuning question. That question gets more expensive with every task added. On NYUv2 with three tasks, a grid search over fixed task weights still fell short of an adaptive scheme “even after 100 networks trained”. Networks whose weights sat far from the adaptive solution had “almost double the errors per task”. That is the GradNorm result, published in 2018 by Chen and three co-authors. Their algorithm sets the weights during training from gradient magnitudes, with one asymmetry hyperparameter. The figure to carry away is the hundred. A shared backbone is cheap at inference and costly to balance.

Merge operations determine whether paths must share coordinates and how downstream layers can separate their contributions. Three named systems made the three main choices, and each choice has a published price and a third-party scoreboard.

Addition. ResNet merges by element-wise addition on an identity shortcut, and the merge itself is free: “The shortcut connections in Eqn.(1) introduce neither extra parameter nor computation complexity.” It was introduced to fix a measured failure, not a suspected one. A 56-layer plain network had higher training error than a 20-layer one on CIFAR-10. An ensemble of six residual nets of different depth, only two of them 152-layer, took 1st place in ILSVRC 2015 with 3.57% top-5 test error. The organisers' own results table lists MSRA at 0.03567, ahead of ReCeption at 0.03581.

Concatenation. U-Net refuses the shared coordinate system and keeps both feature sets distinguishable: “Every step in the expansive path consists of an upsampling of the feature map followed by a 2x2 convolution (“up-convolution”) that halves the number of feature channels, a concatenation with the correspondingly cropped feature map from the contracting path, and two 3x3 convolutions, each followed by a ReLU.” That is 23 convolutions and no fully connected layers. The concatenated skip is what carries the resolution the contracting path threw away. U-Net reported IOU 0.9203 on PhC-U373 against 0.83 for the second-best 2015 entry, and 0.7756 on DIC-HeLa against 0.46. The Cell Tracking Challenge organisers score entries against hidden ground truth and publish their own benchmark in Nature Methods. It ranks FR-Ro-GE first by overall performance on both PhC-C2DH-U373 (0.951) and DIC-C2DH-HeLa (0.828).

Gating. The Squeeze-and-Excitation block multiplies instead of adding or concatenating — global average pool, two FC layers, sigmoid, then channel-wise scaling. The paper names the second half: “The aggregation is followed by an excitation operation, which takes the form of a simple self-gating mechanism that takes the embedding as input and produces a collection of per-channel modulation weights.” The price is about 2.5 million extra parameters and a 0.26% relative FLOP increase on ResNet-50, about 3.87 against 3.86 GFLOPs. SENets won ILSVRC 2017 classification with 2.251% top-5 error, matching the organisers' table entry WMW at 0.02251.

And a merge assumption can be wrong in a way you can measure. The Squeeze-and-Excitation ablation shows the gate's nonlinearity is load-bearing: on SE-ResNet-50, sigmoid gives 6.03% top-5 error, tanh 6.38%, ReLU 6.98%. Same block, same parameter count, three different answers.

Retrieval systems often encode queries and candidates separately, then compare their embeddings cheaply. Candidate vectors can be precomputed and indexed.

This efficiency limits interaction before the final similarity score, though a later reranker can add richer cross-input attention for a smaller candidate set.

Both halves of that trade have been measured. One published retriever is two BERT-base encoders, one for questions and one for passages, with no interaction until a dot product. Its 21 million Wikipedia passages are encoded once and indexed with FAISS offline. Even with the interaction removed it “outperforms a strong Lucene-BM25 system greatly by 9%-19% absolute in terms of top-20 passage retrieval accuracy”. That is 78.4% against 59.1% on Natural Questions. The costs move to where the design put them. Building the dense index takes about 8.5 hours on a single server, against roughly 30 minutes for a Lucene inverted index. Query time then runs at 995.0 questions per second, against 23.7. Karpukhin and seven co-authors published all of it in 2020.

The motif is not specific to retrieval, and neither is the evidence. CLIP is the same shape across two modalities: “CLIP jointly trains an image encoder and a text encoder to predict the correct pairings of a batch of (image, text) training examples.” The towers were trained on 400 million (image, text) pairs and meet only at a cosine similarity in a shared embedding space. Zero-shot ImageNet accuracy rose from Visual N-Grams' 11.5% to 76.2%, matching a supervised ResNet-50 that had 1.28 million labelled examples. An independent group then re-measured the claim from outside. In 2023 Cherti and colleagues put OpenAI's ViT-L/14 at 75.5% zero-shot ImageNet, and reproduced 75.2% with the same architecture trained on open LAION-2B data.

Ignore the branding at first. Trace the graph as a set of contracts.

Run the five steps on AlphaFold2 and the name falls away. Inputs and outputs: an Nseq x Nres MSA array and an Nres x Nres pair array in, 3D coordinates out. Shared parameters: 48 Evoformer blocks in one trunk. Shape changes and merges: the exchange between the MSA and pair representations, which is the whole reason the block exists. Losses and ablation: whatever is left when you ask which of those two representations you could remove. The diagram becomes a list of motifs you already have names for, and 244.0217 against 90.8241 is what those motifs, assembled that way, scored.