Encoder, Decoder, and Encoder–Decoder Model Families

All can be adapted. But their pretraining masks and interfaces create useful biases — and the boxes leak at the edges.

The clearest evidence that they leak comes from the people who built one of them. BART is a denoising sequence-to-sequence model, published in 2020. Its abstract describes its own architecture entirely in terms of the other two columns: “It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and other recent pretraining schemes.” (The spelling “Tranformer” is as printed in the original.) BART matched RoBERTa on GLUE and SQuAD, and added gains of up to 3.5 ROUGE on abstractive summarization, dialogue and question answering.

So read the three columns below as positions on a continuum, described by the visibility their inputs get and the objective they were trained under. Not as a taxonomy of separate species.

An encoder can let each input position attend to tokens on both sides, subject to the model’s masking scheme. That helps when the task needs a representation of the whole input — classification, or named-entity recognition.

The family’s defining bias is not folklore. Its authors wrote it into the first sentence of the abstract, in 2019: “BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers.” Not one direction, and not one layer. Both directions, in all layers.

What made this the default starting point for classification and extraction work is a set of numbers, not an impression. The paper reported new state-of-the-art results on eleven NLP tasks. It pushed the GLUE score to 80.5, MultiNLI accuracy to 86.7%, SQuAD v1.1 test F1 to 93.2 and SQuAD v2.0 test F1 to 83.1.

None of that makes the model “more understanding” because of what the family is called. It is biased toward producing contextual states that downstream heads can use. The eleven-task sweep is the measured consequence of that bias — a claim about benchmark scores, not about comprehension.

A decoder-only language model predicts each next token from preceding context. Instructions, demonstrations, retrieved passages and tool descriptions are serialized into that context, which makes the interface unusually flexible. The same flexibility leaves a request open to more than one reading. The model may continue the style of an instruction without satisfying its hidden business constraints, so applications still need schemas, tools and validation.

The three families arrived within two years of each other, and each kept its own interface. BERT came in 2019, with a GLUE score of 80.5. T5 followed in 2020, recasting every text-based language problem into a text-to-text format and introducing the Colossal Clean Crawled Corpus. GPT-3 arrived the same year with 175 billion parameters and few-shot prompting — a count its paper describes as ten times more than any previous non-sparse language model.

None of the three retired the others. All three are still deployed.

Encoder–decoder systems provide a distinct representation path for the source and a causal path for the target. The join between the two paths is a specific piece of architecture, and it has been specified in print since the family began.

The 2017 Transformer paper defined it in its section on encoder and decoder stacks: “In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack.” That inserted third sub-layer is the entirety of what the step below calls attending across roles. It is not a mode the model enters. It is a layer that is there in every decoder block.

The design was justified by a translation result: 28.4 BLEU on WMT 2014 English-to-German, after training for 3.5 days on eight GPUs.

Document why the selected bias helps the intended task. Two rows of this table have published price tags attached to them, and both are large enough to decide a design on their own.

Step 3 asks about reuse of source representations. Reimers and Gurevych measured what that reuse is worth. Finding the most similar pair among 10,000 sentences takes about 50 million inference computations with a BERT cross-encoder. Their 2019 abstract states the consequence: “This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT.” Same family, comparable accuracy. The gap between hours and seconds is decided by one thing: whether the encoder output is computed once per document or once per pair.

Step 5 asks for an end-to-end benchmark, and quality moves as well as cost. Dense Passage Retrieval, in 2020, opened with the claim its experiments then supported: “In this work, we show that retrieval can be practically implemented using dense representations alone, where embeddings are learned from a small number of questions and passages by a simple dual-encoder framework.” That retriever outperformed a strong Lucene-BM25 system by 9%-19% absolute in top-20 passage retrieval accuracy across open-domain QA datasets.

And because MTEB found no method dominating across its 58 datasets and 33 benchmarked models, the last step cannot be delegated to somebody else’s leaderboard. The evaluation set has to be yours.