Skip to content
AI.info

Deep architectures

Encoder, Decoder, and Encoder–Decoder Transformers

Compare bidirectional encoders, causal decoders, and source-conditioned encoder–decoder Transformers across objectives, outputs, latency, and failure modes.

By the end you can

Comparison

Three canonical Transformer families

They reuse similar blocks. They impose different visibility and output contracts.

The three are not co-equal inventions, and the order matters. The encoder–decoder came first. The other two are what remains when one half is removed.

The Transformer arrived in 2017 as a sequence-transduction model: a source encoded, a target generated against it. Eight authors introduced it, in the paper called Attention Is All You Need. The justification was translation, on a public benchmark, with the numbers stated. It reported 28.4 BLEU on the WMT 2014 English-to-German task. The abstract records the other pair directly: “On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature.”

Note what the second half of that sentence is doing. 3.5 days on eight GPUs is a claim about the training cost of the two-stack design, not only about its quality. Encoder-only and decoder-only are later specialisations of that configuration. Each of them gives something up.

FigureComparison · 3 columns

Encoder-only

Build contextual representations with bidirectional self-attention.

  • Reads the whole available input
  • Useful for classification and retrieval encoders
  • No inherent autoregressive generator
  • Often pooled or used tokenwise

Decoder-only

Predict the next element with causal self-attention.

  • One growing sequence interface
  • Natural open-ended generation
  • Prefill followed by autoregressive decode
  • Input and output share one context stream

Encoder–decoder

Encode a source bidirectionally and decode a target causally with cross-attention.

  • Separates source and target roles
  • Strong for conditional transduction
  • Encoder cost paid once
  • Two sets of representations and masks

Training objective and architecture constrain each other

A masked or denoising objective can train an encoder to use both left and right context, while a causal next-step objective trains a decoder under one-way visibility.

Objectives can be modified, but visibility remains decisive: a model cannot use future target elements during generation simply because they were present in a different training task.

The cleanest demonstration holds the network fixed and moves only the mask. UniLM is one shared Transformer that becomes a unidirectional, a bidirectional, or a sequence-to-sequence model, according to what its self-attention is permitted to see. Dong and colleagues published it in 2019. The abstract puts it in one line: “The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on.”

The single pre-trained model raised CNN/DailyMail abstractive summarisation ROUGE-L to 40.51, a gain of 2.04 absolute. It raised CoQA generative question-answering F1 to 82.5, a gain of 37.1. Same blocks, same weights, three families. The mask is the variable, not the block name.

Architecture defines legal evidence; the objective teaches how to exploit that evidence.

Visual

Information flow by family

Follow one output position back to the evidence it may use.

The first row is the one BERT made ordinary. The 2019 paper describes its model as “designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers”. Devlin and colleagues wrote it.

They report new state-of-the-art results on eleven natural language processing tasks. GLUE went to 80.5, a 7.7-point absolute improvement. MultiNLI accuracy went to 86.7%. SQuAD v1.1 test F1 reached 93.2, and SQuAD v2.0 test F1 reached 83.1. None of those numbers comes from generating a token at a time. That stack has no autoregressive output loop to run.

FigureHierarchy · 4 levels

  • Encoder representation

    May combine all unmasked input positions.

    • Decoder prefix state

      May combine only previous target positions under a causal mask.

      • Cross-attended decoder state

        May combine the target prefix with all permitted source representations.

        • Task head or token distribution

          Converts the final state into a score, embedding, class, or next-element distribution.

Example

Architecture family follows the output job

Task labels alone do not determine the family. Product behavior matters.

Two of the bullets below have been measured rather than argued. Take retrieval first. A cross-encoder scores one pair at a time, and that cost was measured in 2019: finding the most similar pair in a collection of 10,000 sentences takes about 50 million BERT inference computations. Reimers and Gurevych state the consequence in the Sentence-BERT abstract: “This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT.”

Khattab and Zaharia arrived at the same structure independently, in ColBERT in 2020. Encoding query and document separately, they report, runs two orders of magnitude faster than a BERT model scoring each pair, with four orders of magnitude fewer FLOPs per query. Nothing in the encoder is cleverer. One vector per item can be computed once and reused; a pairwise scorer cannot.

Now the completion bullet. GPT-3 is where the single growing sequence was shown to carry arbitrary tasks with no gradient step at all: “Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model.”

The task description, the demonstrations and the answer all live in one context stream. That is the interface. It is why the decoder-only family owns interactive products.

  • Document embedding: an encoder-only model produces one reusable vector per item for retrieval, with no autoregressive generation — about 5 seconds instead of about 65 hours over 10,000 sentences.
  • Interactive completion: a decoder-only model extends a prefix in one unified prompt–response context — the interface GPT-3 carried at 175 billion parameters, tasks and demonstrations specified purely as text.
  • Translation: an encoder–decoder keeps a distinct source memory while the target grows autoregressively — the configuration the 2017 Transformer was first justified on, at 28.4 and 41.8 BLEU across the two WMT 2014 pairs.
  • Extractive tagging: an encoder supplies a contextual state for every input position — the tokenwise output behind BERT's eleven-task sweep and its 93.2 SQuAD v1.1 test F1.
  • Constrained report generation: either decoder family can work, but source length, caching, and control requirements differ.

Analogy

Reading, continuing, and translating are different desks

One desk reads an entire dossier and annotates it, a second continues a document from its current prefix, and a third reads one document while drafting another.

Separate desks stay separate. Neural models can be adapted across tasks and share pretraining. The three families still differ in the visibility each output position has, and in what each output is conditioned on.

The same Transformer block behaves differently when its access pattern and output loop change.

Steps

Select a Transformer family from the interface outward

Start with legal evidence and output behavior before considering model size.

This comparison was run, not asserted. Encoder–decoder, decoder-only language-model and prefix-LM variants were pitted against one another inside a single text-to-text framework. Raffel and colleagues published the result in 2020, in the Journal of Machine Learning Research.

Their conclusion: “we found the original encoder-decoder form worked best in our text-to-text framework. Though an encoder-decoder model uses twice as many parameters as ‘encoder-only’ (e.g. BERT) or ‘decoder-only’ (language model) architectures, it has a similar computational cost.”

Note which quantity that sentence controls. Parameters differ. Compute does not. The conclusion is about compute. The study pre-trains on the “Colossal Clean Crawled Corpus”, about 750 GB of filtered April 2019 web text, and scales to about 11 billion parameters.

FigureProcess · 5 steps

  1. 1. Classify the output

    Representation, aligned labels, fixed target, or open-ended sequence.

  2. 2. Declare visibility

    Bidirectional source, causal target, and any protected segments.

  3. 3. Estimate source reuse

    Decide whether one encoded source conditions many target steps or requests.

  4. 4. Model latency phases

    Separate encoding or prefill from token-by-token decode.

  5. 5. Compare matched baselines

    Hold data, quality target, and serving conditions as stable as possible.

Key idea

Decoder-only is not a universal replacement

A decoder-only model can emulate many tasks through prompting or fine-tuning, but that does not make it the cheapest or best-conditioned architecture; bidirectional encoding and a separate source memory remain useful.

Compare data efficiency, sequence lengths, serving pattern, controllability, and representation quality. A model that is general at scale is not thereby the right choice under your constraints.

Two matched studies say this with results rather than taste, and they disagree with the folk ranking in the same direction. The first put causal decoder-only, non-causal decoder-only and encoder–decoder models against one another at over 5 billion parameters and more than 168 billion tokens. Wang and colleagues published it in 2022. The winner flips with what you do after pretraining: “Our experiments show that causal decoder-only models trained on an autoregressive language modeling objective exhibit the strongest zero-shot generalization after purely self-supervised pretraining. However, models with non-causal visibility on their input trained with a masked language modeling objective followed by multitask finetuning perform the best among our experiments.”

The second reached a matching conclusion independently. Zhang and colleagues ran a scaling study in October 2025, from about 150M to about 8B parameters. Decoder-only came out more compute-optimal in pretraining. Encoder–decoder was comparable or better after instruction tuning.

Neither study crowns a family. Both make the answer a function of the adaptation regime you are actually going to use.

Choose a family for the workload, not for its cultural prominence.

Architectures can combine family patterns

Prefix language models, retrieval-conditioned decoders, multimodal encoders, and bidirectional–causal hybrids modify the canonical visibility patterns. What matters is the actual mask and interface.

Avoid calling a system encoder-only or decoder-only when extra cross-attention, memory, or bidirectional prefixes materially change the graph; document the legal connections explicitly.

"Retrieval-conditioned decoder" stays an abstraction until one is drawn. DeepMind drew one. RETRO is a causal decoder reading a 2-trillion-token retrieval database, and its 2022 abstract lists the parts: “RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training.” DeepMind's own account adds the two levels the graph operates on: “The RETRO architecture interleaves regular self-attention at a document level and cross-attention with retrieved neighbors at a finer passage level”.

Label that system decoder-only and you have concealed a retriever, an encoder and a second attention path from anyone reasoning about its costs or its failure modes. The concealment is expensive, because the graph is the point. RETRO matched GPT-3 and Jurassic-1 on the Pile with 25× fewer parameters, and by DeepMind's write-up “a 7.5 billion parameter RETRO model outperforms the 175 billion parameter Jurassic-1 on 10 out of 16 datasets”.

Family labels are useful summaries only when they match the real attention graph.

Key takeaways