Sequence-to-Sequence Architectures

A sequence-to-sequence model estimates an output sequence conditioned on an input sequence. The encoder builds representations of the source. The decoder predicts target elements under a chosen dependency order. The architecture has to decide what source information stays reachable at every target step. A single fixed vector is one option. It is worth knowing how well that option worked before studying how it broke.

One fixed vector was enough to beat the best translation system of its day. In 2014 Sutskever and his co-authors trained an ensemble of five deep LSTMs and squeezed every source sentence through a single vector. Four layers of 1,000 cells each. 380M parameters apiece in the published version, 384M in the preprint, which also describes an 8,000-dimensional sentence state. Their abstract states the outcome: “Our main result is that on an English to French translation task from the WMT-14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words.” The tabulated figure is BLEU 34.81 at beam size 12 on the WMT'14 English-to-French test set. The phrase-based system it was competing with scored 33.30. Rescoring that baseline's publicly released 1000-best lists with the same ensemble reached 36.5. Felix Stahlberg, reviewing the field years later, calls it “one of the first working standalone NMT systems that did not rely on any SMT baseline”. One fixed vector, and the old technology was behind.

The most instructive number in that paper is not the headline one. Reverse the word order of every source sentence, leave the targets untouched, and test perplexity drops from 5.8 to 4.7. Decoded BLEU rises from 25.9 to 30.6. Stahlberg records the same fact independently: the authors “reported gains from simply feeding the sequence in reversed order”. Nothing about the language changed. Only where in the fixed vector the opening source words ended up. A preprocessing trick that large is a reading on the interface, not on the task.

The failure had a shape, and it was measured before it was named. Cho and three co-authors had the measurement in October 2014. When “the encoder extracts a fixed-length representation from a variable-length input sentence”, the resulting system “performs relatively well on short sentences without unknown words, but its performance degrades rapidly as the length of the sentence and the number of unknown words increase”. The culprit was named the year after, by Bahdanau and his co-authors at ICLR 2015. They “conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture”, and propose “allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word”. The bottleneck was never in the language. It was in the interface.

The history changed the source–target interface one step at a time. First a fixed context vector, consumed by the decoder. Then soft alignment, where each decoder step combines source states with learned weights. Then multi-head cross-attention, where several learned projections retrieve source evidence. Then hybrid memory or retrieval, which exposes external or compressed context beyond the base encoder states.

The second step is the one that shipped, and it has a date. On 26 September 2016 Google posted the paper describing its Neural Machine Translation system, with thirty-one authors on it. The system is an LSTM with an 8-layer encoder and an 8-layer decoder. Its attention connects the bottom decoder layer to the top encoder layer. The abstract ends on the measurement: “Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system.”

The following day GNMT went into production for Chinese-to-English in Google Translate — then about 18 million translations a day. The company's own announcement gives the range: “GNMT reduces translation errors by more than 55%-85% on several major language pairs measured on sampled sentences from Wikipedia and news websites with the help of bilingual human raters”. That is what the second step on this timeline was worth. Not a redrawn diagram. A production switch with a date on it, eighteen months after the bottleneck was named.

The boundary to the third step is marked exactly in Stahlberg's review, which takes GNMT as its worked example of a recurrent encoder-decoder: “GNMT (Fig. 13a) uses regular attention”, while the Transformer “uses multi-head attention”. The steps on this timeline are not eras. They are differences in how many learned projections get to consult the encoder states.

The decoder sees different histories in the two regimes. Teacher forcing feeds the ground-truth previous target during training: stable supervision, clean histories, and recovery failures that stay hidden. Autoregressive inference feeds the decoder its own generated outputs. That matches deployment, but an early error alters all later context, and the system needs stopping rules and search. Scheduled or mixed strategies expose the model to selected predicted histories during training. That reduces the mismatch and changes the training distribution.

Three regimes were measured side by side on the MSCOCO captioning development set in 2015, by Bengio and three co-authors. The teacher-forced baseline scored BLEU-4 28.8, METEOR 24.2, CIDEr 89.5. Always feeding the model its own sampled token — the regime that matches inference exactly — collapsed to 11.2 / 15.7 / 49.7. The scheduled mixture reached 30.6 / 24.3 / 92.1. So the mismatch is not a nuance for a limitations paragraph. Training the model purely on its own history takes BLEU-4 from 28.8 down to 11.2. The alternative is to tell the decoder, at every step, something it did not actually produce.

The mixture that wins the benchmark carries a defect of its own, found within a year. The critique came in November 2015, from Ferenc Huszar: “Here we show that despite this impressive empirical performance, the objective function underlying scheduled sampling is improper and leads to an inconsistent learning algorithm.” His argument is that the method “is not a consistent estimation strategy”. As the sampling rate goes to zero, the divergence is minimised at the factorised distribution rather than the true joint. Hold the two results together: 30.6 against 28.8 on the leaderboard, and an estimator that does not converge on the thing it is estimating. A benchmark column cannot show you the second fact.

The architecture extends far beyond translation, and every destination brings its own baseline to beat. Summarization is the cautionary case. On the CNN/Daily Mail test set, a plain sequence-to-sequence-with-attention summarizer with a 50k vocabulary scored ROUGE-1 31.33. The pointer-generator with coverage that See and colleagues proposed in 2017 scored 39.53. Then the baseline: emit the article's first three sentences unchanged, no model at all, and score 40.34. Lead-3 beat every abstractive system in their table. Their own paper concedes the point: “It may be that news article style makes the lead-3 baseline very strong with respect to any metric”. Two years later a Salesforce Research team found the pattern intact and attributed it to layout bias in news corpora: “State-of-the-art models only slightly outperform the Lead-3 baseline, which generates summaries by extracting the first three sentences of the source document.” The architecture was working. The benchmark was measuring where the sentences sat in the document.

A poor output can originate in source representation, decoder dynamics, or search. A trace is only useful if it separates them.

1. Check source coverage. Identify omitted, duplicated, or overemphasized input regions.

2. Separate model score from decoding. Compare greedy, beam, constrained and sampled outputs. On one Transformer and one test set, greedy scored BLEU 29.3, beam-10 scored 30.3, and exact search over the same distribution scored 2.1. This step is not a formality you can skip when the numbers look reasonable.

3. Inspect prefix recovery. Introduce one wrong token and measure whether the decoder can return to a valid trajectory. The 11.2 BLEU-4 recorded for pure free-running training on MSCOCO is what the absence of that ability looks like at the extreme.

4. Slice by length and structure. Evaluate long inputs, long outputs, rare reorderings and repeated motifs. Two of the findings above are length findings wearing other clothes: reversing the source moved BLEU from 25.9 to 30.6, and exact search produced a length ratio of 0.06.

5. Test stopping behavior. Measure premature termination, endless loops and length bias. The empty translation is the limiting case of premature termination, and on one standard set it is the model's own global optimum for 51.8% of sentences.

Greedy decoding, beam search, sampling, constraints and length penalties can produce different outputs from the same model. The architecture defines conditional scores. The decoding procedure chooses a path.

How far apart can those two things drift? Far enough that the model's own best answer is silence. Stahlberg and Byrne ran exact inference — not a wider beam, exact — over the entire WMT15 English-German news-test2015 set, 2,169 sentences, with a Transformer base model. Greedy decoding scored BLEU 29.3 with 73.6% search errors. Beam-10 scored 30.3 with 57.7%. Exact search, which by construction commits no search errors at all, scored 2.1, at a length ratio of 0.06. The reason is in their abstract: “For more than 50% of the sentences, the model in fact assigns its global best score to the empty translation, revealing a massive failure of neural models in properly accounting for adequacy.” The precise figure is 51.8%. Even a beam of 100 still leaves 53.62% search errors. Eikema and Aziz treat the result as established — “the evidence that Stahlberg and Byrne (2019) provide is strong” — and add a sampling measurement of their own: an empty sequence appears at least once in 1,000 samples for only 7.2%-29.1% of held-out inputs.

Read that as a fact about reporting. A published BLEU of 30.3 there is a joint property of two things: a model whose highest-scoring output is silence, and a search too weak to find it. Credit the number to the architecture and you have credited it to the wrong half of the system.

The milder version of the same effect is routine. Across eight language directions, at beam sizes from 1 to 1,000, quality peaks at a narrow beam and then falls. Koehn and Knowles state it as their finding 6: “Beam search decoding only improves translation quality for narrow beams and deteriorates when exposed to a larger search space.” The optimal beam ranges from 4 for Czech-English to about 30 for English-Romanian, a factor of seven across language pairs. Sentence-length normalisation helped in 5 of the 8 pairs and does not remove the decline. The main cause is that wider beams produce shorter translations. Stahlberg's review reproduces the pattern on a Transformer for English-German WMT15. BLEU peaks at beam size 10, and wider beams give “a steady drop in translation performance because the generated translations are becoming too short”. The optimal beam has to be retuned per architecture, training technique and language pair.

Report both pieces. An unreported beam size can move a result further than an architecture change.