Recurrence: State, Memory, and Sequence Order

A basic recurrent network combines the current input with the previous hidden state. One parameter set computes the new state, and that new state becomes the previous state for the following input.

Everything else in this lesson follows from that single loop: the depth it creates when unrolled, the gradients that have to travel back along it, the fixed width of what it carries, and the sequential execution it forces.

A recurrent layer is compact in code. During execution it expands into one operation per timestep, and the same parameters appear again and again across the unrolled graph.

How many copies is a training decision, and it has a published value. Recurrent Neural Network Regularization, in 2015, trained two-layer LSTMs on Penn Treebank — 929k training words, a 10k vocabulary — and states the setup plainly: “Both LSTMs have two layers and are unrolled for 35 steps.” That is 35 copies of the same cell in the graph, for every update. The large model clipped the gradient norm at 10. Regularized, at 1,500 units per layer, it reached test perplexity 78.4, against 114.5 for a non-regularized LSTM. The medium model, 650 units per layer, clipped at 5 and reached 82.7.

Backpropagation through time is ordinary reverse-mode differentiation applied to that graph. Contributions from every one of the 35 uses accumulate into the same shared parameter gradients.

Earlier states influence the loss through many repeated transition Jacobians, and the condition under which that product misbehaves was proved rather than described. On the difficulty of training recurrent neural networks, in 2013, showed that a spectral radius below 1 for the recurrent weight matrix is sufficient for the long-term gradient components to vanish, and that a spectral radius above 1 is necessary for them to explode.

The remedy tested there is as blunt as the diagnosis is sharp: clip the gradient norm. A 50-unit tanh RNN trained with norm clipping at a threshold of 1 reached a 100% success rate over 5 seeds on the temporal order problem at sequence lengths 50–200. Then it went past the distribution it was trained on: “The same trained model can address sequences of length up to 5000 steps, lengths never seen during training.”

Path length is not an abstraction either. It is worth measurable accuracy. The 2014 sequence-to-sequence paper reversed the word order of the source sentence and left the target sentences unreversed, changing nothing else about the model or the data. The result: “By doing so, the LSTM's test perplexity dropped from 5.8 to 4.7, and the test BLEU scores of its decoded translations increased from 25.9 to 30.6.” Reversal does not change the average distance between corresponding words. What it changes is that the first words of source and target become adjacent, so the problem's "minimal time lag" was greatly reduced. That is 4.7 BLEU on WMT'14 English–French, bought by shortening dependencies alone.

Truncated backpropagation limits how far gradients travel during one update. It reduces compute and memory. It also changes which long-term dependencies training can credit.

LSTM and GRU units add gates. The gates regulate writing, retaining, and exposing state.

Which gates carry the weight has been tested twice, from opposite directions. One study compared the vanilla LSTM against eight variants on speech recognition, handwriting recognition and polyphonic music modelling, tuning each variant's hyperparameters separately by random search. Its scale was “the results of 5400 experimental runs (≈ 15 years of CPU time)”. The verdict, published in IEEE Transactions on Neural Networks and Learning Systems in 2017, was that “none of the variants can improve upon the standard LSTM architecture significantly”. Two pieces did stand out. The study demonstrated “the forget gate and the output activation function to be its most critical components”.

The other study searched instead of ablating. It evaluated over ten thousand different RNN architectures in 2015, and none was uniformly better than the LSTM or the GRU. Its ablation result is the one to keep: “We discovered that the input gate is important, that the output gate is unimportant, and that the forget gate is extremely significant on all problems except language modelling.” The same paper reports the one setting you can act on tomorrow — initialize the LSTM forget-gate bias to 1 and the gap between LSTM and GRU closes. Removing a gate is the experiment that identifies it. Adding one usually is not.

Gating was not only a benchmark story. Google's own Neural Machine Translation system was built on it, and its abstract says so: “Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using residual connections as well as attention connections from the decoder network to the encoder.” It reduced translation errors by an average of 60% against Google's phrase-based production system in human side-by-side evaluation. On 27 September 2016 Google announced the deployment: “The Google Translate mobile and web apps are now using GNMT for 100% of machine translations from Chinese to English—about 18 million translations per day.”

Autoregressive sequence models may condition each step on a previous token or output. The gap that opens between training and inference has a name, and a 2016 ICLR paper gave it: “We refer to this discrepancy as exposure bias which occurs when a model is only exposed to the training data distribution, instead of its own predictions.”

The same paper priced it. On IWSLT'14 German–English, greedy BLEU-4 rose from 17.74 for a cross-entropy-trained LSTM to 20.12 for the same model trained with scheduled-sampling-style Data as Demonstrator, and to 20.73 for their sequence-level MIXER. Summarization moved with it, ROUGE-2 13.01 to 16.22, and so did MSCOCO captioning, BLEU-4 27.8 to 29.16.

The gap is wide enough to be worth closing on purpose. Scheduled sampling, proposed at NIPS 2015, closes it during training: feed the model its own previous prediction instead of the true token, and raise the probability of doing so as training proceeds. On the MSCOCO development set for image captioning the baseline scored BLEU-4 28.8, METEOR 24.2 and CIDER 89.5. The same model trained with scheduled sampling scored 30.6, 24.3 and 92.1. The authors used the method in their winning entry to the 2015 MSCOCO image captioning challenge.

Recurrence creates a path whose length grows with sequence distance, and execution that is inherently sequential. Attention Is All You Need put that cost in a table rather than a sentence. Its Table 1 gives a recurrent layer O(n) sequential operations and an O(n) maximum path length between any two positions. A self-attention layer gets O(1) of each. The paper states it directly: “As noted in Table 1, a self-attention layer connects all positions with a constant number of sequentially executed operations, whereas a recurrent layer requires O(n) sequential operations.” Jakob Uszkoreit made the same point for Google Research in plain terms in 2017: “Reading one word at a time, this forces RNNs to perform multiple steps to make decisions that depend on words far away from each other”. The model built on the constant-path alternative reached 28.4 BLEU on WMT 2014 English-to-German, and a single-model 41.0 on English-to-French, after 3.5 days on eight GPUs.

Recurrent networks remain valuable when streaming state, compact memory, or causal stepwise updates match the task. Architecture choice should follow constraints rather than chronology.