LSTM, GRU, and Gated Memory

The cell creates multiple controlled paths rather than one unrestricted rewrite. The diagram shows four of them. They are not equally important, and one of them was not in the original design at all.

The evidence for that is an ablation. Eight LSTM variants, each missing one component, were run across three datasets. Two of the amputations the cell could not survive: “The first important observation based on Figure 3 is that removing the output activation function (NOAF) or the forget gate (NFG) significantly hurt performance on all three datasets.” Two others cost nothing measurable. Coupling the input and forget gates (CIFG) and removing the peephole connections (NP) did not significantly change mean performance. Greff and colleagues name the two survivors in their abstract: “the forget gate and the output activation function”.

The forget gate is a retrofit. It is absent from the 1997 cell. Gers and two colleagues added it three years later, in 2000, and a 2002 paper by the same first author builds on “LSTM with forget gates (Gers et al., 2000)”. A 2015 search over ten thousand architectures arrived at the same gate from the opposite direction. So the four-way diagram is a map of the mechanism, not a ranking of it. The path the evidence protects is the one that decides what stays.

A gate near one can preserve a component across many steps. A gate near zero removes it quickly. Different state dimensions can therefore learn different effective memory horizons.

This mechanism improves access to long dependencies. It does not ensure the model discovers the right memories. Data, objectives, initialization and gradient paths still determine what the gates learn.

Both address recurrent memory. Their interfaces, their costs and their computational power differ, and each difference has been measured.

Count the weights first. PyTorch's nn.LSTM stores input-hidden and hidden-hidden weights of shape (4*hidden_size, ·). That is four blocks, W_ii|W_if|W_ig|W_io, for the input gate, the forget gate, the cell candidate and the output gate. nn.GRU stores (3*hidden_size, ·): W_ir|W_iz|W_in, for the reset gate, the update gate and the candidate. Three blocks against four. The same width costs a GRU less, and the same budget buys it more. Chung and colleagues spent the difference the second way, sizing the models to match: “As the primary objective of these experiments is to compare all three units fairly, we choose the size of each model so that each model has approximately the same number of parameters.” The budget bought 46 GRU units against 36 LSTM units at roughly 20×10³ parameters.

The separation is not only arithmetic. At finite precision and linear computation time, the LSTM's cell state can act as a register of counters. The GRU's state cannot. Weiss and two colleagues opened a short 2018 paper with the strict ordering: “In particular, we show that the LSTM and the Elman-RNN with ReLU activation are strictly stronger than the RNN with a squashing activation and the GRU.” Six authors put space bounds on the same gap in a formal hierarchy of RNN architectures in 2020. Its “Building Blocks” section records both: “Merrill (2019) showed that the s-LSTM has Θ(log n) space” against “Merrill (2019) showed the s-GRU has Θ(1) space” — a finite-state class. A third group found the separation by experiment. Suzgun and colleagues trained cells on Dyck-1 in 2019. Single-unit LSTMs generalised far beyond the training set, while the GRUs “could not generalize well to longer and deeper sequences”.

And the GRU's two gates are themselves one more than some tasks require. Ravanelli and colleagues removed one in 2018: “First, we analyze the role played by the reset gate, showing that a significant redundancy with the update gate occurs.” Their Light GRU keeps a single gate and replaces tanh with ReLU activations coupled to batch normalisation. It cut per-epoch training time by more than 30% over a standard GRU. It also improved recognition accuracy consistently — across tasks, features, noise conditions, and both DNN-HMM and end-to-end CTC paradigms. Zhou and colleagues got to one gate independently in 2016, with the Minimal Gated Unit: “Experiments on various sequence data show that MGU has comparable accuracy with GRU, but has a simpler structure, fewer parameters, and faster training.” Two teams, two years apart, found the same gate expendable.

Logs and probes make the hidden mechanism less mysterious — but only against a baseline of what the gates were set up to do. One initialization mattered more than the architecture search that found it. That search covered over ten thousand different RNN architectures in 2015. It reported the finding in a line: “We found that adding a bias of 1 to the LSTM’s forget gate closes the gap between the LSTM and the GRU.” Keras ships it as the default. Its LSTM layer documents unit_forget_bias as Boolean, default True: “If True, add 1 to the bias of the forget gate at initialization.” — “This is recommended in Jozefowicz et al.” A forget gate that starts open is therefore the intended condition. The diagnostic question is not whether it sits high. It is whether it can still close when the sequence demands it.

The choice should reflect task horizon and deployment constraints rather than habit. Step 2 of the process below — matching parameter budgets — is not a hypothetical. It has been run in public, twice, in the same paper.

Three cells were sized to the same budget instead of the same width. For polyphonic music that meant 36 LSTM units against 46 GRU units against 100 tanh units at roughly 20×10³ parameters: 19.8×10³ for the LSTM, 20.2×10³ for the GRU. For raw speech signal modelling it meant 195 units against 227 against 400, at roughly 169×10³ parameters — 169.1×10³ and 168.9×10³. The gated units clearly beat tanh. Between the two gated cells, Chung and colleagues refuse to declare a winner: “However, our results are not conclusive in comparing the LSTM and the GRU, which suggests that the choice of the type of gated recurrent unit may depend heavily on the dataset and corresponding task.” The search over ten thousand architectures found the same absence of a dominant cell. Its best candidate “outperforms both the LSTM and the recently-introduced Gated Recurrent Unit (GRU) on some but not all tasks”.

That is the strongest available argument for running the five steps on your own data. Two independent groups declined to answer the question in general. One had a matched-budget table. The other had ten thousand architectures.

Gated recurrence still compresses history into a fixed-width state. It can preserve selected information over long gaps, but retrieval remains indirect and capacity limited. When a task needs detailed access to many past elements, attention, external memory, or retrieval may provide a more appropriate path. Hybrid architectures can combine recurrence with those mechanisms.

The bottleneck is not something a better cell design has removed. That has been checked at scale. The eight-variant ablation calls itself “the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling”. Hyperparameters were “optimized separately using random search”, and the importance of each component was “assessed using the powerful fANOVA framework”. The study summarises “the results of 5400 experimental runs (≈15 years of CPU time)”, and its verdict on the eight redesigns is flat: “none of the variants can improve upon the standard LSTM architecture significantly”. Fifteen years of CPU time bought a better understanding of which two components matter. It did not buy a wider channel.