LSTM and GRU

By the end of this lesson, you will be able to...

• trace how LSTM gates preserve, write, and expose a memory value over several sequence steps;

• explain how a GRU provides gated updating with one recurrent state;

• choose a bounded starting comparison between LSTM and GRU without treating either as a guarantee of long-range memory.

Idea in one sentence: LSTMs and GRUs make recurrence selective: learned gates decide how much old state survives, how much new evidence enters, and—on an LSTM—how much memory is exposed.

Core Insight

Suppose the conveyor in the previous lesson reports a sharp wobble, then many quiet readings, then a slow drift. The wobble should matter only when the later drift appears. A vanilla RNN must keep the wobble useful while its one hidden state is repeatedly overwritten. When training or state updates lose that evidence, the final drift looks ordinary.

The tempting repair is “make the hidden state wider.” More dimensions can help under some constraints, but width alone does not tell the update what to protect. A gated recurrent model adds learned control values between zero and one. They are not hand-written rules such as “remember every wobble.” They are functions learned from the current input and previous recurrent state.

An LSTM gives the protected path an explicit name: the cell state. At each step, gates decide what fraction of that path to keep, what candidate information to write, and what fraction to reveal as the visible hidden state. A GRU keeps the same core idea—selective carry-forward—but uses one recurrent state and fewer gates.

The stronger model is therefore not “gates remember everything.” It is: the model can learn a controllable route through time, then we test whether that route actually carries the context needed by the task.

The LSTM Has Two Kinds of State

An LSTM carries two vectors at each time step:

• c_t, the cell state, is the memory path carried forward;
• h_t, the hidden state, is the step-level representation exposed to the next layer or output.

For one scalar teaching unit, the standard update is:

f_t = sigmoid(... current input, previous hidden state ...)
i_t = sigmoid(... current input, previous hidden state ...)
g_t = tanh(... current input, previous hidden state ...)
o_t = sigmoid(... current input, previous hidden state ...)

c_t = f_t × c_(t-1) + i_t × g_t
h_t = o_t × tanh(c_t)

Plain meaning:

• the forget gate f_t keeps a fraction of old cell state;
• the input gate i_t controls how much candidate content g_t is written;
• the output gate o_t controls how much of the current cell state becomes visible as h_t.

The names describe roles, not fixed decisions. A gate close to 1 passes most of its relevant quantity; a gate close to 0 blocks most of it. The gate values themselves are learned from data, just as the recurrent weights are.

This separates two jobs that a vanilla RNN combines in one update. The cell can preserve a useful trace even when the hidden state exposes only part of it at the current step. That distinction is helpful when the model should retain a clue without making it dominate every immediate output.

Trace One Memory Decision

Return to the conveyor. Let the cell state represent how much the model should retain a possible wobble-to-drift pattern. The numbers below are deliberately synthetic gate outputs, chosen to make the update inspectable; they are not values measured from a trained model.

Start with c_0 = 0. At time 1 the wobble arrives. At times 2 and 3 the line is quiet. At time 4 a drift arrives. We use a simple candidate g_t for “new relevant evidence.”

The quiet readings do not force a complete overwrite. Their input gates are low, while their forget gates preserve most of the existing cell value. When drift arrives, the input gate permits new evidence to combine with the retained wobble trace. The larger output gate then lets more of this combined memory appear in h_4.

Notice what this trace does and does not show. It shows a possible learned behavior of the mechanism. It does not prove that a real LSTM will learn the right gates from any data set, or that .653 is a risk probability. A real output head, loss, and validation protocol still decide how a hidden state supports a decision.

So far, we have seen that the LSTM cell update includes an additive carry path: retained old memory plus controlled new content. This matters because information does not have to pass through one undifferentiated replacement at every time step.

Why This Helps Training, but Does Not Solve It Automatically

In a vanilla RNN, temporal credit passes through repeated nonlinear updates. In the LSTM cell update, a contribution to c_(t-1) is multiplied directly by f_t on its path to c_t. When learned forget gates remain suitably close to one, that path can preserve both forward information and a useful gradient longer than a repeatedly overwritten state path.

That is a mechanism, not a guarantee. If forget gates close too often, if new writes interfere, if gates saturate in an unhelpful direction, or if the data does not make the dependency learnable, relevant context can still disappear. Longer sequences also cost more sequential computation, because a recurrent step depends on the preceding state.

The useful signal is task evidence by dependency distance. For the conveyor, group evaluation examples by the gap between wobble and drift. Then compare the vanilla RNN and the gated candidate under the same data split, state width, training budget, and output contract. A gated model that merely improves average accuracy but still fails on the long-delay bucket has not met the memory requirement.

A GRU Keeps the Gating Idea Smaller

A gated recurrent unit (GRU) does not maintain separate c_t and h_t states. It maintains one recurrent state, usually written h_t. In a common convention:

z_t = sigmoid(... current input, previous state ...)
r_t = sigmoid(... current input, previous state ...)
n_t = tanh(... current input, r_t × previous state ...)

h_t = z_t × h_(t-1) + (1 - z_t) × n_t

Here the update gate z_t chooses the balance between keeping old state and taking the candidate n_t. Under this convention, a high z_t retains more old state. The reset gate r_t decides how much previous state participates while forming the candidate. Formula conventions vary by source or library, so read the update equation rather than memorizing a gate name as “always keep” or “always replace.”

For a tiny continuation of the conveyor trace, suppose h_(t-1) = .70, the new candidate is n_t = .20, and z_t = .85 during a quiet reading:

h_t = .85 × .70 + (1 - .85) × .20
    = .595 + .030
    = .625

Most of the prior state survives. If later evidence makes z_t smaller, the candidate can replace more of it. The GRU therefore retains the important design move: update state by a learned mixture instead of a compulsory full rewrite.

Compare the Boundary, Not the Brand Name

For the same input and hidden widths, a common LSTM implementation computes four affine gate blocks; a common GRU computes three. An LSTM therefore typically has more parameters and maintains both cell and hidden state, while a GRU has a more compact state interface. That is a resource and complexity trade-off, not an ordering of quality.

Choose neither merely because its name appears in a tutorial. If the data sequence is short and the required context is local, the extra machinery may not earn its cost. If a task needs long, selective retention, compare gated variants against a clear vanilla baseline and inspect where each fails. Neither LSTM nor GRU makes an unobserved early event recoverable, and neither replaces a dataset with adequate examples of the dependency.

Common Confusions

Confusion: “A forget gate decides whether to erase a memory.”

Why it is tempting: the name sounds binary. Better model: it emits a learned continuous value. In the LSTM update, that value scales the old cell state; it can retain part of the state rather than making an all-or-nothing decision.

Confusion: “The cell state is the output.”

Why it is tempting: both values travel through time. Better model: the cell state is the LSTM's memory path. The output gate transforms that memory into the hidden state exposed at the current step.

Confusion: “Gated models eliminate vanishing gradients.”

Why it is tempting: they were designed to improve long-term credit assignment. Better model: the additive carry path can help, but learned gate values, interference, data, optimization, and sequence length still determine whether the needed dependency is learned.

Confusion: “GRU is simply a smaller LSTM.”

Why it is tempting: both use gates and address the same pressure. Better model: a GRU uses a different state design and update equation. It is a distinct trade-off, not an LSTM with a few parts removed at random.

Check Your Understanding

Check: In the LSTM table, why does the cell state remain positive at time 3 even though its candidate g_3 is zero?

Think first, then reveal.

Answer: The forget gate retains 0.95 of c_2, while the low input gate writes no new candidate. The cell update is a controlled carry-forward, not a full replacement with the current event.

Check: Under the GRU equation in this lesson, what does an update gate z_t near one do?

Think first, then reveal.

Answer: It weights h_(t-1) heavily and the new candidate lightly, so most of the old state is retained. Check the displayed equation because other notations may reverse the label or arrangement.

Practice: Test the Claimed Memory Benefit

A team must recognize a wobble only when a drift occurs 25–40 readings later. They can afford either a vanilla RNN or a GRU with the same hidden width, but only one training budget.

Design the smallest comparison that would justify choosing the GRU.

Model answer: Keep the data split, preprocessing, hidden width, output head, training steps, and threshold fixed. Evaluate both models in delay buckets, especially 25–40, and report overall quality as well as that bucket's recall and false-positive rate. Log training stability and parameter count. Choose the GRU only if it improves the required long-delay behavior enough to justify its added state machinery under the stated budget; a better average score that misses the long-delay requirement is insufficient.

Resources

• [DOCUMENTATION] PyTorch: LSTM — Focus: the four gate equations and the hidden/cell-state tensor interface.
• [DOCUMENTATION] PyTorch: GRU — Focus: the three-gate update and its state contract.
• [PAPER] Long Short-Term Memory — Focus: the original long-range credit-assignment motivation.
• [PAPER] Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation — Focus: an early GRU formulation and its reset/update gates.
• [PAPER] Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling — Focus: why LSTM-versus-GRU choice needs empirical evidence.

Key Takeaways

• An LSTM separates a carried cell state from the hidden state it exposes, using learned forget, input, and output gates.
• A GRU uses learned update and reset behavior with one recurrent state, preserving the main selective-update idea with a smaller mechanism.
• Gates can create a more controllable path for information and credit through time, but they do not guarantee useful long-range memory.
• Compare gated models against the required delay, resource budget, and failure signal; architecture names are not evidence.