RNN Fundamentals

By the end of this lesson, you will be able to...

• trace how a vanilla recurrent neural network turns a sequence into changing hidden state;

• explain why the same parameters are reused at every time step and what that buys us;

• identify why long-range credit assignment can fail in a vanilla RNN and what evidence would reveal the failure.

Idea in one sentence: An RNN reads one item at a time, carrying a learned summary forward, but repeated updates can make old information and its training signal fade or blow up.

Core Insight

Suppose a conveyor sensor reports one value every second. A short bump is normal. A persistent drift after the bump can mean that a package is beginning to jam. The classifier cannot decide from the current value alone; it needs some context from earlier seconds.

One tempting solution is to give a feedforward network a fixed window of, say, the last ten readings. That can work when every useful pattern fits in exactly ten positions. It is awkward when sequences have different lengths, and it gives position one and position ten separate input weights even though both are measurements of the same kind.

A vanilla recurrent neural network (RNN) uses a different model. It applies one learned update repeatedly. Each update receives the current observation and the state produced by the preceding update. The state is not a log of the past. It is a small, learned summary of the past that the model can revise at every step.

That makes sequence processing visible: read, update state, move on. It also creates the central limit of a vanilla RNN. A fact from early in the sequence must survive many updates, and training must send useful credit back through every one of them.

The Small Sequence We Need to Read

Use a deliberately tiny teaching model. Let x_t be a single normalized sensor feature at time t; positive values mean stronger evidence of drift. Let h_t be one hidden-state number. A real RNN normally has vectors, but one number lets us inspect the mechanism without hiding it in matrix notation.

The update is:

h_t = tanh(W_x x_t + W_h h_(t-1) + b)

Plain meaning: mix the new observation with the previous summary, then produce the next summary.

In this example: W_x says how strongly the current sensor value matters; W_h says how much of the prior state participates in the next update; b shifts the response. tanh keeps this toy state between -1 and 1.

Technical name: h_t is the hidden state, and using the output state as an input at the next position is recurrence.

Assume these values only for this worked calculation: W_x = 0.8, W_h = 0.5, b = 0, and h_0 = 0. They are illustrative weights, not measurements from a trained warehouse model.

At time 4, the RNN does not receive the entire sequence again. It receives x_4 = 0.3 and the state h_3 = 0.320. The previous state lets the same current reading mean something different than it would at the start of a sequence.

The output can be attached at every step, for sequence labeling, or only after the final state, for a sequence-level decision. For example, a later output layer could turn h_4 into a jam-risk score. This lesson focuses on the state update; a state value alone is not yet a calibrated operational decision.

One Cell, Unrolled Through Time

It is easy to draw four separate boxes and mistakenly think the RNN has four independent cells. It has one cell with one set of parameters. We draw copies only to make time visible.

x_1, h_0 --[same parameters]--> h_1
x_2, h_1 --[same parameters]--> h_2
x_3, h_2 --[same parameters]--> h_3
x_4, h_3 --[same parameters]--> h_4

This drawing is called unrolling the RNN. The parameters W_x, W_h, and b are shared across all four rows. Sharing gives the model a useful assumption: a sensor reading should be interpreted by the same kind of state-update rule wherever it appears in the sequence. It also lets one model process a sequence with four readings or forty without adding a separate parameter block for each position.

This is an architectural preference, not a promise that every position is interchangeable. If absolute position matters, the input or the surrounding design must represent it. An RNN does not automatically know that a reading came from the first second rather than the last one; it only knows what the current input and carried state make available.

So far, we have seen that recurrence turns a stream into an evolving representation while reusing one update rule. This matters because the model can learn a state useful for the task instead of requiring an engineer to hand-write a summary statistic for every possible sequence length.

Where the Simple Memory Model Breaks

The first model of hidden state is often too generous: “it remembers the past.” It is more accurate to say that it is a fixed-size state repeatedly rewritten by learned transforms. It can preserve useful information, but it has no separate protected memory path in a vanilla RNN.

Suppose the early bump should matter only if a slow drift appears 30 readings later. The state must carry the relevant part of the bump through 29 updates while also absorbing ordinary readings. A later state can lose that evidence because it was overwritten, compressed with other evidence, or mapped into a saturated part of tanh where changing the input barely changes the output.

The forward difficulty has a matching training difficulty. To learn that the early bump mattered, the loss at a later time needs to affect the parameters and state near the early time. Training does this with backpropagation through time (BPTT): unroll the computation, then apply the same chain-rule logic used for a layered network in reverse through the time steps.

For the scalar teaching model, the local sensitivity is:

∂h_t / ∂h_(t-1) = W_h × (1 - h_t²)

The derivative of a later state with respect to an earlier state multiplies many such local sensitivities. If five illustrative factors are each about 0.5, their product is 0.5⁵ = 0.03125. A small learning signal reaches the early state. If repeated factors instead have large magnitude, the product can grow rapidly. In vector RNNs these are matrix Jacobians rather than single numbers, but the same repeated-product pressure produces vanishing or exploding gradients.

This is evidence about a learning path, not proof that every long sequence fails. A trained model can handle useful dependencies over some distance. The boundary appears when performance depends on early context yet the model behaves almost as if that context were absent, or when gradient norms become extremely small or unstable during training. Compare a short-context baseline with a task that deliberately varies the required delay; do not infer long-memory ability merely because the model accepts a long tensor.

What Changes When We Use Recurrence

Before recurrence, the package classifier could treat each reading independently or consume a fixed, flattened window. After recurrence, it can expose a state at each time step, reuse its parameters over variable-length input, and make its current interpretation depend on a learned summary of earlier observations.

That does not turn the hidden state into an explanation of the package. A high-dimensional hidden vector is an internal representation, not a human-readable event log. For a safety-relevant setting, inspect labeled examples, delay-specific evaluation slices, and failure cases rather than treating one state dimension as a diagnosis.

There is also a trade-off. Unrolling makes computation and training work grow with sequence length, while the state bottleneck and repeated gradient path make distant dependencies harder. A vanilla RNN is a good fit for a bounded sequence problem when a compact evolving state is plausible and evidence shows the needed context is reachable. It becomes risky when the task requires reliable retention across long, interfering spans. The next lesson introduces LSTMs and GRUs, which add learned gates to control what is kept, changed, and exposed.

Common Confusions

Confusion: “An RNN stores every earlier input.”

Why it is tempting: the hidden state moves through the whole sequence. Better model: it is a fixed-size learned summary. The useful question is not whether it has seen an input, but whether the trained state still carries information about it when the output needs it.

Confusion: “Four unrolled boxes mean four parameter sets.”

Why it is tempting: the drawing shows four copies. Better model: the copies show four executions in time. The same W_x, W_h, and b are used at every step.

Confusion: “Variable-length input means long-range memory is solved.”

Why it is tempting: the loop can continue for any number of steps. Better model: accepting a longer sequence and learning credit across a long dependency are different capabilities. BPTT still has to carry useful gradient through repeated updates.

Check Your Understanding

Check: In the worked table, why is h_3 not zero even though x_3 is zero?

Think first, then reveal.

Answer: The update includes 0.5h_2. The earlier bump changed h_2, so part of that state enters the step-3 calculation. A zero current input does not erase the previous state by itself.

Check: A model unrolls for 20 steps and uses the same W_h at each one. What is the main concern if the task depends on evidence from step 1 at step 20?

Think first, then reveal.

Answer: The early information may be overwritten in the forward state, and the training signal from step 20 must pass through many repeated derivatives during BPTT. Either effect can make the distant dependency difficult to learn.

Practice: Choose the Missing Evidence

A team trains a vanilla RNN to flag a jam when a brief wobble is followed by a slow drift 40 readings later. Overall validation accuracy looks good, but the team has not separated examples by the wobble-to-drift delay.

What is the smallest useful next check, and what would it tell the team?

Model answer: Evaluate the same labeled pattern in delay buckets, for example 2–5, 10–15, 20–25, and 35–40 readings. If performance drops mainly as the delay grows while the local sensor pattern is otherwise comparable, that is evidence that the model is not retaining or learning the long dependency reliably. It does not by itself identify whether the cause is data scarcity, state interference, or vanishing/exploding gradients; inspect gradient norms and controlled baselines before choosing a remedy.

Resources

• [DOCUMENTATION] PyTorch: RNN — Focus: the recurrence equation and the input, output, and hidden-state tensor contract.
• [TUTORIAL] Dive into Deep Learning: Recurrent Neural Networks — Focus: unrolling and backpropagation through time.
• [ARTICLE] CS231n: Recurrent Neural Networks — Focus: read the hidden state as a shared computation across sequence positions.
• [PAPER] Learning Long-Term Dependencies with Gradient Descent Is Difficult — Focus: why repeated temporal derivatives make long-range learning difficult.

Key Takeaways

• A vanilla RNN uses the current input and prior hidden state to produce the next hidden state, so sequence context can change the meaning of the current item.
• Unrolling reveals repeated execution in time, not separate cells: one parameter set is shared across the sequence.
• Hidden state is a learned, fixed-size summary, not a perfect memory of every input.
• Long-range dependencies can fail because information is repeatedly rewritten and BPTT multiplies many local sensitivities; delay-specific evaluation and gradient signals make that boundary inspectable.