Text Generation with RNNs

By the end of this lesson, you will be able to...

• trace an RNN generator from a start token through next-token scores, decoding, feedback, and a stop token;

• predict how greedy decoding, temperature, and top-k sampling change a next-token distribution;

• diagnose why good teacher-forced loss can coexist with weak free-running output.

Idea in one sentence: An RNN generator predicts a distribution for one next token, chooses from it, feeds that choice back into its own state, and therefore makes decoding policy part of the system's behavior.

Core Insight

Suppose the event-code sequence from the previous lesson has already been encoded as context for an operator message. The decoder begins with <BOS> and should generate:

inspect belt on lane 4 <EOS>

It is tempting to think that text generation means “ask the network for the sentence.” The actual mechanism is smaller and more fragile. The network offers scores for the next token. A decoding rule chooses one token. That token becomes part of the context for the next choice. The loop continues until <EOS> or a declared length limit.

The initial model says that a high-probability next token is enough for a good message. It works for one local prediction. It breaks when an early chosen token becomes an input for later steps. The stronger model treats generation as a stateful feedback loop with two separate components: the model distribution and the decoding policy that acts on it.

What the RNN Produces at One Step

At generation step t, the recurrent decoder combines the previous token embedding, its previous hidden state, and any encoder context. It produces logits—unnormalized scores—over the vocabulary:

previous token + h_(t-1) + encoder context
                 -> RNN update -> h_t
                 -> logits over vocabulary
                 -> probability distribution

For a vocabulary of three candidate words, a softmax turns logits into probabilities. Use these synthetic logits after a particular prefix:

The numbers are an illustrative decoder output, not a measurement from a trained model. The model has not said “belt is true.” It has assigned higher probability to belt under this prefix and its learned representation. Whether that is an acceptable output depends on the source evidence, vocabulary contract, and decoding rule.

The core training objective is next-token prediction. For a known target, the model is trained to assign high probability to the true next token conditioned on an earlier target prefix. RNN hidden state carries the prefix information forward instead of recomputing the whole prefix as a new independent feature vector at each step.

Run the Generation Loop

Here is a compact free-running trace. Assume the encoder context describes a wobble followed by drift on lane 4. The selected tokens are illustrative greedy choices, not a model execution.

This is autoregressive generation: each predicted token changes the next input and hidden state. A length limit is a safety boundary, not an alternative to a learned stop token. If the model fails to emit <EOS>, the caller needs a declared behavior such as truncation and an invalid-output result rather than silently treating a run-on sentence as complete.

So far, we have seen that a generated sequence is produced by a chain of local choices. This matters because one apparently harmless choice can alter every later input to the decoder.

Training History and Inference History Are Not the Same

During teacher-forced training, the target message is known. To train the prediction of belt, the decoder receives the true earlier token inspect. At free-running inference, it receives its own earlier choice instead.

Consider a second, synthetic trace. At step 1, a stochastic policy chooses check rather than the intended inspect. The next prediction is now conditioned on check, a prefix that differs from the target history used at that point in teacher-forced training. It may continue with sensor, then omit the lane, even if the model had low loss on the correct local pairs.

training path:  <BOS> -> inspect -> belt -> on -> lane -> 4
inference path: <BOS> -> check   -> sensor -> ...

This is a visible form of train–inference mismatch, often called exposure bias. It does not mean teacher forcing is invalid. It means teacher-forced loss measures a narrower question: “how well did the model predict the next token given correct previous tokens?” A generator also needs evaluation while consuming its own outputs.

Useful evidence includes complete-sequence validity, required-field preservation, correct stopping, repetition rate, and error cases by generated length. Token accuracy alone can hide an unusable sequence: five locally likely tokens can still produce a message with the wrong lane or no <EOS>.

Change the Decoding Policy, Change the Output Distribution

The model's logits are not yet a token choice. Three policies expose different trade-offs.

Greedy decoding always selects the highest-probability token. In the toy distribution, it selects belt. It is deterministic for fixed model state and inputs, which can be useful for a tightly constrained system. But it may repeat a locally favored token or miss a globally better continuation because it never considers another path.

Temperature sampling rescales logits before softmax:

probabilities = softmax(logits / T)

With the illustrative logits [2.0, 1.0, 0.0], the distribution changes as follows:

Low temperature does not make a model more truthful. It makes the policy more concentrated on the model's existing preference. High temperature does not make output creative in a useful sense; it gives more probability mass to alternatives, including bad ones.

Top-k sampling keeps only the k highest-scoring candidates, renormalizes their probabilities, and samples from that reduced set. With k = 2, stop is excluded from this particular toy choice. This can prevent an implausible tail token from being sampled, but it can also remove a valid rare option. It does not repair an incorrect model distribution or verify that the generated message is supported by the source.

For the operator-message contract, greedy or explicitly constrained decoding is usually the safer preference because wording and fields must be auditable. For an open-ended creative task, controlled sampling may be useful. The preference follows the output constraint, not a universal decoding rule.

Where the Loop Breaks

The feedback loop gives generation its flexibility and its risks. Early errors can change the later state; a peaked but wrong distribution can produce confident nonsense; a flat distribution can inject irrelevant tokens; and a decoder can repeat a high-probability pattern or stop at the wrong time.

Inspect the loop, not just the final sentence. Save the prompt, chosen token, top candidate probabilities, temperature or top-k setting, generated length, and stop reason for a bounded evaluation set. These are diagnostic artifacts, not proof that a model understands the source. They let an evaluator locate whether a failure began in the distribution, the policy, feedback after an early token, or the stop boundary.

The trade-off is direct: more deterministic decoding improves repeatability and auditability but can lock in a local mistake; broader sampling explores alternatives but raises variability and the chance of unsupported output. Neither choice solves an encoder bottleneck, poor training data, or the underlying RNN's limited long-range memory.

Review Check: Which Structure Does the Model Carry?

The recurrent cluster now provides a contrast with the CNN cluster. A CNN carries local spatial structure through shared filters and spatial feature maps. An RNN generator carries ordered history through recurrent state and feeds chosen tokens back through time. Neither is “the neural-network way” in general; each is an inductive bias matched to a type of input structure.

For a defect photograph, shuffling pixels destroys the spatial pattern a CNN needs. For an operator message, shuffling tokens destroys the prefix history a recurrent decoder needs. The model choice should begin with the structure the task must preserve, then be checked against real evidence.

Check Your Understanding

Check: A decoder assigns probabilities belt=.665, sensor=.245, and stop=.090. What does greedy decoding choose, and what does that choice become at the next step?

Think first, then reveal.

Answer: It chooses belt, the highest-probability token. The decoder then receives belt with its updated state as part of the next generation step.

Check: What changes when temperature rises from 0.5 to 2.0 for fixed logits?

Think first, then reveal.

Answer: The distribution becomes flatter, so lower-scoring candidates receive more probability and sampled output becomes more variable. The model weights and source evidence have not changed.

Practice: Choose a Decoding Boundary

A creative writing tool may generate several plausible continuations. A maintenance console may emit only approved instructions that must name the correct lane and stop cleanly.

Choose a decoding preference for each system and state one evaluation signal that would reveal its boundary.

Model answer: The creative tool can use a stated temperature and top-k sampling because variety is part of its purpose; evaluate diversity alongside repetition and incoherence rates. The maintenance console should prefer greedy or constrained decoding with a fixed vocabulary and length limit; evaluate exact field preservation, invalid-token rate, and correct <EOS> behavior. A deterministic decoder can still repeat a model error, so it needs source-grounded validation too.

Resources

• [TUTORIAL] Dive into Deep Learning: Language Models and Dataset — Focus: next-token prediction and recurrent language-model data.
• [ARTICLE] CS231n: Recurrent Neural Networks — Focus: language modeling and the autoregressive loop.
• [PAPER] Sequence to Sequence Learning with Neural Networks — Focus: free-running decoding after teacher-forced training.
• [PAPER] Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks — Focus: a response to train–inference mismatch in autoregressive generation.

Key Takeaways

• An RNN generator predicts one next-token distribution at a time, then feeds its chosen token back into the next recurrent step.
• Teacher-forced loss describes prediction under correct history; free-running evaluation reveals feedback, repetition, and stop failures.
• Temperature and top-k change the decoding policy, not the learned model or the source evidence.
• Choose decoding from the output contract: controlled variety for open-ended work, and constrained or deterministic behavior for auditable messages.