Sequence-to-Sequence Models

By the end of this lesson, you will be able to...

• trace a classic encoder–decoder model from a source sequence to an autoregressively generated target sequence;

• distinguish teacher forcing during training from feedback of the model's own outputs at inference;

• identify when a fixed encoder context is a useful simplification and when it becomes a bottleneck.

Idea in one sentence: A sequence-to-sequence model first encodes one ordered sequence, then generates another one token by token, but a single fixed summary can lose the details the decoder later needs.

Core Insight

Suppose a warehouse system receives a short sequence of event codes:

WOBBLE  QUIET  QUIET  DRIFT  LANE-4

The system must produce a short operator message:

inspect belt on lane 4

This is not ordinary sequence classification. A classifier could assign one fixed label such as possible_jam. Here the output is another ordered sequence whose length need not match the input. It must decide which words to emit, in which order, and when to stop.

The first reasonable model is to let one recurrent network read the event codes and emit a message somehow. That leaves two jobs tangled together: retain the source meaning and manage the target sentence. A classic sequence-to-sequence (seq2seq) design separates them. An encoder reads the source sequence. A decoder uses the resulting context to generate the target one token at a time.

This separation makes the transformation inspectable. It also makes a limitation visible: if the encoder gives the decoder only one fixed vector, every source detail that matters later must fit into that one vector.

The Contract Is a Transformation, Not a Label

Write the bounded task contract before selecting a model:

source:  a variable-length sequence of declared event-code tokens
target:  a variable-length sequence of approved operator-message tokens
start:   <BOS> (beginning of sequence)
stop:    <EOS> (end of sequence)

The vocabulary and the preprocessing are part of the contract. For this lesson, LANE-4 is one source token and 4 is one target token. That is a teaching simplification; a production language boundary would need a stated tokenization, vocabulary, validation, and safety policy.

The distinction from nearby designs matters:

Seq2seq is useful when generating the target sequence is genuinely part of the task. It is not automatically better than a classifier or template. If safety requires exactly one of five approved messages, a constrained classifier plus template may be easier to validate than free token generation.

The Encoder Reads the Source Once

In the original recurrent design, an encoder uses an RNN, LSTM, or GRU cell to update state while reading source tokens. Let e_t be its state after source item x_t:

x_1 -> encoder -> e_1
x_2, e_1 -> encoder -> e_2
x_3, e_2 -> encoder -> e_3
x_4, e_3 -> encoder -> e_4
x_5, e_4 -> encoder -> e_5 = context

For the classic fixed-context design, the final encoder state e_5 becomes a context vector, c. Plainly: c is the compressed source summary handed to the decoder. The encoder's job is not to choose the final words. Its job is to make information from the source available in a representation the decoder can use.

The previous two lessons make the recurrent part less mysterious. The encoder has shared parameters across source positions, and gated cells can selectively retain useful state. Seq2seq changes the architectural boundary: the end of source reading becomes the start of target generation.

The Decoder Writes One Token at a Time

The decoder has its own recurrent state, d_t. At each target position it combines its previous state, the previous target token, and the context c, then produces a distribution over the next token.

previous token + previous decoder state + context
                    -> decoder state
                    -> scores for the next token
                    -> choose one token

For the source above, a simplified inference trace looks like this:

The decoder is autoregressive: its earlier generated token becomes part of the input for the next decision. It does not need the target to have five tokens because the source had five tokens. It stops when its output policy chooses <EOS> or reaches a stated maximum length.

This trace is a teaching model. In an actual model, the next-token choice comes from a probability distribution, and decoding policy matters. Greedy selection, beam search, or constrained decoding can produce different target sequences from the same scores. The next lesson examines this feedback loop directly through text generation.

So far, we have seen that encoder and decoder divide a sequence transformation into source representation and target generation. This matters because we can inspect whether an error arose while interpreting the source, carrying context, or feeding prior generated output back into the decoder.

Training Gives the Decoder a Helpful Previous Token

During training, the target message is known. The common procedure, teacher forcing, supplies the true preceding target token to the decoder. For the target <BOS> inspect belt on lane 4 <EOS>, the training pairs are:

This makes the local prediction problem well-defined: at every step, the decoder receives a correct history. At inference, it cannot receive the unknown true message. It must instead receive its own previous prediction.

Suppose it predicts check instead of inspect at step 1. At step 2 it now conditions on check, a history it did not receive in this exact form during teacher-forced training. The error can change later choices or compound into a malformed message. This difference is called exposure bias or train–inference mismatch in this setting.

Teacher forcing is not a mistake; it is an efficient training choice. The boundary is that low teacher-forced loss does not alone establish generation quality. Evaluate complete decoded sequences with the same decoding policy that the system will use. For an operator-message task, also inspect invalid tokens, omitted lane identifiers, unsafe commands, and whether the model stops correctly.

Where One Fixed Context Runs Out of Room

The fixed-context architecture is beautifully small:

entire source -> one context vector c -> all decoder decisions

It works as a useful teaching model and can be adequate for short, regular transformations. But now extend the source with a maintenance override, an earlier lane change, a sensor-confidence code, and two separate fault events. The target may need one particular detail near the end of the output. The decoder must recover that detail from the same single c that also represents every other source fact.

That is the fixed-context bottleneck. It is not a claim that vectors have a literal, known number of “fact slots.” It is a design pressure: all source information crosses one fixed-size boundary before generation begins. Longer or more detailed inputs make that boundary harder to use reliably.

The first diagnostic is not “the source is long, so the design is wrong.” Construct evaluation slices that vary source length and place the required detail at different source positions. If exact lane or event details disappear disproportionately on longer inputs or when their position moves, the fixed-context boundary is a credible hypothesis. Data imbalance, tokenization, decoding policy, and labeling errors can produce similar symptoms, so compare controlled baselines before declaring the cause.

Later architectures let a decoder consult encoder states more selectively rather than relying only on one final vector. That response is the motivation for attention; its mechanism belongs to a later track boundary and is not needed to understand the encoder–decoder contract here.

Trade-offs and Limits

Encoder–decoder seq2seq gives a clear transformation boundary and supports unequal input and output lengths. It costs sequential decoding: each new token depends on the preceding decoder state and generated history. It can still fail through source compression, early decoding errors, incorrect stopping, or an output that is fluent but unsupported by the event codes.

For a constrained task, the best design depends on the promise. A template may be preferable when wording must be deterministic. A seq2seq decoder may be justified when the output order and length genuinely vary and the evaluation can test complete sequences. The signal to watch is not aggregate token accuracy alone; it is exact task behavior such as preserving required source fields, producing a valid <EOS>, and avoiding unsupported or unsafe target tokens.

Common Confusions

Confusion: “Seq2seq means the source and target must have equal length.”

Why it is tempting: small diagrams often show matching rows. Better model: the decoder emits until <EOS>, so target length is a learned generation outcome under a maximum-length boundary.

Confusion: “The decoder sees the true previous word while generating.”

Why it is tempting: that is what teacher forcing shows during training. Better model: at inference, it normally receives its own previous output. This is why full decoded-sequence evaluation matters.

Confusion: “A fixed context is an exact copy of the source.”

Why it is tempting: diagrams use one neat arrow from encoder to decoder. Better model: it is a learned, fixed-size representation. The fixed boundary can become a bottleneck when later target steps need detailed source information.

Confusion: “Any text output requires seq2seq.”

Why it is tempting: the target is a sequence of words. Better model: if only a small approved message set is allowed, classification or structured extraction plus a template can provide a clearer contract and safer evaluation.

Check Your Understanding

Check: During teacher forcing, which token does the decoder receive before it is trained to predict belt in the example target?

Think first, then reveal.

Answer: It receives the true preceding token, inspect. During inference, it would receive whatever token it generated at the preceding step instead.

Check: A model preserves the source's broad fault type but starts omitting lane identifiers as source sequences gain extra unrelated events. Which design pressure is plausible?

Think first, then reveal.

Answer: The final encoder context may be a bottleneck: all source information must cross one fixed-size representation before decoding. Test position and length slices before treating that hypothesis as proved.

Practice: Select a Safer Transformation Boundary

A plant has five legally approved operator messages. Each message is determined by three extracted fields: fault type, lane, and urgency. The wording must be exact and auditable.

Should the first design be free seq2seq generation, or structured prediction plus a fixed template? Give a reason and name one test that would validate your choice.

Model answer: Start with structured prediction plus a fixed template. The output space is small, wording must be exact, and the fields provide an inspectable contract; free generation adds decoding failure modes without a demonstrated benefit. Validate field extraction on held-out examples, then verify that every permitted field combination produces exactly one approved message and that no invalid combination is emitted.

Resources

• [PAPER] Sequence to Sequence Learning with Neural Networks — Focus: the original recurrent encoder–decoder formulation.
• [PAPER] Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation — Focus: variable-length encoding and decoding with gated recurrence.
• [PAPER] Neural Machine Translation by Jointly Learning to Align and Translate — Focus: the fixed-context bottleneck that motivated selective encoder access.
• [TUTORIAL] PyTorch: Translation with a Seq2Seq Network and Attention — Focus: inspect the encoder/decoder interface before studying its attention extension.

Key Takeaways

• Seq2seq transforms one ordered sequence into another; it is not merely a classifier with a longer label.
• The encoder creates source context, while the decoder generates the target autoregressively until <EOS>.
• Teacher forcing makes training history correct, whereas inference feeds back the model's own outputs; evaluate complete decoded sequences.
• A single fixed context can be adequate for short transformations but becomes a testable bottleneck when later output needs detailed source information.