Seq2Seq Model architecture



In traditional NNs (e.g. MLPs, CNNs), the input and output are fixed-size vectors. However, many tasks involve variable-length sequences (e.g. sentences, time series). For this reason we need a different architecture that can handle sequences of varying lengths. In sequence-to-sequence (seq2seq) models, the architecture can deal with these:

We will focus on the Many to Many case, and follow it with a practical example of machine translation.

We will discuss the following:

Architecture

The seq2seq architecture consists of two main components: an encoder and a decoder: - Encoder: The encoder takes the input sequence and processes it to produce a fixed-size context vector. Capturing the meaning of the input sequence in a single vector.

  • Decoder: The decoder takes the context vector from the encoder and input tokens then generates the output sequence one token at a time.

If we take the example of trainng to translate English to Arabic, where the english input is “Ahmed loves reading” and the output is “أحمد يحب القراءة”, the encoder will process the English sentence and produce a context vector that captures its meaning. The decoder then takes this context vector and generates the Arabic sentence.

Here is a simple diagram:

seq2seq

Each component (encoder and decoder blocks) can be implemented using different types of neural networks, RNN, LSTM, GRU, or even Transformers. Here is how it looks like with LSTMs:

seq2seq-lstm

The \(h_t\) and \(c_t\) are the hidden and cell states of the encoder containing the information about the input sequence. The decoder uses these states to generate the output sequence.

Training

In training the loss error signal flows from the decoder back to the encoder updating the weights of both components. The decoder input contains two special tokens:

  • ‘’ (start of sequence) — the decoder’s first input, the signal “begin generating.

  • ‘’ (end of sequence) — what the decoder learns to emit when the translation is done.

For the above example, the decoder input and target would be:

  • Input: [‘’, ‘أحمد’, ‘يحب’, ‘القراءة’]

  • Target: [‘أحمد’, ‘يحب’, ‘القراءة’, ‘’]

Teacher Forcing

In training the Seq2Seq model, we feed the output of the first decoder step (the predicted token) as the input to the next step. However, early in training the decoder is terrible — if you feed its own wrong guesses back as input, the errors compound: one bad guss and the whole rest of the sentance never gets a useful and meaningful meaning.

The solution of this is to use teacher forcing, in which the next time step (t+1) does not get the prediction from previous time step (t) as the input, inseted it gets the true value. So even if the decoder messes up step 3, step 4 still gets the correct token 3 as input.

Tip

The analogy:

teacher forcing is like a music student practicing with a teacher who gently puts their hand back in the right place after every wrong note — instead of letting them play an entire piece off-key from one early slip.

Example - Machine Translation

The following is a detailed experiment for training a seq2seq model for translating English sentances to Arabic. English to Arabic translation Experment

Findings

We used simple 100 data pairs for training:

pairs = [
    ("hello", "مرحبا"),
    ("goodbye", "مع السلامة"),
    ("thank you", "شكرا لك"),
    ....
]

Both encoder and decoder are built with LSTM, with hidden_size=64 and embed_size=32. The encoder reads the English sentence and produces a context vector (hidden, cell), which the decoder uses as its starting state to generate the Arabic sentence word by word.

After training for 300 epochs, the loss drops close to zero — the model memorizes all 100 pairs.

The bottleneck problem

When we tested the model on inputs it had never seen, something revealing happened:

print(translate("sad happy hungry me"))   # → الأب يقرأ الجريدة
print(translate("see her please"))        # → أعيش في القاهرة

The outputs are fluent, correct Arabic — but completely unrelated to the input. The model is retrieving memorized sentences rather than translating.

This is not a bug. This is the bottleneck problem, a fundamental limitation of the plain seq2seq design.

The encoder crushes the entire English sentence into a single fixed-size vector (h, c). All word-level detail — which words appeared, in what order — gets compressed into those two vectors before the decoder sees anything. This led to the model assigning an ID code for each whole sentance, not finding word meanings. When feeding new sentances, the context vector lands somewhere between memorized examples, then the decoder just picks the colsest ID codes (arabic vectors) it recorded during training.

The decoder never got to look at the word “happy.” It only saw the squeezed summary. The alignment between English words and Arabic words was thrown away at the bottleneck.

What comes next — Attention

The fix is attention (Bahdanau et al. 2014). Instead of forcing the decoder to work from a single (h, c), attention lets the decoder look back at every encoder step — one for each English word — and decide which words to focus on when generating each Arabic word.

This restores the word-level alignment the bottleneck discarded. The decoder generating “يحب” can now attend strongly to the English word “loves” and ignore “Ahmed” and “reading” — something a fixed vector simply cannot do.

Comments