flowchart LR
x1(["I"]) --> h1(("h₁"))
x2(["love"]) --> h2(("h₂"))
x3(["machine"]) --> h3(("h₃"))
x4(["learning"]) --> h4(("h₄"))
h1 --> h2 --> h3 --> h4
Neural Machine Translation by Jointly Learning to Align and Translate
Note
Paper: Neural Machine Translation by Jointly Learning to Align and Translate
Authors: Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio
Year / Venue: 2014 (arXiv), presented at ICLR 2015
Why I’m reading this: I found the seq2seq bottleneck myself while training an English→Arabic translator — the model memorized whole sentences instead of translating, because the encoder crushes everything into one fixed-size vector. See my seq2seq bottleneck writeup. This is the paper that names that exact problem and proposes attention as the fix.
1. The One-Sentence Idea
The current encoder-decoder translation model is limited and a bottleneck in improving the translation quality. The authors propose a new approach * by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word.*
2. The Problem
The idea of encoder-decoder translation is to take a source sentence, encode it into a fixed-size vector, and then decode that vector into a target sentence(Translation). The problem is that the fixed-size vector is copressing all the necessary information of the source sentence into a single vector, leading to lose individual word meanings and their alignments, also the model is unable to handle long sentences.
3. The Key Idea (How They Fixed It)
Instead of encoding the source sentence into a single fixed-size vector, the authors propose to encode it into a sequence of vectors, then during decoding, the model can attend to those vectors and decides which parts of the source sentance to focus on to generate the next target word.
4. Simplfied Explanation of the Model
Encoder
They used a bidirectional RNN as the encoder, which reads the source sentence and produces a sequence of hidden states, and each hidden state contains information about the whole source sentence with emphasis around that word.
If the input sentence is “I love machine learning”, the encoder will produce a sequence of hidden states \(h_1, h_2, h_3, h_4\) for each word in the sentence. Each hidden state \(h_i\) is a vector that contains information about the whole sentence with emphasis on the word \(x_i\).
Decoder
The previous archeticture used a single context vector \(c\) to generate each word in the target sentence. The new architecture uses a different context vector \(c_i\) for each target word \(y_i\). How these \(c_i\) vectors are computed??
The encoder produces a sequence of hidden states \(h_1, h_2, ..., h_T\) (annotations) for the source sentence. The \(c_i\) is just a weighted sum of these annotations; multiplying each annotation \(h_j\) by a weight \(\alpha_{ij}\), then summing them up to get \(c_i\).
\[ c_i = \sum_{j=1}^{T} \alpha_{ij} h_j \]
NOw how the weights \(\alpha_{ij}\) are computed?
They first compute a score called alignment score \(e_{ij}\). While the decoder is generating the \(i^{th}\) target word, it has a hidden state \(s_{i-1}\), which contains information about everything translated so far. The alignment score \(e_{ij}\) uses this hidden state \(s_{i-1}\) and look to all the encoder hidden states \(h_j\) to compute a weight for each \(h_j\).
Then for each \(e_{ij}\), they apply a softmax function to get the weights \(\alpha_{ij}\).
\[ \alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T} \exp(e_{ik})} \]
The alignment model is essentially asking one question:
“Given what I’ve translated so far, which source word should I focus on next?”
5. How This Connects to What I Already Know
Taking the single context vector \(c\) like we did in the seq2seq model, then relay on that vector only to generate the target sentence is destructive, because the model is forced to compress all the information of the source sentence into a single vector. The solution this paper proposes is to use a different context vector \(c_i\) for each target word \(y_i\). While generating the \(i^{th}\) target word, the model needs to look back at the source sentence and decide which parts of the source sentence are relevant to generating the next target word. This is called attention.
6. Implementation Code
You can find the implementation code in my attention_v1.ipynb notebook. The code is based on the PyTorch framework and implements the attention mechanism as described in the paper. It includes the encoder, decoder, and the attention mechanism.
Comments