Effective Approaches to Attention-based Neural Machine Translation



Note

Paper: Effective Approaches to Attention-based Neural Machine Translation

Authors: Minh-Thang Luong, Hieu Pham, Christopher D. Manning

Year / Venue: 2015

Why I’m reading this: Following the Bahdanau et al. (2014) paper, this paper proposes a new attention mechanism called “global attention” and “local attention” that improves the performance of neural machine translation models.

1. The One-Sentence Idea

Improving on the attention mechanism proposed by Bahdanau et al. (2014) by introfucing two new types of attention: global attention and local attention.

2. The Problem

Luong is asking:

“Can we compute attention in a simpler and faster way?”

Bahdanau et al. (2014) used a small neural network to compute the attention weights resulting from comparing the decoder and encoder hidden states. Loung however, used a simpler approach by using a dot product between the decoder hidden state and the encoder hidden states to compute the alignment scores then the attention weights.

Another problem is that the global attention is exprensive to compute, So the authors asked: “Do we really need to look at every source word every time to get the context vector?”

3. The Key Idea (How They Fixed It)

Both local and global attention mechanisms works by taking the decoder’s current hidden state \(h_t\), and derive a context vector \(c_t\) that captures the relevant information from the encoder’s hidden states. The context vector \(c_t\) is then used to generate the next target word. The difference between the two mechanisms is how they compute the context vector \(c_t\):

Global Attention:

Suppose the input sentence is “I love machine learning”. The encoder produces a sequence of hidden states \(h_1, h_2, h_3, h_4\), one for each word.

I      love      machine      learning

h₁     h₂        h₃          h₄

When generating one target word, global attention, at each step takes the current decoder hidden state \(h_s\) and looks at every encoder hidden state:

flowchart TD
    s(["Decoder hs₁"]) --> h1(("h₁"))
    s --> h2(("h₂"))
    s --> h3(("h₃"))
    s --> h4(("h₄"))

Every encoder hidden state participates — this is what makes it global, as opposed to local attention, which only looks at a small window around one position.

this coparison gives the alignment score \(a_t(s)\) that is used to compute the context vector \(c_t\) as a weighted sum of all the encoder hidden states.

The alignment score: \[ \text{score}(h_t, \bar{h}_s) = h_t^\top \bar{h}_s \]

Decoder hidden
hₜ = [2 1 3]

h₂ = [1 2 3]

The dot product is: 2×1+1×2+3×3 = 13

A large dot product means that the two vectors are similar. It simply answers the question: “How similar are these two vectors?”.

After getting the score, in each time step, they apply a softmax function to get the attention weights \(a_t(s)\).

Then the context vector \(c_t\) : \[ c_t = \sum_{s=1}^{S} a_t(s) \bar{h}_s \]

Soft Attention: Soft attention proposed to address the expensive computation of the global attention mechanism. Instead of looking at all encoder hidden states, soft attention only looks at a small window of context. To do that, they first predict a position \(p_t\) in the source sentence, and a window of size \(D\) around that position is used to compute the context vector \(c_t\).

  • How do they compute the position \(p_t\)?
    • Monotonic Attention (local-m): decoder at step 1 looks at encoder hidden state 1, and so on. This is a simple approach, but it assumes that the source and target sentences are monotonic, which is not always the case.
    • Predictive Alignment (local-p): the model learns where to look in the source sentence by predicting a position \(p_t\) using a small neural network.

In Predictive Alignment (local-p), the process of finding the alignment score \(a_t(s)\) is as follows: - Predict center position \(p_t\) using a small neural network. - Select a local window of size \(D\) around \(p_t\). - Compute alignment scores (same as Global Attention) given the window of encoder hidden states. - Multiply by a Gaussian centered at \(p_t\).

\[ a_t(s) = \text{align}(h_t, \bar{h}_s) \exp\left(-\frac{(s - p_t)^2}{2\sigma^2}\right) \]

4. The Evidence

What experiment, table, or figure actually convinced you the idea works? Quote a real number or result — not “it performed better,” but “BLEU score went from X to Y on dataset Z.” Also note: what are they comparing against — is that comparison fair?

5. Terms I Had to Look Up

Running glossary. Every paper has 3-5 words that stop you cold — write them here as you hit them, in your own definition, not the textbook one.

  • term: meaning

6. A Question I’d Push Back On

Pick one thing you’re skeptical of or confused by — an assumption, a missing comparison, a claim that feels too strong. You don’t need to be “mean” (per your original notes) — genuinely curious/skeptical is enough for a first pass.

7. How This Connects to What I Already Know

Link to your own posts/experiments. Did this paper explain something you’d already noticed, contradict something you assumed, or introduce something totally new?

My Takeaway

2-3 sentences: if you forgot everything else about this paper, what’s the one thing you’d want to remember?

Comments