Attention Is All You Need

Note

Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin

Year / Venue: 2017 (arXiv)

Why I’m reading this: After reading the Bahdanau et al. (2014) and Luong et al. (2015) papers, this paper is the next step in the evolution of attention-based neural machine translation.

1. The One-Sentence Idea

Force yourself to compress the whole paper into one plain-English sentence, no jargon. If you can’t, you haven’t understood it yet — that’s fine, come back to this line last.

2. The Problem

The nature of sequential processing used before using recurrent networks precludes parallelization, becoming a critical issue at longer sequence lengths. The transformer avoids the need for recurrence and insted rely entirely on attention to draw global dependancies between input and output.

3. The Key Idea (How They Fixed It)

The transformer architecture uses self-attention mechanisms to process sequences in parallel. An encoder that consists of a stack of identical layers, each with two sub-layers: a multi-head self-attention mechanism and a position-wise fully connected feed-forward network. The decoder is also composed of a stack of identical layers, with an additional multi-head attention layer that performs attention over the output of the encoder stack. The model uses positional encodings to retain information about the order of the sequence.

Masking introduced becasue transformer does not process sequences in order, so it might attend to future tokens. Masking prevents this by setting the attention weights of future tokens to zero.

Scaled Dot-Product Attention:

Because there is no hidden statss:

Luong                    Transformer

decoder hidden    →      Query (Q)

encoder states    →      Keys (K)

encoder states    →      Values (V)

4. The Evidence

What experiment, table, or figure actually convinced you the idea works? Quote a real number or result — not “it performed better,” but “BLEU score went from X to Y on dataset Z.” Also note: what are they comparing against — is that comparison fair?

5. Terms I Had to Look Up

Running glossary. Every paper has 3-5 words that stop you cold — write them here as you hit them, in your own definition, not the textbook one.

term: meaning

6. A Question I’d Push Back On

Pick one thing you’re skeptical of or confused by — an assumption, a missing comparison, a claim that feels too strong. You don’t need to be “mean” (per your original notes) — genuinely curious/skeptical is enough for a first pass.

7. How This Connects to What I Already Know

Link to your own posts/experiments. Did this paper explain something you’d already noticed, contradict something you assumed, or introduce something totally new?

My Takeaway

2-3 sentences: if you forgot everything else about this paper, what’s the one thing you’d want to remember?