Attention Is All You Need

Note

Paper: Attention Is All You Need

Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin

Year / Venue: 2017 (arXiv)

Why I’m reading this: After reading the Bahdanau et al. (2014) and Luong et al. (2015) papers, this paper is the next step in the evolution of attention-based neural machine translation.

1. The One-Sentence Idea

Instead of reading a sentence one word at a time like an RNN, let every word compare itself directly to every other word at once — that parallel comparison, stacked in layers, is the power of transformers.

2. The Problem

The nature of sequential processing used before using recurrent networks precludes parallelization, becoming a critical issue at longer sequence lengths. The transformer avoids the need for recurrence and insted rely entirely on attention to draw global dependancies between input and output.

3. The Key Idea (How They Fixed It)

The transformer architecture uses self-attention mechanisms to process sequences in parallel. An encoder that consists of a stack of identical layers, each with two sub-layers: a multi-head self-attention mechanism and a position-wise fully connected feed-forward network. The decoder is also composed of a stack of identical layers, with an additional multi-head attention layer that performs attention over the output of the encoder stack. The model uses positional encodings to retain information about the order of the sequence.

Masking introduced becasue transformer does not process sequences in order, so it might attend to future tokens. Masking prevents this by setting the attention weights of future tokens to zero.

Scaled Dot-Product Attention:

Because there is no hidden statss:

Query ──────┐
            │
Key ────────┤ → QKᵀ → softmax → weights
            │                         │
Value ────────────────────────────────┘
                                      │
                                      ▼
                                   output

4. How This Connects to What I Already Know

This paper didn’t introduce attention to me — Bahdanau and Luong already had. What it did was strip attention out of its RNN scaffolding and show it could stand on its own as the entire mechanism, not bound onto a recurrent encoder-decoder. Cross-attention in the Transformer’s decoder is literally the same soft-alignment idea from Luong’s paper (a query asking “what source information do I need right now,” compared against keys, blended from values) — just without a hidden state driving it.

Building the encoder-decoder from scratch made a few of the paper’s specific design choices click in a way reading alone never would have:

• The √d_k scaling factor stopped being an arbitrary formula the moment I saw my own toy Q/K/V example collapse onto a single dominant key, purely because of vector magnitude, not real relevance.

• Positional encoding’s necessity was obvious in theory, but I only really felt it after confirming with my own code that shuffling token order left the raw attention scores completely unchanged.

• The warmup learning-rate schedule went from “an odd formula in section 5.3” to something I understood after training my own Transformer with a flat learning rate and watching it collapse to predicting <unk> for every single word — then watching the exact Noam schedule from the paper fix it.

Reading the paper this time was less “learning something new” and more “getting the precise vocabulary and justification for things I’d already stumbled into by building it.”