⌗ ATTENTION IS ALL YOU NEED — Vaswani et al., NeurIPS 2017
Attention Is All You Need
Eight researchers made a bet that broke a decade of convention: delete the recurrence entirely. No loop, no hidden state passed through time — just every word looking at every other word at once, in parallel. Attention itself became the whole model. This lecture opens the machine block by block: every formula, every matrix, every design decision — computed live, nothing a picture.
THREE VIEWS OF ONE MACHINE
Figure 1, one-to-one — the paper’s own block diagram, redrawn exactly as printed in 2017 — and fully interactive: click any block to open its inner machinery, from the Figure-2 attention internals down to live LayerNorm and ReLU arithmetic. The explorer — the whole machine annotated, with a step-by-step TRACE. The data-flow map — the same machine as a paper-style component pipeline.
Every section below then takes one block apart: the retrieval intuition, the scaled dot-product math, eight heads drawn over a real sentence, the sin/cos position waves, the causal mask, the training recipe — all of it live.
Hover to trace a block’s wires · click to open its inner machinery.
Q · K · V — the Query comes from the decoder’s own Add & Norm; Keys and Values arrive from the encoder’s output. That middle box is cross-attention.
N × — each stack is one block repeated 6 times. The encoder (left) reads the whole sentence; the decoder (right) writes left to right.
Side loops — every attention and FFN sublayer is wrapped by a residual branch into its Add & Norm: the gradient rails of §07.
⊕ — sinusoidal positional encodings, added once at the very bottom of each stack (§06). Attention alone is order-blind.
Click any block — its inner architecture opens: the attention kernel of Figure 2, the residual rail, the position-wise MLP — computed live.
Fig. 1 — “The Transformer — model architecture.” Redrawn one-to-one from Vaswani et al., Attention Is All You Need, NeurIPS 2017.
THE MACHINE, ANNOTATED — CLICK ANY BLOCK
encoder → K, V → decoderThe same machine as a paper-style component map — generated with archify in its editorial (print) preset. Hover a component to trace its relationships, switch light/dark, or search inside. Open full screen ↗
Seq2Seq squeezed a whole sentence into one vector; attention was bolted on from outside to rescue it. 2017’s bet: if attention does the real work anyway, delete the RNN it was rescuing.
02 THE SEQUENTIAL BOTTLENECK
Why not just a better RNN?
Three lectures of RNNs left two wounds that no gate could close. Memory: information from word 1 reaches word 9 only by surviving every hidden-state hand-off in between. Speed: step 9 cannot start until step 8 is done — a sentence is a queue, and GPUs idle. Self-attention removes both at once.
So how does a word “look at” another word? The same way you find a book in a library — and the paper’s genius was turning that lookup into three learnable matrices.
03 QUERIES, KEYS & VALUES
Attention is a retrieval system
A soft version of dictionary lookup. Each word emits a query (what am I looking for?) and a key (what do I offer?); the dot product of query·key scores the match, a softmax turns scores into weights, and the word’s new representation is a weighted blend of everyone’s value. Below, every vector is hand-set and 4-dimensional so you can verify each dot product by eye.
THE SENTENCE — CLICK ANY WORD TO MAKE IT THE QUERY
SCORES & WEIGHTS — live softmax over all 12 keys
THE OUTPUT — a weighted blend of all values
WHERE EACH DIMENSION CAME FROM — one slice per word, sized by its weight
Hover a slice for the split. The blend IS the new representation of the query word.
FROM TOY VECTORS TO THE REAL THING
In the real model, every word’s q, k and v are linear projections of its embedding — three matrices the model learns by gradient descent. Same sentence, two readings: switch to “it” → wide above and watch the same pronoun attend somewhere new, because its context changed its query. That is how BERT resolves pronouns — and you just watched the mechanism.
Retrieval intuition, done. Now the paper compresses the entire mechanism into one formula — six symbols you will compute by hand, step by step, right now.
04 SCALED DOT-PRODUCT ATTENTION
One formula, five steps
The heart of the paper — and of every model since. Scores, scale, softmax, blend: step through a complete 3×3 computation with every number on screen, then break it yourself with the √dk slider to see why the word “scaled” is load-bearing.
THE √dk LAB — WHY SCALING IS NOT OPTIONAL
A dot product of two random dk-dim vectors has variance ≈ dk: as dimensions grow, raw scores grow with them, and softmax tips into a saturated near-one-hot — the gradient-killing regime from Lecture 03. Six candidate keys, real dot products, deterministic vectors:
WITHOUT SCALING · raw QKᵀ
WITH SCALING · QKᵀ/√d_k
WHY DOT PRODUCTS (AND NOT THE 2015 ADDITIVE ATTENTION)?
The attention of Lecture 07 scored pairs with a small feed-forward net (additive attention). The paper tried both: dot-product attention was just as good and far faster — it is two matrix multiplies and a softmax, optimized to the metal on GPU hardware. When two mechanisms tie on quality, the compute-friendly one wins. That is a lesson the rest of this lecture keeps teaching.
One attention distribution is one point of view — one pattern of “who matters to whom”. Language needs many at once. The paper’s answer: run eight attentions in parallel, in eight smaller subspaces.
05 MULTI-HEAD ATTENTION
Eight specialists, one sentence
Instead of one 512-dim attention, MultiHead runs 8 heads of 64 dims each — every head with its own WQ, WK, WV, so each learns a different relationship: coreference, adjacency, syntax, position. Then all 8 outputs are concatenated (8×64 = 512) and mixed by one final matrix WO. Below: eight hand-set head personalities drawn as arcs over the paper’s own sentence — click words, switch heads, then view all 8 at once.
ATTENTION WEIGHTS — TOP 4 (click any word to change the query)
THE MECHANICS — SAME TOTAL WORK
Each head works on dk = 512/8 = 64 dims, so 8 heads cost about the same as one full-width head — eight representation subspaces essentially free. The paper’s ablation agrees: one head costs ~0.9 BLEU, and 32 heads costs too — eight is a sweet spot, not a magic number. The personalities above are hand-set for teaching — but trained models really do grow previous-token heads, delimiter heads and coreference heads, and the paper’s Figure 3 shows “it” resolving exactly like head 1.
Notice what attention never learned: which word came first. Shuffle the sentence and every dot product survives — attention sees a bag of words. Something must inject the order, and the paper reaches for sine waves.
06 POSITIONAL ENCODING
Order, without recurrence
Self-attention is permutation-invariant: it treats its input as a set. “Dog bites man” and “man bites dog” would produce identical outputs, just reordered. The fix: add a position signature to every embedding — fixed sine and cosine waves whose wavelengths grow geometrically from 2π to ~60000 positions. Each position gets a unique code, nearby positions get similar codes, and the network can learn to read relative offsets straight off the dot products.
THE FULL MAP — 50 POSITIONS × 64 DIMENSIONS (hover any cell for its exact value)
ONE POSITION’S CODE — all 64 dims, fast → slow
RELATIVE POSITION IS READABLE — dot(PE(p), PE(p+Δ)) for Δ = 0…24
The similarity profile depends on the offset, not the absolute position — exactly what a model needs in order to learn “the word two slots back”.
THE PAPER’S FORMULAS — ADDED, ONCE, AT THE BOTTOM
PE(pos, 2i+1) = cos( pos / 100002i/dmodel )
Each dimension pair is a sine wave of wavelength 2π·100002i/d — a geometric progression from 2π to 10000·2π, so some dims tick every step while others barely move across a whole document. The team also tried learned position embeddings: results were nearly identical, and they chose sinusoids for one elegant reason — they extrapolate to sentences longer than any seen in training. (Modern LLMs went back to learned embeddings — with clever tricks to extend them; that story belongs to a later lecture.)
Attention and positions give the model knowledge of the sentence — but a stack needs plumbing: stability, depth, and a place for per-word computation. Enter the three-part block the whole field is built from.
07 THE ENCODER BLOCK
Residuals, LayerNorm, and the FFN
One encoder block = multi-head attention → Add&Norm → feed-forward → Add&Norm, stacked six times. The two Add&Norms are safety rails: the residual keeps the original signal alive no matter what the sublayer did, and LayerNorm keeps every vector well-behaved at any depth. The feed-forward network is where each position, having gathered its context, gets private processing — 512 → 2048 → ReLU → 512. Every transformation below runs on real numbers you can edit.
LAYER NORMALIZATION, LIVE — edit x, watch every number
Output: —
THE POSITION-WISE FFN + RESIDUAL RAILS — toy scale 4 → 8 (ReLU) → 4; the paper: 512 → 2048 → 512
HIDDEN LAYER (8 units) — negative units die at ReLU
PROJECTED BACK (4 dims)
Read the rails: the residual sum keeps a copy of x alive inside the output — a gradient highway (Lecture 03’s fix, generalized to depth) — and LayerNorm re-centers the sum so 6 stacked blocks stay numerically calm.
The encoder understands sentences. The decoder writes them — and writing means never being allowed to read your own future, while constantly consulting everything the encoder understood.
08 THE DECODER
Masked attention & the bridge
The decoder block has three sublayers: masked self-attention over its own partial output, cross-attention to the encoder, then the FFN. The causal mask enforces the arrow of time during training — position t may consult anything before it, nothing after — and cross-attention is the doorway through which the decoder reads the source sentence.
THE CAUSAL MASK — pick the row (which position is attending), toggle the mask
ROW t’S ATTENTION WEIGHTS (softmax of its scores)
CROSS-ATTENTION — WHERE DECODER MEETS ENCODER (step through the arrows)
Every component is now on the bench: embeddings, positions, eight heads, masks, cross-attention, rails. Time to reassemble the machine and watch it actually translate a sentence.
09 THE FULL WALKTHROUGH
Watch it translate
From “I am a student” to “Je suis étudiant” — and every stage below is the exact stage the real model runs: one pass over the encoder, then one full pass per generated token. Step through it; the state board tracks the source and the output as it grows.
The architecture is only half of the 2017 paper. The other half is quieter and just as important: the recipe that made six stacked blocks actually converge — and the record that proved the bet.
10 TRAINING THE ORIGINAL
The 2017 recipe — warmup & smoothing
Deep fresh stacks are fragile: with random weights, a big early learning rate destroys the network before it learns anything. The paper’s answer — the Noam schedule — warms the learning rate up linearly, then decays it with the square root of the step count. Alongside: dropout 0.1, label smoothing 0.1, Adam, beam search 4. Every number below is that schedule, computed live.
THE NOAM SCHEDULE — lrate = d⁻⁰·⁵model · min(step⁻⁰·⁵, step · warmup⁻¹·⁵)
Ride the curve: linear warmup through the fragile early steps, then a 1/√step decay for the long fine-tuning tail. Too-long warmup (40k) starves early training of learning rate.
LABEL SMOOTHING ε — the target stops being a spike
WHAT THE MODEL IS TRAINED TO SAY (5-way toy)
CROSS-ENTROPY IF THE MODEL PREDICTS [0.86, 0.05, 0.04, 0.03, 0.02]
The smoothed loss is numerically worse — and the paper says so: perplexity went up. But BLEU and accuracy went up. A model trained toward 0.9 instead of 1.0 stays humble, calibrates its confidence, and translates better. The metric you optimize is not always the metric you want.
New records in hand — 28.4 BLEU where the best 2017 systems had 26.3 — the paper makes its deeper, computational argument for why this architecture is the right shape. It is Table 1, and it is why everything after 2017 looks like this.
11 WHY SELF-ATTENTION WON
The honest scoreboard
The paper’s Table 1 compares three ways of mixing information across a sequence on three axes: compute per layer, how many steps must run in order, and how far information must travel between any two words. Drag n (sentence length) and d (model width) and watch the trade-offs move — including where attention’s advantage ends.
| Property | Self-Attention | Recurrent | Convolution |
|---|---|---|---|
| Complexity / layer | O(n²·d) | O(n·d²) | O(k·n·d²) |
| Sequential operations | O(1) — fully parallel | O(n) — a chain | O(1) |
| Max path length | O(1) — direct edge | O(n) | O(logk n) |
| Long-range learning | Native — one hop | Vanishing gradients | Needs many stacked layers |
The original was encoder–decoder. Each of its three descendants kept one half of the machine — and one of those halves, scaled up, became the large language model.
12 THE LEGACY
One architecture, three descendants
Strip the 2017 machine and two clean halves remain. Keep the encoder, and you get a model that reads bidirectionally — BERT. Keep the decoder, and you get a model that predicts the next token — GPT, and every LLM after it. Keep both, and you have the original translation shape — T5, BART, Whisper. Same blocks, same attention, three destinies.
Full bidirectional self-attention: every word sees every word — past and future. Trained by deleting words and guessing them from both sides (masked language modeling). No causal mask; next-token prediction is replaced by fill-in-the-blank.
Best at understanding: search, classification, entity recognition.
Only the causal-masked half: position t sees positions < t, always. Trained on the humblest task imaginable — predict the next token — over the whole internet. Scaling THIS half, with the recipe of §10, produced the modern LLM.
Best at generating: chat, code, reasoning — the ChatGPT lineage.
The original 2017 shape, kept intact: an encoder reads the input freely, a masked decoder writes the output, cross-attention bridges them. Translation generalized into “text-to-text”: every task phrased as input → output.
Best at transformation: translation, summarization, speech-to-text.
One-hot → BoW → TF-IDF → Word2Vec → RNN/LSTM (context, sequential) → Seq2Seq + attention (Lecture 07) → TRANSFORMER (you are here) → pretrained LLMs (next stop). Every arrow in that ladder was a workaround for a constraint the Transformer deletes.
Not attention itself — 2015 had that. The paper’s radical edit was subtraction: attention no longer assists a recurrent model, it replaces it. Recurrence, convolutions, the sequential relay — gone; what remains is attention, FFNs, and two safety rails per block.
Self-attention pays O(n²) in compute and memory, and generation stays one-token-at-a-time. The entire efficiency literature since — sparse attention, linear attention, FlashAttention, KV-caching — is the ongoing negotiation over this bill. The architecture won anyway.
13 CODE LAB
The whole lecture, in runnable code
Three tabs: attention from absolute scratch in NumPy — the same cat/sat/mat numbers as §04 — then the idiom you will actually use in PyTorch and Keras, with the mask and the Add&Norm rails in place.
14 MASTERY CHECK