⌗ LECTURE 10 · MULTI-HEAD ATTENTION · every arrow carries its shape
Words in, same shape out
The Transformer’s most important promise hides in the shapes: a sentence of n tokens × 512 numbers enters multi-head attention, gets projected, split into 8 heads of 64, scored, blended, concatenated, projected again — and comes out (n × 512), identical to what entered. This page follows every arrow of that journey with its shape attached — words → embeddings → W_Q, W_K, W_V → Q, K, V → heads → scores → concat → output — computed live on a checkable toy.
02 THE CAST
Every dimension, named
Shapes only confuse when the symbols are anonymous. Here is the full cast — pick the number of heads and the sentence length, and watch every derived shape update. The paper’s choice is h = 8:
03 Q · K · V, IN PICTURES
What each word looks for, offers, holds
Before more block diagrams: the classic picture. Every token’s embedding is multiplied by three learned matrices, producing three vectors with three distinct jobs. Pick a word, then click QUERY, KEY or VALUE:
THE FORMULA, AS BOXES — then one token, sat, carried through it by hand (head 0 of our toy; per head in the paper):
03 FIGURE 2, WITH SHAPES
The paper’s block diagram, annotated
The right panel of the paper’s Figure 2 — redrawn with the dimension of every wire. Hover any block for its job:
Left panel of the same figure = what happens INSIDE each Scaled Dot-Product block: matmul(Q,Kᵀ) → ÷√dₖ → mask → softmax → matmul(·V) — the Lecture 08 machine.
05 THE FULL FLOW
Words → output, seven stages
The complete journey on the toy: 4 tokens (“the cat sat here”), d_model = 8, h = 2 heads, d_k = 4. Every matrix is real — computed live, every shape labeled. Follow the numbers:
06 THE SPLIT
One matrix, eight specialists
The split is a reshape, not a computation — but who owns which columns decides everything. Each head’s slice produces its own attention personality:
07 WHY HEADS — AND WHY IT SCALES
Subspaces, not brute force
The paper’s motivation, verbatim: multi-head attention lets the model attend to information from different representation subspaces at different positions jointly — a single averaged head inhibits this. And the surprise: it costs almost nothing extra.
Head 0’s 64 dims might learn “who does what to whom”; head 3’s, “where in the sentence”; head 6’s, coreference. One full-width head must average all of those into a single pattern per token.
8 heads × n²·64 per head ≈ n²·512 for one big head — the paper: total computational cost is similar to single-head attention with full dimensionality. You buy 8 views for the price of 1.
The paper tried h = 1: worse. h = 32: worse. h = 8 with d_k = 64 was the sweet spot — and the design every later model inherited.
08 THE CONTRACT
Same shape in, same shape out
Trace every shape from input to output — the final line is why Transformers can stack:
09 CODE LAB
Every shape, confirmed in PyTorch
Print the shape after every arrow of the flow — then run the same pipeline through PyTorch’s built-in module:
010 MASTERY CHECK