CAS logo CAS NLP LABInteractive Tutorial Series ← Series

⌗ LECTURE 10 · MULTI-HEAD ATTENTION · every arrow carries its shape

Words in, same shape out

The Transformer’s most important promise hides in the shapes: a sentence of n tokens × 512 numbers enters multi-head attention, gets projected, split into 8 heads of 64, scored, blended, concatenated, projected again — and comes out (n × 512), identical to what entered. This page follows every arrow of that journey with its shape attached — words → embeddings → W_Q, W_K, W_V → Q, K, V → heads → scores → concat → output — computed live on a checkable toy.

WORDS→ X (n, 512)→ W_Q · W_K · W_V→ Q, K, V (n, 512)→ SPLIT (8, n, 64)→ ATTN × 8→ CONCAT→ Wₒ→ OUT (n, 512)
THE ONE-SENTENCE SUMMARY

Multi-head attention runs 8 smaller attentions in parallel — each on a learned 64-dim slice of the words — then concatenates their findings and mixes them with Wₒ. The paper: MultiHead(Q,K,V) = Concat(head₁…head₈)·Wₒ. Shapes: (n×512) in, (n×512) out — always.

MEET THE DIMENSIONS ↓ JUMP TO THE FLOW ↓

02 THE CAST

Every dimension, named

Shapes only confuse when the symbols are anonymous. Here is the full cast — pick the number of heads and the sentence length, and watch every derived shape update. The paper’s choice is h = 8:

The toy on this page: to keep every matrix visible, the flow sections (§05–§06) shrink everything — 4 tokens, d_model = 8, h = 2, d_k = 4. The shapes scale by the same rule; every number stays checkable.

03 Q · K · V, IN PICTURES

What each word looks for, offers, holds

Before more block diagrams: the classic picture. Every token’s embedding is multiplied by three learned matrices, producing three vectors with three distinct jobs. Pick a word, then click QUERY, KEY or VALUE:

THE FORMULA, AS BOXES — then one token, sat, carried through it by hand (head 0 of our toy; per head in the paper):

Same token, second opinion: head 1 asks a different question about the very same sat — its budget is [0.20, 0.12, 0.51, 0.18]: mostly sat itself, via the event dims. Different subspace, different answer — that is why the paper runs 8 heads instead of 1.

03 FIGURE 2, WITH SHAPES

The paper’s block diagram, annotated

The right panel of the paper’s Figure 2 — redrawn with the dimension of every wire. Hover any block for its job:

Left panel of the same figure = what happens INSIDE each Scaled Dot-Product block: matmul(Q,Kᵀ) → ÷√dₖ → mask → softmax → matmul(·V) — the Lecture 08 machine.

05 THE FULL FLOW

Words → output, seven stages

The complete journey on the toy: 4 tokens (“the cat sat here”), d_model = 8, h = 2 heads, d_k = 4. Every matrix is real — computed live, every shape labeled. Follow the numbers:

06 THE SPLIT

One matrix, eight specialists

The split is a reshape, not a computation — but who owns which columns decides everything. Each head’s slice produces its own attention personality:

07 WHY HEADS — AND WHY IT SCALES

Subspaces, not brute force

The paper’s motivation, verbatim: multi-head attention lets the model attend to information from different representation subspaces at different positions jointly — a single averaged head inhibits this. And the surprise: it costs almost nothing extra.

Different subspaces

Head 0’s 64 dims might learn “who does what to whom”; head 3’s, “where in the sentence”; head 6’s, coreference. One full-width head must average all of those into a single pattern per token.

The cost argument

8 heads × n²·64 per head ≈ n²·512 for one big head — the paper: total computational cost is similar to single-head attention with full dimensionality. You buy 8 views for the price of 1.

The ablation

The paper tried h = 1: worse. h = 32: worse. h = 8 with d_k = 64 was the sweet spot — and the design every later model inherited.

08 THE CONTRACT

Same shape in, same shape out

Trace every shape from input to output — the final line is why Transformers can stack:

What happens next (Lecture 11): this (n × 512) output is added back to the input by a residual connection — x + MultiHead(LN(x)) — then normalized and fed to the FFN. The same-shape contract is what makes that addition legal.

09 CODE LAB

Every shape, confirmed in PyTorch

Print the shape after every arrow of the flow — then run the same pipeline through PyTorch’s built-in module:

MULTI-HEAD · TWO WAYS

010 MASTERY CHECK

Eight questions — on shapes.

YOUR SCORE

0 / 8