CAS logo CAS NLP LABInteractive Tutorial Series ← Series

⌗  ATTENTION IS ALL YOU NEED — Vaswani et al., NeurIPS 2017

Attention Is All You Need

Eight researchers made a bet that broke a decade of convention: delete the recurrence entirely. No loop, no hidden state passed through time — just every word looking at every other word at once, in parallel. Attention itself became the whole model. This lecture opens the machine block by block: every formula, every matrix, every design decision — computed live, nothing a picture.

TOKENS→ EMBEDDING→ + POSITION→ SELF-ATTENTION→ FEED FORWARD→ × 6→ CROSS-ATTN→ NEXT WORD
14Sectionsintuition → math → code
12+Live Demosevery matrix computed on the spot
8Attention Headseach one drawn over the sentence
8Quiz Checksthe mastery gate
THE ONE-SENTENCE SUMMARY

The Transformer replaces recurrence with self-attention: softmax(QKᵀ/√d_k)·V — every position queries every other, blends what it finds, and six stacked blocks of attention + feed-forward layers do everything an RNN did, but in parallel, with a direct connection between any two words. That one formula is the ancestor of BERT, GPT and every modern LLM.

THREE VIEWS OF ONE MACHINE

Figure 1, one-to-one — the paper’s own block diagram, redrawn exactly as printed in 2017 — and fully interactive: click any block to open its inner machinery, from the Figure-2 attention internals down to live LayerNorm and ReLU arithmetic. The explorer — the whole machine annotated, with a step-by-step TRACE. The data-flow map — the same machine as a paper-style component pipeline.

Every section below then takes one block apart: the retrieval intuition, the scaled dot-product math, eight heads drawn over a real sentence, the sin/cos position waves, the causal mask, the training recipe — all of it live.

START WITH WHY ↓

Hover to trace a block’s wires · click to open its inner machinery.

1

Q · K · V — the Query comes from the decoder’s own Add & Norm; Keys and Values arrive from the encoder’s output. That middle box is cross-attention.

2

N × — each stack is one block repeated 6 times. The encoder (left) reads the whole sentence; the decoder (right) writes left to right.

3

Side loops — every attention and FFN sublayer is wrapped by a residual branch into its Add & Norm: the gradient rails of §07.

4

⊕ — sinusoidal positional encodings, added once at the very bottom of each stack (§06). Attention alone is order-blind.

→

Click any block — its inner architecture opens: the attention kernel of Figure 2, the residual rail, the position-wise MLP — computed live.

Fig. 1 — “The Transformer — model architecture.” Redrawn one-to-one from Vaswani et al., Attention Is All You Need, NeurIPS 2017.

Seq2Seq squeezed a whole sentence into one vector; attention was bolted on from outside to rescue it. 2017’s bet: if attention does the real work anyway, delete the RNN it was rescuing.

02 THE SEQUENTIAL BOTTLENECK

Why not just a better RNN?

Three lectures of RNNs left two wounds that no gate could close. Memory: information from word 1 reaches word 9 only by surviving every hidden-state hand-off in between. Speed: step 9 cannot start until step 8 is done — a sentence is a queue, and GPUs idle. Self-attention removes both at once.

THE RNN WAYone word at a time
hidden-state steps: 0 / 12

THE ATTENTION WAYall words at once

h₁ → h₉RNN path: word 1’s influence rides 8 hand-offs of one fixed-size vector — each one can forget.
1 stepAttention path: word 1 attends directly to word 9. The gradient flows through a single edge.
n ×RNN training time grows linearly with sentence length — the steps cannot overlap on a GPU.
~1×Self-attention layers are giant matrix multiplies — the thing GPUs were born for, all steps at once.

The 2017 argument in one breath: attention had already been patched INTO seq2seq models (Lecture 08) to feed decoder words the right encoder words. The paper’s move was to ask what the RNN it was rescuing was still for — and to answer: nothing. Keep the attention, retire the recurrence.

So how does a word “look at” another word? The same way you find a book in a library — and the paper’s genius was turning that lookup into three learnable matrices.

03 QUERIES, KEYS & VALUES

Attention is a retrieval system

A soft version of dictionary lookup. Each word emits a query (what am I looking for?) and a key (what do I offer?); the dot product of query·key scores the match, a softmax turns scores into weights, and the word’s new representation is a weighted blend of everyone’s value. Below, every vector is hand-set and 4-dimensional so you can verify each dot product by eye.

THE SENTENCE — CLICK ANY WORD TO MAKE IT THE QUERY

SCORES & WEIGHTS — live softmax over all 12 keys

THE OUTPUT — a weighted blend of all values

WHERE EACH DIMENSION CAME FROM — one slice per word, sized by its weight

Hover a slice for the split. The blend IS the new representation of the query word.

FROM TOY VECTORS TO THE REAL THING

Q = X·WQ  ·  K = X·WK  ·  V = X·WV   (X = the n×512 embedding matrix; the W’s are learned)

In the real model, every word’s q, k and v are linear projections of its embedding — three matrices the model learns by gradient descent. Same sentence, two readings: switch to “it” → wide above and watch the same pronoun attend somewhere new, because its context changed its query. That is how BERT resolves pronouns — and you just watched the mechanism.

Retrieval intuition, done. Now the paper compresses the entire mechanism into one formula — six symbols you will compute by hand, step by step, right now.

04 SCALED DOT-PRODUCT ATTENTION

One formula, five steps

The heart of the paper — and of every model since. Scores, scale, softmax, blend: step through a complete 3×3 computation with every number on screen, then break it yourself with the √dk slider to see why the word “scaled” is load-bearing.

THE FORMULA THE WHOLE DECADE RUNS ON

Attention(Q, K, V) = softmax( QKT / √dk ) · V

THE √dk LAB — WHY SCALING IS NOT OPTIONAL

A dot product of two random dk-dim vectors has variance ≈ dk: as dimensions grow, raw scores grow with them, and softmax tips into a saturated near-one-hot — the gradient-killing regime from Lecture 03. Six candidate keys, real dot products, deterministic vectors:

KEY / QUERY DIMENSIONALITY dk4
1416642561024

WITHOUT SCALING · raw QKᵀ

WITH SCALING · QKᵀ/√d_k

WHY DOT PRODUCTS (AND NOT THE 2015 ADDITIVE ATTENTION)?

The attention of Lecture 07 scored pairs with a small feed-forward net (additive attention). The paper tried both: dot-product attention was just as good and far faster — it is two matrix multiplies and a softmax, optimized to the metal on GPU hardware. When two mechanisms tie on quality, the compute-friendly one wins. That is a lesson the rest of this lecture keeps teaching.

One attention distribution is one point of view — one pattern of “who matters to whom”. Language needs many at once. The paper’s answer: run eight attentions in parallel, in eight smaller subspaces.

05 MULTI-HEAD ATTENTION

Eight specialists, one sentence

Instead of one 512-dim attention, MultiHead runs 8 heads of 64 dims each — every head with its own WQ, WK, WV, so each learns a different relationship: coreference, adjacency, syntax, position. Then all 8 outputs are concatenated (8×64 = 512) and mixed by one final matrix WO. Below: eight hand-set head personalities drawn as arcs over the paper’s own sentence — click words, switch heads, then view all 8 at once.

ATTENTION WEIGHTS — TOP 4 (click any word to change the query)

THE MECHANICS — SAME TOTAL WORK

headi = Attention(XWiQ, XWiK, XWiV)  ·  MultiHead(X) = Concat(head1…head8)·WO

Each head works on dk = 512/8 = 64 dims, so 8 heads cost about the same as one full-width head — eight representation subspaces essentially free. The paper’s ablation agrees: one head costs ~0.9 BLEU, and 32 heads costs too — eight is a sweet spot, not a magic number. The personalities above are hand-set for teaching — but trained models really do grow previous-token heads, delimiter heads and coreference heads, and the paper’s Figure 3 shows “it” resolving exactly like head 1.

Notice what attention never learned: which word came first. Shuffle the sentence and every dot product survives — attention sees a bag of words. Something must inject the order, and the paper reaches for sine waves.

06 POSITIONAL ENCODING

Order, without recurrence

Self-attention is permutation-invariant: it treats its input as a set. “Dog bites man” and “man bites dog” would produce identical outputs, just reordered. The fix: add a position signature to every embedding — fixed sine and cosine waves whose wavelengths grow geometrically from 2π to ~60000 positions. Each position gets a unique code, nearby positions get similar codes, and the network can learn to read relative offsets straight off the dot products.

THE FULL MAP — 50 POSITIONS × 64 DIMENSIONS (hover any cell for its exact value)

ONE POSITION’S CODE — all 64 dims, fast → slow

SLIDE THE POSITION — pos 7highlighted row in the map

RELATIVE POSITION IS READABLE — dot(PE(p), PE(p+Δ)) for Δ = 0…24

The similarity profile depends on the offset, not the absolute position — exactly what a model needs in order to learn “the word two slots back”.

THE PAPER’S FORMULAS — ADDED, ONCE, AT THE BOTTOM

PE(pos, 2i)   = sin( pos / 100002i/dmodel )
PE(pos, 2i+1) = cos( pos / 100002i/dmodel )

Each dimension pair is a sine wave of wavelength 2π·100002i/d — a geometric progression from 2π to 10000·2π, so some dims tick every step while others barely move across a whole document. The team also tried learned position embeddings: results were nearly identical, and they chose sinusoids for one elegant reason — they extrapolate to sentences longer than any seen in training. (Modern LLMs went back to learned embeddings — with clever tricks to extend them; that story belongs to a later lecture.)

Attention and positions give the model knowledge of the sentence — but a stack needs plumbing: stability, depth, and a place for per-word computation. Enter the three-part block the whole field is built from.

07 THE ENCODER BLOCK

Residuals, LayerNorm, and the FFN

One encoder block = multi-head attention → Add&Norm → feed-forward → Add&Norm, stacked six times. The two Add&Norms are safety rails: the residual keeps the original signal alive no matter what the sublayer did, and LayerNorm keeps every vector well-behaved at any depth. The feed-forward network is where each position, having gathered its context, gets private processing — 512 → 2048 → ReLU → 512. Every transformation below runs on real numbers you can edit.

LAYER NORMALIZATION, LIVE — edit x, watch every number

μ = —mean of x — subtracted out
σ² = —variance — divided out (√(σ²+ε))
ε = 10⁻⁵numerical guard inside the square root

Output: —

LN(x) = γ ⊙ (x − μ) / √(σ² + ε) + β   — per token, per layer; γ, β learned

THE POSITION-WISE FFN + RESIDUAL RAILS — toy scale 4 → 8 (ReLU) → 4; the paper: 512 → 2048 → 512

fixed small weights — computed live

HIDDEN LAYER (8 units) — negative units die at ReLU

PROJECTED BACK (4 dims)

FFN(x) = max(0, xW₁+b₁)W₂+b₂   — the SAME MLP at every position, independently

Read the rails: the residual sum keeps a copy of x alive inside the output — a gradient highway (Lecture 03’s fix, generalized to depth) — and LayerNorm re-centers the sum so 6 stacked blocks stay numerically calm.

Why the FFN matters: attention moves information between positions but transforms little per position; the FFN is where the heavy per-token computation happens — and it holds about two-thirds of the block’s parameters. Post-2017 interpretability work even localizes factual knowledge to these FFN layers: they behave like the model’s key-value memory.

The encoder understands sentences. The decoder writes them — and writing means never being allowed to read your own future, while constantly consulting everything the encoder understood.

08 THE DECODER

Masked attention & the bridge

The decoder block has three sublayers: masked self-attention over its own partial output, cross-attention to the encoder, then the FFN. The causal mask enforces the arrow of time during training — position t may consult anything before it, nothing after — and cross-attention is the doorway through which the decoder reads the source sentence.

THE CAUSAL MASK — pick the row (which position is attending), toggle the mask

ATTENDING POSITION tsat
visible past / self masked future (−∞) FUTURE LEAK — cheating! not generated yet

ROW t’S ATTENTION WEIGHTS (softmax of its scores)

CROSS-ATTENTION — WHERE DECODER MEETS ENCODER (step through the arrows)

The training trick hiding under the mask: at inference the decoder generates one token per full pass — inherently sequential. But at TRAINING time the whole target sentence is fed in at once, and the mask makes each position blind to its future. Parallel training, sequential generation: one mask delivers both.

Every component is now on the bench: embeddings, positions, eight heads, masks, cross-attention, rails. Time to reassemble the machine and watch it actually translate a sentence.

09 THE FULL WALKTHROUGH

Watch it translate

From “I am a student” to “Je suis étudiant” — and every stage below is the exact stage the real model runs: one pass over the encoder, then one full pass per generated token. Step through it; the state board tracks the source and the output as it grows.

cross-attention weights & next-word probabilities are illustrative — the stages and their order are exact

Count the cost: the encoder ran once. Every output token then costs one full decoder pass — so generating n tokens means n sequential trips through 6 blocks. That is the real price of autoregression, and it is why inference speed (KV-caching, speculative decoding) became its own research field the moment this architecture won.

The architecture is only half of the 2017 paper. The other half is quieter and just as important: the recipe that made six stacked blocks actually converge — and the record that proved the bet.

10 TRAINING THE ORIGINAL

The 2017 recipe — warmup & smoothing

Deep fresh stacks are fragile: with random weights, a big early learning rate destroys the network before it learns anything. The paper’s answer — the Noam schedule — warms the learning rate up linearly, then decays it with the square root of the step count. Alongside: dropout 0.1, label smoothing 0.1, Adam, beam search 4. Every number below is that schedule, computed live.

THE NOAM SCHEDULE — lrate = d⁻⁰·⁵model · min(step⁻⁰·⁵, step · warmup⁻¹·⁵)

TRAINING STEP4,000
—learning rate at this step
—the warmup peak

Ride the curve: linear warmup through the fragile early steps, then a 1/√step decay for the long fine-tuning tail. Too-long warmup (40k) starves early training of learning rate.

LABEL SMOOTHING ε — the target stops being a spike

SMOOTHING ε0.10

WHAT THE MODEL IS TRAINED TO SAY (5-way toy)

CROSS-ENTROPY IF THE MODEL PREDICTS [0.86, 0.05, 0.04, 0.03, 0.02]

—vs. a one-hot target
—vs. the smoothed target

The smoothed loss is numerically worse — and the paper says so: perplexity went up. But BLEU and accuracy went up. A model trained toward 0.9 instead of 1.0 stays humble, calibrates its confidence, and translates better. The metric you optimize is not always the metric you want.

New records in hand — 28.4 BLEU where the best 2017 systems had 26.3 — the paper makes its deeper, computational argument for why this architecture is the right shape. It is Table 1, and it is why everything after 2017 looks like this.

11 WHY SELF-ATTENTION WON

The honest scoreboard

The paper’s Table 1 compares three ways of mixing information across a sequence on three axes: compute per layer, how many steps must run in order, and how far information must travel between any two words. Drag n (sentence length) and d (model width) and watch the trade-offs move — including where attention’s advantage ends.

SEQUENCE LENGTH n128
REPRESENTATION DIM d512

PropertySelf-AttentionRecurrentConvolution
Complexity / layerO(n²·d)O(n·d²)O(k·n·d²)
Sequential operationsO(1) — fully parallelO(n) — a chainO(1)
Max path lengthO(1) — direct edgeO(n)O(logk n)
Long-range learningNative — one hopVanishing gradientsNeeds many stacked layers

The original was encoder–decoder. Each of its three descendants kept one half of the machine — and one of those halves, scaled up, became the large language model.

12 THE LEGACY

One architecture, three descendants

Strip the 2017 machine and two clean halves remain. Keep the encoder, and you get a model that reads bidirectionally — BERT. Keep the decoder, and you get a model that predicts the next token — GPT, and every LLM after it. Keep both, and you have the original translation shape — T5, BART, Whisper. Same blocks, same attention, three destinies.

ENCODER-ONLY · BERT, 2018

Full bidirectional self-attention: every word sees every word — past and future. Trained by deleting words and guessing them from both sides (masked language modeling). No causal mask; next-token prediction is replaced by fill-in-the-blank.

Best at understanding: search, classification, entity recognition.

DECODER-ONLY · GPT, 2018 → LLMs

Only the causal-masked half: position t sees positions < t, always. Trained on the humblest task imaginable — predict the next token — over the whole internet. Scaling THIS half, with the recipe of §10, produced the modern LLM.

Best at generating: chat, code, reasoning — the ChatGPT lineage.

ENCODER–DECODER · T5, BART, WHISPER

The original 2017 shape, kept intact: an encoder reads the input freely, a masked decoder writes the output, cross-attention bridges them. Translation generalized into “text-to-text”: every task phrased as input → output.

Best at transformation: translation, summarization, speech-to-text.

Where you are on the map

One-hot → BoW → TF-IDF → Word2Vec → RNN/LSTM (context, sequential) → Seq2Seq + attention (Lecture 07) → TRANSFORMER (you are here) → pretrained LLMs (next stop). Every arrow in that ladder was a workaround for a constraint the Transformer deletes.

What actually changed in 2017

Not attention itself — 2015 had that. The paper’s radical edit was subtraction: attention no longer assists a recurrent model, it replaces it. Recurrence, convolutions, the sequential relay — gone; what remains is attention, FFNs, and two safety rails per block.

The honest bill

Self-attention pays O(n²) in compute and memory, and generation stays one-token-at-a-time. The entire efficiency literature since — sparse attention, linear attention, FlashAttention, KV-caching — is the ongoing negotiation over this bill. The architecture won anyway.

13 CODE LAB

The whole lecture, in runnable code

Three tabs: attention from absolute scratch in NumPy — the same cat/sat/mat numbers as §04 — then the idiom you will actually use in PyTorch and Keras, with the mask and the Add&Norm rails in place.

ATTENTION · THREE WAYS

14 MASTERY CHECK

Eight questions — one per component.

YOUR SCORE

0 / 8