CAS logo CAS NLP LABInteractive Tutorial Series ← Series

⌗ LECTURE 09 · SELF-ATTENTION, BEFORE THE FORMULAS · intuition first, math later

Self-Attention, step by step

A descriptive tutorial. We start from a clearly stated problem, then build the real machine in five visual steps — Query/Key/Value vectors, the dot product, the softmax, the weighted sum, and the matrix form — every number of a 3-word, 2-dimension example computed live on screen. The formula appears only at the end, as the summary of what you already computed.

1 · Q K V→ 2 · DOT PRODUCT→ 3 · WEIGHTS→ 4 · BLEND→ 5 · MATRIX FORM
THE ONE-SENTENCE SUMMARY

A word alone is ambiguous. Self-attention lets every word ask a question, compare it against everyone’s labels, and rebuild itself as a blend of the most useful company — all words at once, in rounds. That is the entire mechanism. Everything else on this page is this story, drawn.

START WITH THE PROBLEM ↓ SEE THE PLAN ↓

02 THE PROBLEM

An embedding cannot see its context

Every word enters the model as one fixed vector — the same bank vector whether the sentence is about rivers or money. The sentence around it carries the information that decides, but nothing reads it. State the problem precisely and the solution designs itself:

PROBLEM STATEMENT

Given a sentence of words, compute for every word a new representation that depends on the words around it — automatically, in parallel, and in a way a network can learn. The new representation should mix in information from whichever words are relevant, by amounts the network itself decides.

THE PROBLEM, DEMONSTRATED — same word, two sentences, two meanings

03 THE MACHINE IN FIVE STEPS

The plan

Five steps, each with its own section, each computed live on one running example — the 3-word phrase bank river money with 2-dimension word vectors, small enough to check every product by eye:

The running example: embeddings x(bank) = [0.50, 0.10], x(river) = [0.90, 0.20], x(money) = [0.20, 0.80] — and three small 2×2 weight matrices we will meet in Step 1. Nothing else. Every number on this page follows from these.

04 STEP 1 · QUERY, KEY, VALUE

Three projections of every word

For each word, the embedding x is multiplied by three small learned matrices: W_Q produces the Query (what the word looks for), W_K the Key (what it contains), W_V the Value (what it contributes to a blend). Click a word and read its three projections, component by component:

PICK A WORD — its three projections, arithmetic shown

The matrices are 2×2 because our embeddings are 2-dimensional — in Lecture 08 they are 64×512 and LEARNED: their numbers start random and are tuned by training. Here they are hand-set so every product is checkable.

05 STEP 2 · THE DOT PRODUCT

Matching a query against every key

How relevant is “river” to “bank”? Multiply the query of one word with the key of the other, component-wise, and add. One dot product per (word, word) pair. Step through the arithmetic:

query: q(bank) = [0.50, 0.60] — bank is looking for its company

06 STEP 3 · SCORES TO WEIGHTS

One attention budget per word

Raw scores are just numbers — any size, any sign. The softmax turns them into an attention budget: positive weights that sum to exactly 100%. Step through the three stages:

07 STEP 4 · THE WEIGHTED SUM

Blending the values

Each word’s Value vector — the actual information — is multiplied by its attention weight, and the results are added. The sum is the word’s new representation:

What was achieved: bank’s new vector [0.50, 0.44] is no longer the old [0.50, 0.10]. It has absorbed its company — and the amounts absorbed were decided by the machine itself, from the words’ own content. Every word of the sentence gets the same treatment.

08 STEP 5 · ALL WORDS AT ONCE

The same steps as matrix products

Steps 1–4 ran for one word. Stack the word vectors into a matrix and the whole sentence computes in four matrix products — no loop, full parallelism. Step through the stages:

09 THE FORMULA, ASSEMBLED

Five steps → one line

Nothing new below. Every piece is a step you just computed — click each:

Attention(Q, K, V) = softmax( QKT / √dₖ ) · V

And “self”-attention: Q, K and V are all projections of the same X — the sentence consulting itself. (In Lecture 08’s translation machine the keys and values come from a different sentence: cross-attention.)

Continue the attention path — next up

Luong Attention — The Mathematics →
Four scorers (dot, general, concat, location), global vs local attention with a predicted window, and input-feeding — computed on the same style of verified toy.

Then the Bahdanau deep dive

Attention — The Mathematics →
The original 2014 paper computed to the last decimal: Wₐs + Uₐh, the tanh bottle, softmax α, the convex blend — one full decoder step by hand.

And the full lecture

Lecture 08 · Attention & Alignment →
The complete interactive lecture: the one-vector bottleneck, soft search, the RNNsearch machine and the famous alignment matrix — with the Urdu running example.

10 MASTERY CHECK

Eight questions — on the mechanics.

YOUR SCORE

0 / 8