⌗ LECTURE 09 · SELF-ATTENTION, BEFORE THE FORMULAS · intuition first, math later
Self-Attention, step by step
A descriptive tutorial. We start from a clearly stated problem, then build the real machine in five visual steps — Query/Key/Value vectors, the dot product, the softmax, the weighted sum, and the matrix form — every number of a 3-word, 2-dimension example computed live on screen. The formula appears only at the end, as the summary of what you already computed.
02 THE PROBLEM
An embedding cannot see its context
Every word enters the model as one fixed vector — the same bank vector whether the sentence is about rivers or money. The sentence around it carries the information that decides, but nothing reads it. State the problem precisely and the solution designs itself:
THE PROBLEM, DEMONSTRATED — same word, two sentences, two meanings
03 THE MACHINE IN FIVE STEPS
The plan
Five steps, each with its own section, each computed live on one running example — the 3-word phrase bank river money with 2-dimension word vectors, small enough to check every product by eye:
04 STEP 1 · QUERY, KEY, VALUE
Three projections of every word
For each word, the embedding x is multiplied by three small learned matrices: W_Q produces the Query (what the word looks for), W_K the Key (what it contains), W_V the Value (what it contributes to a blend). Click a word and read its three projections, component by component:
PICK A WORD — its three projections, arithmetic shown
The matrices are 2×2 because our embeddings are 2-dimensional — in Lecture 08 they are 64×512 and LEARNED: their numbers start random and are tuned by training. Here they are hand-set so every product is checkable.
05 STEP 2 · THE DOT PRODUCT
Matching a query against every key
How relevant is “river” to “bank”? Multiply the query of one word with the key of the other, component-wise, and add. One dot product per (word, word) pair. Step through the arithmetic:
06 STEP 3 · SCORES TO WEIGHTS
One attention budget per word
Raw scores are just numbers — any size, any sign. The softmax turns them into an attention budget: positive weights that sum to exactly 100%. Step through the three stages:
07 STEP 4 · THE WEIGHTED SUM
Blending the values
Each word’s Value vector — the actual information — is multiplied by its attention weight, and the results are added. The sum is the word’s new representation:
08 STEP 5 · ALL WORDS AT ONCE
The same steps as matrix products
Steps 1–4 ran for one word. Stack the word vectors into a matrix and the whole sentence computes in four matrix products — no loop, full parallelism. Step through the stages:
09 THE FORMULA, ASSEMBLED
Five steps → one line
Nothing new below. Every piece is a step you just computed — click each:
And “self”-attention: Q, K and V are all projections of the same X — the sentence consulting itself. (In Lecture 08’s translation machine the keys and values come from a different sentence: cross-attention.)
Luong Attention — The Mathematics →
Four scorers (dot, general, concat, location), global vs local attention with a predicted window, and input-feeding — computed on the same style of verified toy.
Attention — The Mathematics →
The original 2014 paper computed to the last decimal: Wₐs + Uₐh, the tanh bottle, softmax α, the convex blend — one full decoder step by hand.
Lecture 08 · Attention & Alignment →
The complete interactive lecture: the one-vector bottleneck, soft search, the RNNsearch machine and the famous alignment matrix — with the Urdu running example.
10 MASTERY CHECK