◈ DEEP DIVE — ATTENTION: THE MATHEMATICS · formula first, every number machine-verified
Three equations are the whole idea
Attention is not a metaphor — it is three formulas. This page presents each one formula first, decodes every identifier, reads it in plain English, justifies every operation, expands every sum, computes every number by hand, and only then offers intuition. Work top to bottom and you will be able to explain each symbol, each shape, and each why — unaided.
1 · SCORE — how relevant is word j?
\[ e_{ij} = v_a^T \tanh\!\left( W_a s_{i-1} + U_a h_j \right) \]2 · WEIGHTS — scores to a budget
\[ \alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x}\exp(e_{ik})} \]3 · CONTEXT — the weighted read
\[ c_i = \sum_{j=1}^{T_x} \alpha_{ij}\, h_j \]THE CHAIN AT A GLANCE — one decoder step, six stages
02 THE INDEXING MAP
Before any formula: i, j, k, Tx
Attention notation confuses almost everyone for exactly one reason: the indices. Four of them appear constantly, and each lives on a different side of the machine. This card is the permanent reference — the same four colors return beside every formula below.
decoder / output position — the word being generated. Fixed while the softmax runs.
encoder / input position — a specific source word being attended to.
temporary summation index — visits every encoder position 1…Tx inside a sum, then vanishes.
total number of input positions — the source sentence’s length in tokens.
Tx ON A REAL SENTENCE — “I love machine learning”
h1=Ih2=loveh3=machineh4=learning
Tx = 4. On this page’s running example — The European Economic Area was established in 1994 — Tx = 8, and i counts the Urdu words being produced (i = 3 is the step that writes علاقہ).
03 PART 1 — ENCODER HIDDEN STATES
The memories: hj
What does every symbol mean?
| \(x_j\) | the input token at position \(j\) — a word, or its embedding | token |
| \(j\) | encoder position — which source word | 1…Tx |
| \(h_{j-1}\) | the previous encoder state — the memory built so far | \(\mathbb{R}^{d_h}\) |
| \(h_j\) | the annotation: everything the encoder knows about position \(j\) and its context | \(\mathbb{R}^{d_h}\) |
| Encoder | the recurrent function (a GRU in the paper) that folds \(x_j\) into the running memory | function |
The paper actually runs two encoders — forward \(\overrightarrow{h_j}\) and backward \(\overleftarrow{h_j}\) — and concatenates: \(h_j = [\,\overrightarrow{h_j} ; \overleftarrow{h_j}\,]\), so each annotation knows its left and right context (1000 + 1000 = 2000 dims). Our toy uses 4.
Read it in English
“Walk the source sentence word by word; at each position \(j\), fold the new word \(x_j\) into the memory of everything before it. The result \(h_j\) is that word’s annotation — the note the decoder may later consult.”
Why does this step exist?
Because the decoder will need to query individual words, not the sentence as a blur. One state per position is what makes per-word search possible at all. (In Lecture 07’s seq2seq only \(h_{T_x}\) survived — that deletion of \(h_1 \dots h_{T_x-1}\) was the bottleneck.)
Numbers — the four annotations of our running example
Each is a real \(\mathbb{R}^4\) vector from the parent lecture’s toy — these are the actual \(h_j\) values every later formula will consume.
Intuition — only now
Think of a meticulous reader with a stack of index cards, one per word: card \(j\) records not just the word but what was happening around it. The encoder is that reader; \(h_j\) is card \(j\).
12 annotations \(h_1 \dots h_{12}\) — one per position — and \(T_x = 12\). The count of \(h\)’s always equals \(T_x\), regardless of sentence content.
04 PART 2 — THE DECODER QUERY
What the decoder asks with: si−1
What does every symbol mean?
| \(s\) | the decoder state — its working memory: what has been generated and what it implies | \(\mathbb{R}^{d_s}\) |
| \(i\) | decoder position — the output word about to be produced | 1…Ty |
| \(i-1\) | the step before — attention must score before word \(i\) exists, so it uses the freshest state available: the one after word \(i-1\) | index |
Read it in English
“Before writing output word \(i\), the decoder asks: based on everything I have written so far, which source word is useful now? The state \(s_{i-1}\) is that question, in vector form.”
Why the previous state?
Because \(y_i\) — the word being chosen — does not exist yet. Using \(s_i\) would mean attending after deciding, which is circular. \(s_{i-1}\) is the honest, causal query: everything known strictly before the decision. (Compare the causal mask of Lecture 11: the same arrow of time, enforced by indexing.)
Numbers
Our running step: the decoder has written یورپی معاشی (i = 3 next), and its state is this exact vector. Dimension 2 is loud (1.8) — the query’s dominant need.
Intuition — only now
An open-book exam: \(s_{i-1}\) is your half-written answer plus the feeling of what’s missing. The \(h_j\) are the textbook’s highlighted passages. Part 3 is the act of matching question to passages.
Because \(s_i\) is computed using \(c_i\), which is computed from the attention scores — the scores would depend on a state that depends on them. Training would chase a circle; and at generation time the future literally does not exist yet.
05 PART 3 — THE ALIGNMENT SCORE
Equation (5): eij
The heart of the mechanism — one learnable comparison between a query and every memory. Formula first; then every symbol; then every operation’s reason to exist.
What does every symbol mean?
| \(e_{ij}\) | the raw compatibility / alignment score: how relevant is encoder position \(j\) while producing output position \(i\). Any real number — not a probability | \(\mathbb{R}\) — scalar |
| \(i\) | decoder position (the output word being produced) | fixed |
| \(j\) | encoder position (the source word being scored) | 1…Tx |
| \(s_{i-1}\) | the decoder’s query state (Part 2) | \(\mathbb{R}^{d_s}\) |
| \(h_j\) | encoder position \(j\)’s annotation (Part 1) | \(\mathbb{R}^{d_h}\) |
| \(W_a\) | query lens — transforms what the decoder wants into the comparison space | \(\mathbb{R}^{d_a \times d_s}\) |
| \(U_a\) | memory lens — transforms what each source word offers into the same space | \(\mathbb{R}^{d_a \times d_h}\) |
| \(v_a\) | the judge — weights which comparison dimensions matter, collapsing to one number | \(\mathbb{R}^{d_a}\) |
| \(v_a^T\) | transpose — flips the column vector to a row so the product \(v_a^T x = \sum_d v_{a,d}\, x_d\) is a scalar (a dot product) | \(\mathbb{R}^{1 \times d_a}\) |
| \(\tanh\) | squashes each comparison dimension into (−1, 1) and adds nonlinearity | \(\mathbb{R}^{d_a} \to \mathbb{R}^{d_a}\) |
| \(+\) | vector addition — legal here because both sides land in \(\mathbb{R}^{d_a}\) (see the shape chain below) | \(\mathbb{R}^{d_a}\) |
The dimensional chain — why every operation is shape-legal
Read it in English
“Transform the decoder’s current requirement into the comparison space; transform source word \(j\)’s representation into that same space; add them; squash; and let the judge read out one number — how compatible this memory is with this moment of writing.”
Why does each operation exist?
Why \(W_a s_{i-1}\)? Raw \(s\) speaks “decoder language”; raw \(h\) speaks “encoder language”. \(W_a\) is the learned lens that expresses the need in comparison features. Computed once per step — it does not depend on \(j\).
Why \(U_a h_j\)? The matching lens for the offer side, computed per word. Two lenses instead of one direct comparison: the model must learn which features make a good match — maybe “is this a noun?”, maybe “am I a verb needing a subject?” — and lenses are where that knowledge lives.
Why add? Adding is the cheapest way to test agreement: dimensions where need and offer point the same way grow; opposing ones cancel. (Bahdanau’s “additive” attention, vs Luong’s dot-product \(s^T h\) — Lecture 11 keeps the softmax and the blend but swaps this scorer for a dot product.)
Why tanh? Two jobs: bound — raw sums are unbounded, and numbers of wildly different scales cannot be compared fairly, so every comparison dimension is squashed into (−1, 1); and bend — without a nonlinearity the whole scorer collapses to one linear map, far too weak to express “this kind of word matches that kind of need”.
Why \(v_a^T\)? Three squashed numbers are still three numbers, and the next stage (softmax) needs ONE score per word. \(v_a\) is the learned weighting of comparison dimensions — the judge’s scoring rubric.
Expand it — the full arithmetic for ONE word (Area, the eventual winner)
Numbers — all four words, computed live
e = [1.2891 (The), 1.7152 (European), 1.3384 (Economic), 1.9510 (Area)] — raw scores, no scale yet. Check one row with a pencil; the tables are computed live from the same constants.
Intuition — only now
A match needs two prepared sides: one shopping list (the query, written once per step) held against every shelf (the memories, one description each). \(W_a\) and \(U_a\) taught the shopper which features matter; tanh keeps the shouting civil; \(v_a\) is the shopper’s scoring rubric.
\(W_a s_{i-1}\) has no \(j\) in it — it is the query side, identical for every word. Only \(U_a h_j\) varies with \(j\). Real implementations exploit this: compute \(U_a H\) for the whole sentence as one matrix product, then reuse the result for every \(i\).
06 PART 4 — THE ATTENTION SOFTMAX
Equation (6): αij
Four raw scores in; a budget of attention out. This is the page’s most detailed decode.
What does every symbol mean?
| \(\alpha_{ij}\) | the attention weight: when generating output word \(i\), how much attention goes to input word \(j\). In (0, 1); the row sums to 1 | \((0,1)\) |
| \(i\) | decoder position — fixed throughout this formula | fixed |
| \(j\) | the specific encoder position whose share we compute | 1…Tx |
| \(e_{ij}\) | the raw score from Part 3. \(e_{ij} \neq \alpha_{ij}\) — raw score vs normalized weight: | \(\mathbb{R}\) |
| \(\exp\) | the exponential, \(\exp(x) = e^x \approx 2.718^x\) — always positive, order-preserving, amplifying | \(\mathbb{R} \to \mathbb{R}^+\) |
| numerator | \(\exp(e_{ij})\) — word \(j\)’s exponentiated claim on the attention budget | \(\mathbb{R}^+\) |
| \(\sum\) | the summation — add one term per source position (expanded below) | operator |
| \(k\) | temporary index — exists only inside the sum: keep \(i\) fixed and visit every encoder position \(k = 1, 2, \dots, T_x\) | 1…Tx |
| \(e_{ik}\) | the raw score of encoder position \(k\) — the same row of Part-3 scores, read at position \(k\) | \(\mathbb{R}\) |
| denominator | \(\sum_k \exp(e_{ik})\) — the total exponentiated mass of the whole source; the normalizer | \(\mathbb{R}^+\) |
| \(T_x\) | total input positions (see the Indexing Map: “I love machine learning” → \(T_x = 4\); our sentence → \(T_x = 8\)) | integer |
Expand the summation — the denominator, written out
Read it in English
“Take word \(j\)’s alignment score, exponentiate it, and divide by the exponentiated scores of all input words added together. The result is word \(j\)’s share of a fixed attention budget — and the whole budget is spent:
Why does each operation exist?
Why \(\exp\)? Three jobs in one function. (1) Positivity — attention cannot be negative; \(e^x > 0\) for every real \(x\). (2) Separation — exponentiation widens gaps: a small score edge becomes a large weight edge (winner-takes-more). (3) Differentiability — the whole map stays smooth, so the scores \(e\) can be trained by gradient descent. (The −max shift in Lecture 11’s code exists only to keep \(e^x\) from overflowing — mathematically it changes nothing, since adding the same constant to every score leaves softmax unchanged.)
Why divide by the denominator? Exponentiated scores are positive but still unbounded — \(2.72, 7.39, 20.09\) are not shares of anything. Dividing each by their total is what converts mass into proportions of a budget: non-negative, summing to exactly 1. That single division is the difference between “scores” and “attention”.
Numbers — a complete 3-word example, step by step
Step 0 — the raw scores. \(e_{i1}=1,\ e_{i2}=2,\ e_{i3}=3\)
Step 1 — exponentiate. \(\exp(1)=2.718,\quad \exp(2)=7.389,\quad \exp(3)=20.086\). The gap doubled (1→3) but the mass nearly octupled (2.72→20.09) — that is separation at work.
Step 2 — the denominator. \(2.718+7.389+20.086 = 30.193\)
Step 3 — each weight. \(\alpha_{i1}=\frac{2.718}{30.193}\approx 0.090,\quad \alpha_{i2}=\frac{7.389}{30.193}\approx 0.245,\quad \alpha_{i3}=\frac{20.086}{30.193}\approx 0.665\)
Step 4 — verify the budget. \(0.090+0.245+0.665 = 1.000\) ✓
Step 5 — interpret. Input position 3 receives ≈ 66.5% of the decoder’s attention at this output step — position 1, 9%.
And with our real four scores: \(\text{softmax}([1.2891, 1.7152, 1.3384, 1.9510]) = [\,0.1812,\ 0.2774,\ 0.1903,\ \mathbf{0.3512}\,]\) — Area, with the top score, takes 35.1% of the budget.
Feel it — drag a score, watch the budget re-split
Intuition — only now
An auction with a fixed pot: every word bids \(\exp(\text{score})\), the house collects all bids, and each word’s payout \(\alpha\) is its bid divided by the pot. Raise one bid and every other payout shrinks — the pot never grows.
\(\alpha_{i1}\) increases — its numerator grows while it also grows the denominator. Every other \(\alpha_{ij}\) must decrease (same own-numerator, bigger denominator): the budget of 1 is simply re-split. Try it on the slider above.
07 PART 5 — THE CONTEXT VECTOR
Equation (7): ci
What does every symbol mean?
| \(c_i\) | the context vector for output position \(i\) — the weighted read of the source, private to this step | \(\mathbb{R}^{d_h}\) |
| \(\alpha_{ij}\) | how much attention position \(j\) receives (Part 4) | \((0,1)\) |
| \(h_j\) | the information stored at encoder position \(j\) (Part 1) | \(\mathbb{R}^{d_h}\) |
| \(\alpha_{ij} h_j\) | position \(j\)’s weighted contribution — the vector scaled by its share of attention | \(\mathbb{R}^{d_h}\) |
| \(\sum\) | combine the weighted contributions of all source positions — \(j\) visits 1…\(T_x\) | operator |
Expand the summation
Read it in English
“Scale every source annotation by how much attention it earned, then add them all up. The context vector is not one selected word — it is a weighted mixture of all encoder representations, re-mixed for every output position \(i\).”
Why a weighted sum — and not just take the best word?
(1) Hedging: often two words matter at once (in Lecture 08, one Urdu word leaned on two English words simultaneously) — a blend keeps both. (2) Learnability: “pick the argmax” is a cliff — a discrete choice with no gradient, so nothing would tell the scorer how to improve. Weights are smooth: every \(\alpha\) receives a gradient. (3) Safety: \(\alpha \ge 0\) and \(\sum_j \alpha_{ij} = 1\) make \(c_i\) a convex combination — a point inside the hull of real annotations. The read is always a plausible mixture of actual word information; it can never point somewhere the sentence never went.
Numbers — a minimal 2-D example, component by component
Numbers — our real step, all four annotations
c = [0.7335, 0.6744, 0.4848, 0.3127] — inside the hull of the four annotations, pulled 35% toward h(Area).
Intuition — only now
A smoothie. The \(h_j\) are fruits; \(\alpha\) is the recipe; \(c_i\) is the glass. Learning never touches the fruit — it adjusts the recipe, one \(\alpha\) at a time, until the drink makes the next word predictable.
\(c_i = 0.5[2,0] + 0.5[0,2] = [1,1]\). Uniform weights give the plain average — no word stands out. The alignment model’s whole job is to make \(\alpha\) non-uniform when that helps predict the next word.
08 PART 6 — THE DECODER UPDATE
Where the context enters the state
What does every symbol mean?
| \(s_i\) | the decoder’s new state after writing word \(i\) — becomes the query \(s_{(i+1)-1}\) of the next step | \(\mathbb{R}^{d_s}\) |
| \(s_{i-1}\) | the previous state (Part 2) — what is already written and implied | \(\mathbb{R}^{d_s}\) |
| \(y_{i-1}\) | the previous output word (as an embedding) — the decoder reads its own text | \(\mathbb{R}^{d_e}\) |
| \(c_i\) | the fresh context (Part 5) — this is where attention enters the decoder, fed to every gate | \(\mathbb{R}^{d_h}\) |
| \(f\) | the GRU: three sigmoid gates and one tanh candidate (Lecture 03’s machinery, now drinking \(c_i\)) | function |
Exactly where \(c_i\) enters — the gate equations
\(c_i\) appears in three of the four lines — that is how literally the context is “used”: it informs the keep/take decision (\(z\)), the reset decision (\(r\)), and the proposed new memory (\(\tilde{s}\)). \(\sigma\) is the sigmoid \(\frac{1}{1+e^{-x}} \in (0,1)\); \(\odot\) is element-wise multiplication.
Read it in English
“Look at what I wrote (\(y_{i-1}\)), what I know (\(s_{i-1}\)), and what I just read (\(c_i\)); decide per dimension how much old memory to keep (\(1 - z_i\)) and how much new, context-informed memory to take (\(z_i\)); write the blend as the new state.”
Numbers — the four stages on our running step
Result: \(s_i = [\,0.6715,\ 1.0611,\ 0.2892,\ 0.5948\,]\). (For hand-computability the toy uses diagonal weight matrices — element-wise vectors; full matrices change the shapes of the arithmetic, not the structure of the equations.)
Intuition — only now
Editing a draft. Most sentences stay; a few get rewritten in light of a source you just re-read. \(z\) is the per-dimension keep/take dial — and the division of labor is the deep point: attention decided what became available; the gates decide how much enters memory.
The \(\mathbf{C\,c_i}\) terms vanish from all three gates, so the update degenerates to a plain GRU step over \((y_{i-1}, s_{i-1})\) — the decoder stops reading the source entirely. That is exactly the Lecture-07 bottleneck returning through the back door.
09 PART 7 — THE OUTPUT PROBABILITY
One score per word, then softmax again
What does every symbol mean?
| \(P(y_i)\) | the probability distribution over the next output word \(y_i\) — one probability per vocabulary entry | \(\mathbb{R}^{V}\), sums to 1 |
| \(y_i\) | the output token being produced at decoder position \(i\) | vocab index |
| \([\,s_i ; c_i\,]\) | concatenation — stack state and context into one vector of \(d_s + d_h\) dims, so the readout can weigh “what I have written” and “what I just read” jointly | \(\mathbb{R}^{d_s + d_h}\) |
| \(W_o\) | the readout / un-embedding — maps the concatenated vector to one logit per vocabulary word | \(\mathbb{R}^{V \times (d_s+d_h)}\) |
| \(b_o\) | the readout bias — one learnable offset per vocabulary word | \(\mathbb{R}^{V}\) |
| softmax | the same normalization as Part 4, over the vocabulary instead of the source — logits to probabilities | \(\mathbb{R}^{V} \to \Delta^{V-1}\) |
The paper’s full readout is a maxout layer over the triple \((y_{i-1}, s_i, c_i)\); the formula above is its modern simplified form. Our toy maps \(s_i\) alone through a fixed 4→5 readout so the arithmetic stays checkable — the live lab below notes the difference.
Read it in English
“Take the new state and the context, stack them, and score every word in the vocabulary; softmax turns the scores into probabilities. The same math as Part 4, one floor up: there softmax distributed attention over source words; here it distributes probability over the vocabulary.”
Why concatenation?
\(s_i\) and \(c_i\) live in different spaces (what I meant to say vs what I just read) — adding them would mix the two on fixed, equal footing. Stacking lets \(W_o\) learn how to weigh them, per dimension, per output word. (Dimensions also differ: \(d_s \ne d_h\) in the paper — addition would not even be shape-legal.)
Numbers — our step’s output
P(علاقہ) = 43.3% — the top pick, the word this entire step was preparing.
Intuition — only now
Two softmaxes, one philosophy: turn arbitrary scores into a fair budget. Part 4 spent the budget on where to look; this one spends it on what to write.
The gates compress \(c_i\) through a state update tuned for memory, not for word choice. Giving the readout direct access to \(c_i\) lets word selection consult the fresh context without competing with memory management — the paper’s ablations showed the direct path matters.
10 THE FULL PIPELINE
Every formula, one translation
The complete dependency chain, then the running example assembled end to end. If you can walk this chain aloud — symbols, shapes, operations, reasons — you own attention’s mathematics.
ONE TRANSLATION, BEGINNING TO END — i = 3, the step that writes علاقہ
“The European Economic Area was established in 1994”, \(T_x = 8\) → annotations \(h_1 \dots h_8\) (toy: the four noun-phrase words shown; \(h_{\text{Area}} = [0.9, 1.6, 0.2, 0.6]\)).
The decoder has written یورپی معاشی; \(s_{i-1} = [0.6, 1.8, 0.2, 0.9]\) asks: which word comes next in the noun phrase?
\(e_{3,j} = v_a^T \tanh(W_a s_{i-1} + U_a h_j)\) → [1.2891, 1.7152, 1.3384, 1.9510] — Area most compatible with “a noun is needed”.
softmax over \(j\) → \(\alpha_{3,j} = [0.1812, 0.2774, 0.1903, \mathbf{0.3512}]\); the row sums to 1.
\(c_3 = \sum_j \alpha_{3,j} h_j = [0.7335, 0.6744, 0.4848, 0.3127]\) — a convex blend leaning on Area.
\(s_3 = \text{GRU}(s_2, e(\)معاشی\(), c_3)\): gates \(z = [0.6683, 0.5920, 0.5799, 0.5498]\) → \(s_3 = [0.6715, 1.0611, 0.2892, 0.5948]\).
\(P(y_3) = \text{softmax}(\text{readout}(s_3))\) → P(علاقہ) = 43.3%. Append it; \(s_3\) becomes the next query; run the chain again — seven more rows complete the matrix of Lecture 08 §08.
Parts 3–7 rerun for every output word: fresh query, fresh α row, fresh \(c_i\). \(T_y\) rows × \(T_x\) columns = the \(O(T_x T_y)\) score matrix the paper itself flagged as its cost.
\(W_a, U_a, v_a\), every gate matrix, and the readout — all by backprop through the softmax Jacobian (see the PyTorch tab). The alignments you saw in Lecture 08 were never labeled; they are this arithmetic, trained.
Lecture 11 keeps softmax-and-blend, swaps the comparator for a dot product; Lecture 12 deletes the RNN around it. The chain you just walked is the skeleton of every LLM.
11 CODE LAB
The walkthrough, runnable
NumPy reproduces every printed number; the PyTorch tab runs the same computation with gradients — and shows the softmax Jacobian that trains the comparator.
12 MASTERY CHECK