◈ DEEP DIVE — LUONG ATTENTION · formula first · same verified toy as the Bahdanau dive
Four scorers, two scopes, one feed
One year after Bahdanau, Luong, Pham & Manning asked the natural follow-ups: how should the match be scored? (four functions) and how much source should be read? (global vs local). This page teaches the paper formula-first — every identifier, every shape, every why, every number computed on the same verified toy as the previous deep dive, so you can watch the same query and memories produce different attention under different scorers.
THE FOUR SCORERS (§03)
\[ \begin{aligned} \text{dot}:&&\; s_t^\top h_i&&\text{general}:&&\; s_t^\top W_a h_i\\ \text{concat}:&&\; v_a^\top \tanh(W_a[s_t;h_i])&&\text{location}:&&\; \alpha_t=\text{softmax}(W_a s_t) \end{aligned} \]GLOBAL vs LOCAL (§04–05)
\[ c_t = \sum_i \alpha_t(i)\, h_i \quad\text{vs}\quad p_t = S\cdot\sigma(v_p^\top \tanh(W_p h_t)) \]THE OUTPUT (§05–06)
\[ \tilde{h}_t = \tanh(W_c[c_t; s_t]) \quad\quad p(y_t) = \text{softmax}(W_s \tilde{h}_t) \]THE SAME STEP, THREE COMPARATORS — SWITCH AND WATCH ATTENTION MOVE
RAW SCORES e(i)
ATTENTION WEIGHTS α(i)
CONTEXT cₜ
02 THE MACHINE
The network, block by block
The whole Luong architecture as one running diagram: a 2-stack LSTM encoder keeps every annotation; the attention stack scores, softmaxes and blends; the attentional vector joins context with state; and input-feeding loops it all onward. Hover a block to trace its wires, click to jump to the section that teaches it, and toggle GLOBAL vs LOCAL to see the window.
GLOBAL ↔ LOCAL — the scope toggle
02 THE INDEXING MAP & NOTATION HANDOFF
From i, j to t, i
Luong re-notates Bahdanau: the decoder step becomes t (not i), and the source position keeps i. One symbol changes meaning from the last deep dive — the map below keeps you honest.
decoder / output step — was i in the Bahdanau dive. \(s_t\) = the state that attends; \(\tilde{h}_t\) = the attention-aware state it produces.
encoder / source position — was j. \(h_i\) (paper: \(\bar{h}_s\)) = source annotation. \(h_i \in \mathbb{R}^{2d}\) here: the encoder is a 2-stack LSTM (1000+1000).
source length — Luong writes S. \(T_y\) = number of output steps.
local half-window — the paper sets D = 10; a local step reads at most 2D+1 source positions. Our toy uses D = 3.
SHAPES AT FULL SCALE
\(h_i \in \mathbb{R}^{2000}\), \(s_t \in \mathbb{R}^{1000}\) — note they now differ: that asymmetry is exactly why the general scorer’s rectangular \(W_a\) exists. Our toy keeps both at 4 so every product stays pencil-checkable.
03 PART 1 — FOUR WAYS TO SCORE A MATCH
The comparator is a design choice
Bahdanau fixed one scorer and moved on. Luong’s first contribution is the question itself: what is the right function to compare a decoder state with a source annotation? Four candidates, formula first — then every symbol, why each exists, and what each did to our verified toy.
Symbols · Read · Why
| \(s_t^\top h_i\) | a plain dot product — \(\sum_d s_{t,d}\, h_{i,d}\): how much the two vectors point the same way | \(\mathbb{R}\) |
| \(s_t\) | decoder state at step \(t\) — the query | \(\mathbb{R}^{d}\) |
| \(h_i\) | source annotation at position \(i\) | \(\mathbb{R}^{d}\) |
“Multiply the two vectors element-wise and add. Nothing learned, nothing to tune — pure geometry.”
Why it exists: as the parameter-free baseline. It requires \(d_s = d\) (equal dims — true in the paper since both are 1000-dim stacks). Its result on our toy is the most extreme: Area takes 87.0% — raw similarity already separates hard.
\(e(\text{Area}) = 0.6(0.9) + 1.8(1.6) + 0.2(0.2) + 0.9(0.6) = 0.54 + 2.88 + 0.04 + 0.54 = \mathbf{4.00}\) — check dimension 2: \(1.8 \times 1.6\) alone contributes 2.88 of the 4.00.
Symbols · Read · Why
| \(W_a\) | the learned match matrix — rescales and mixes dimensions before the dot; it lives BETWEEN the two vectors (shape \(d_s \times d_h\), rectangular when dims differ — which they do here: 1000 vs 2000) | \(\mathbb{R}^{d_s \times d_h}\) |
“First let \(W_a\) judge which dimensions matter for a match, then take the dot product.”
Why it exists: dot is stuck with whatever geometry the encoder happened to learn; \(W_a\) buys the attention its own opinion. And because \(s\) and \(h\) have different widths in this architecture, some matrix is needed anyway. Our toy: a diagonal \(W_a = \operatorname{diag}(0.5, 0.6, 0.4, 0.5)\) — each dimension gets its own importance — softening Area to 66.4%.
\(e(\text{Area}) = \sum_k s_k\, w_k\, h_k = 0.6(0.5)(0.9) + 1.8(0.6)(1.6) + 0.2(0.4)(0.2) + 0.9(0.5)(0.6) = \mathbf{2.284}\)
Symbols · Read · Why
| \([s_t; h_i]\) | concatenation — stack query and memory into one \(2d\)-dim vector | \(\mathbb{R}^{2d}\) |
| \(W_a\) | now maps the pair jointly to comparison space — a block row \([W^{(1)}\; W^{(2)}]\) | \(\mathbb{R}^{d_a \times 2d}\) |
| \(v_a^\top \tanh(\cdot)\) | the Bahdanau readout, verbatim | \(\mathbb{R}\) |
“Stack the pair, mix them jointly, squash, read out one number.”
Why it matters here: expand the block product — \(W_a[s_t;h_i] = W^{(1)}s_t + W^{(2)}h_i\) — and with \(W^{(1)} = W_a^{Bah},\ W^{(2)} = U_a^{Bah}\) this is literally Bahdanau’s additive scorer. One formula, two notations. Our toy proves it numerically: concat scores = [1.2891, 1.7152, 1.3384, 1.9510], digit-for-digit the previous deep dive’s e. (In the paper this variant underperformed — an implementation note, not a theorem.)
Symbols · Read · Why
| \(\alpha_t\) | the whole alignment vector at once — computed from the target state ALONE (per-position scores \(W_a s_t\) truncated/padded to source length) | \(\mathbb{R}^{T_x}\) |
“Don’t look at the source at all — predict the alignment directly from where I am in writing.”
Why it exists: as the cheap lower bound — and as the honest experiment that answered: can attention work without comparing to the source? Answer: alignments came out poor (small unk-replace gains, 16.4 vs 18.6 BLEU). Attention that cannot see cannot align.
THE SCOREBOARD — all four on the same toy (live-computed)
Same \(s_t\), same four \(h_i\), same softmax — only the comparator changed. The softmax normalizes whatever opinion the scorer holds; the opinion is the design choice.
Dot product is only defined between equal-length vectors, so with a 1000-dim decoder and 2000-dim encoder it simply cannot run. \(W_a \in \mathbb{R}^{d_s \times d_h}\) consumes the 2000-dim annotation and emits a 1000-dim vector that matches \(s_t\) — the rectangle is what makes mismatched dims legal.
04 PART 2 — GLOBAL ATTENTION
The Bahdanau read, cleaned up
What changed vs Bahdanau?
| \(\alpha_t(i)\) | same softmax-over-source as equation (6) of the last dive — new letter, same object | \((0,1)\), sums to 1 |
| \(c_t\) | same convex blend of ALL \(T_x\) annotations — “global” means the window is the whole sentence | \(\mathbb{R}^{2d}\) |
| WHAT DIFFERS | not the read — the scorer feeding it (Part 1) and, decisively, where \(c_t\) goes next (Part 3: the attentional vector) | — |
Numbers — global with the general scorer
\(\alpha_t = [0.0793,\ 0.1262,\ 0.1302,\ 0.6644]\) — sum = 1.000 ✓
\(c_t = 0.0793\,h(\text{The}) + 0.1262\,h(\text{European}) + 0.1302\,h(\text{Economic}) + 0.6644\,h(\text{Area}) = \mathbf{[\,0.8047,\ 1.1273,\ 0.3865,\ 0.4531\,]}\)
Dimension 2 by hand: \(0.0793(0) + 0.1262(0.2) + 0.1302(0.3) + 0.6644(1.6) = 0.025 + 0.039 + 1.063 = 1.127\) ✓
Why keep the global variant at all, if local is sharper?
Because global is the upper bound on information: every source position is consultable at every step — exactly Bahdanau’s soft search. It also costs \(O(T_x)\) per step and its attention spreads thin over long sentences; local attention (Part 4) is the cheaper, sharper answer for exactly those cases.
Reading ALL positions every step is already high-capacity — the softmax over \(T_x\) scores does the discriminating. Local attention throws most positions away, so the surviving comparison must be sharper: the learned \(W_a\) provides that extra selectivity. (An empirical regularity, not a law — but a repeatable one.)
05 PART 3 — THE ATTENTIONAL VECTOR
h̃ₜ — where context and state meet the output
What does every symbol mean?
| \(\tilde{h}_t\) | the attentional vector — the step’s true representation: what I meant (\(s_t\)) merged with what I just read (\(c_t\)) | \(\mathbb{R}^{d}\) |
| \([\,c_t ; s_t\,]\) | concatenation — the join Bahdanau buried inside the GRU, here made explicit at the output | \(\mathbb{R}^{3d}\) (2d + d) |
| \(W_c\) | the mixing layer — learns HOW context and state combine | \(\mathbb{R}^{d \times 3d}\) |
| \(W_s\) | the output readout — one logit per vocabulary word, eating \(\tilde{h}_t\) only | \(\mathbb{R}^{V \times d}\) |
Read it in English
“Mix what I wanted with what I found; squash; read the vocabulary off the mixture.”
Why does this layer exist?
Bahdanau fed \(c_i\) into the GRU gates and let the state carry it forward — the context reached the output only after passing through a recurrence tuned for memory, not for word choice. Luong adds a direct, dedicated join: one tanh layer whose only job is fusing the two views. The decoder becomes “attention-aware” at the exact place prediction happens.
Numbers — our step, computed live
\(\tilde{h}_t = [\,0.7937,\ 0.9295,\ 0.5452,\ 0.7728\,]\) — all four dims positive: the tanh of the joint read is an encouraging mixture. This vector — not \(s_t\), not \(c_t\) — is what the vocabulary softmax consumes.
Intuition — only now
Two witnesses, one statement. \(s_t\) saw the draft; \(c_t\) saw the source. \(W_c\) is the interview room where both testimonies are reconciled before the verdict \(p(y_t)\) is read.
The concat \([c_t; s_t]\) stacks \(2d\) (encoder) + \(d\) (decoder) = \(3d\) numbers; a matrix must span every input it multiplies — hence \(W_c \in \mathbb{R}^{d \times 3d}\). Shape arithmetic is the fastest way to catch attention-implementation bugs.
06 PART 4 — LOCAL ATTENTION
Predict where to look, then window it
Global attention reads all \(T_x\) positions every step — expensive, and its attention blurs on long sentences. Local attention predicts ONE center \(p_t\), reads a window of \(2D+1\) around it, and weights the window with a Gaussian. Three formulas, formula first.
What does every symbol mean?
| \(p_t\) | the predicted aligned position — a real number in \([0, S]\), not a rounded integer | \([0, S]\) |
| \(S\) | source length — the rescaler that maps the sigmoid’s (0,1) onto valid positions | integer |
| \(\sigma\) | the sigmoid \(\frac{1}{1+e^{-x}}\) — smooth step into (0,1) | \(\mathbb{R} \to (0,1)\) |
| \(W_p, v_p\) | the position predictor’s lens and judge — trained by gradient descent | \(\mathbb{R}^{a \times d}, \mathbb{R}^{a}\) |
| \(\tilde{h}_s\) | the source-side state the prediction peeks at (monotonic variant: \(p_t = t\), no learning at all) | \(\mathbb{R}^{d}\) |
Numbers — our step
\(S = 8\); with the toy \(W_p, v_p\): \(\sigma(\cdot) = 0.7726\) → \(p_t = 8 \times 0.7726 = \mathbf{6.18}\) — the predictor points between position 6 (“established”) and 7 (“in”).
Read it in English
“Score every position as in global attention, then multiply by a bell curve centered at the predicted spot — softly erasing everything far away. The Gaussian is differentiable, so the prediction \(p_t\) itself learns from the translation loss.”
Why a Gaussian, and not a hard window?
A hard cut (\(i \in [p_t - D, p_t + D]\), zero elsewhere) is a cliff — no gradient tells the predictor it aimed wrong. The Gaussian is a soft mask: full weight at the center, smooth fade at the edges, differentiable almost everywhere. Same trick family as softmax itself: replace a discrete decision with a smooth weighting so training can flow.
Numbers — the window on our sentence
A subtle identity: multiplying the global α by the Gaussian and renormalizing equals the paper’s formula exactly — the global normalizer cancels. The toy lands 93.3% on one position: local attention is sharp.
Intuition — only now
A flashlight instead of a floodlight: predict where to shine, keep a soft halo, black out the rest. Cheaper per step (\(2D+1\) reads, fixed), sharper alignments (AER 0.34–0.36 vs global’s 0.39), and — the paper’s Table 1 — the best single-model BLEU.
The window spans positions 3–9 (clipped to 1–8); the Gaussian gives 0.73 at 5, 0.99 at 6, 0.86 at 7, 0.48 at 8 — and only 0.003 at position 1, 0.02 at 2. Whatever survives the multiply-and-renormalize concentrates near the prediction; positions 1–2 are effectively unread this step.
07 PART 5 — INPUT-FEEDING
Tell the next step where you just looked
What does every symbol mean?
| \(\tilde{h}_t\) | the attentional vector (Part 3) — carried INTO the next decoder step as extra input | \(\mathbb{R}^{d}\) |
| \(e(y_{t-1})\) | the embedding of the word just written — the ordinary autoregressive input | \(\mathbb{R}^{d_e}\) |
Read it in English
“Every step receives the usual previous word plus the previous step’s attention result — so alignment choices stop being independent and start forming a coherent scan of the sentence.”
Why does it exist?
Without it, each step picks its attention blind to the last — a model can hop between unrelated words with no penalty, and can’t learn strategies like “continue rightward from where I was”. Feeding \(\tilde{h}_t\) onward made the model “fully aware of previous alignment choices”, bought +1.3 BLEU, and deepened the effective network across time. It is also general: it needs no attention at all — any stacked RNN can eat its own layer-1 output this way.
Intuition — only now
A translator reading with a finger on the page: each glance starts from where the finger rests. Input-feeding is the finger.
Bahdanau feeds the context \(c_i\) into the GRU gates. Luong feeds the attentional vector \(\tilde{h}_t\) — the post-mixing result — as raw input to the next step, which is strictly more general (any recurrent stack can adopt it, attention or not) and reaches the next step unfiltered by gates.
08 PART 6 — RESULTS
The BLEU waterfall
English–German, WMT&rsquo14 newstest2014, tokenized BLEU — every step of the recipe, in order. (This paper’s experiments are En–De and De–En only; newstest2013 served as the dev set.)
ALIGNMENT QUALITY — Alignment Error Rate (lower is better)
Local attention’s alignments are sharper than global’s — nearly matching the supervised Berkeley aligner — while needing no alignment labels at all.
09 THE FULL PIPELINE
The chain, Luong edition
ONE STEP, END TO END — the general scorer, our verified toy
\(e(i) = s_t^\top W_a h_i\) (diagonal toy \(W_a\)) → [0.158, 0.623, 0.654, 2.284].
softmax over all \(i\) → \(\alpha_t = [0.0793, 0.1262, 0.1302, \mathbf{0.6644}]\).
\(c_t = \sum_i \alpha_t(i) h_i = [0.8047, 1.1273, 0.3865, 0.4531]\) — the global read.
\(\tilde{h}_t = \tanh(W_c[c_t; s_t]) = [0.7937, 0.9295, 0.5452, 0.7728]\) — the attentional vector.
\(p(y_t) = \text{softmax}(W_s \tilde{h}_t)\) — the readout picks the next word.
\(\tilde{h}_t\) joins the input of step \(t+1\) — the finger stays on the page.
The comparator is a design space, not a given. Concat ≡ additive (notation, not mechanism). Location-only attention aligns poorly — matching the source is essential. Local attention with a differentiable window is sharper and cheaper.
Attention still rides inside an RNN — sequential steps, \(O(T_x)\) reads per token, windows to tune. The 2017 paper’s deletion (Lecture 11) answers the first two; the Transformer’s multi-head dot-product scorer is dot-plus-\(W_a\), scaled.
Bahdanau (soft search) → Luong (scorers, local, feeding) → Transformer (09) → GPT (10). Every LLM’s attention still carries this paper’s fingerprints: a scoring function, a softmax budget, a weighted read.
10 CODE LAB
The four scorers & local-p, runnable
11 MASTERY CHECK