⌗ arXiv 1409.0473 · SUBMITTED 1 SEP 2014 · ICLR 2015 (ORAL) ↗
The paper that taught machines to look
Neural Machine Translation
by Jointly Learning to Align and Translate
OPEN THE PAPER (PDF) ↗
2014’s translation models had a fatal habit: squeeze the entire source sentence into one fixed-length vector, then generate the whole translation from that crush. Long sentences collapsed. Three researchers made the smallest possible fix — let the decoder search the source again before every output word, and blend what it finds. They called the weights an alignment. The field never stopped using the word attention. This lecture rebuilds that paper piece by piece — the gap, the search, the score, the matrix — every mechanism live.
SEQ2SEQ + ATTENTION, ONE PICTURE
This is the paper’s whole idea in one view: the encoder turns English into annotations, and the decoder consults them again before every Urdu word — the arcs below ARE the alignment weights α, thick where attention is strong. Hover any word.
Watch the arcs: Urdu is an SOV language, so was established travels to the end — its arcs reach back six words. in 1994 flips to 1994 میں. And The? No arc leaves it strongly — soft attention spreads it thin. Nobody programmed word order; the model learned it from parallel text. §08 turns this map into the full matrix.
THE PAPER’S MODEL, MAPPING ITS ALIGNMENT — ENCODER → ANNOTATIONS → ATTENTION → DECODER
URDU OUTPUT ↓ · ENGLISH SOURCE →hover any word — its alignment weights light up
First, feel the disease before the cure: what exactly was wrong with 2014’s state of the art, and why everyone knew it was wrong only when sentences got long.
02 THE GAP
One vector for a whole sentence
Lecture 07’s encoder–decoder reads the English sentence left to right and hands the decoder one final hidden state — a single fixed-length vector c, however long the input. The encoder-decoder papers of 2014 (Cho et al., Sutskever et al.) both worked this way, and both worked… until they didn’t. The Bahdanau paper opens with the conjecture: “the use of a fixed-length vector is a bottleneck.” Drag the slider and watch why.
THE SOURCE SENTENCE · ITS URDU TRANSLATION
…ALL OF IT, INTO THIS — THE ONLY THING THE DECODER EVER SEES
WHAT THE VECTOR MUST REMEMBER
c isn’t just “the gist” — to translate, the decoder needs from it: every content word, the tense, the number agreement, the word order, the entities, the discourse state — a lossless summary of an arbitrary sentence in a fixed number of floats. The paper’s measured verdict: RNNencdec’s BLEU “deteriorates rapidly” past ~30 words, and its 50-word translations visibly “drift and lose meaning”. A 1000-dimensional vector is huge — but “fixed” is the killer, not “small.”
The fix will be: look again. But before the machine does it, try doing its job yourself — aligning two languages by hand teaches more than any equation.
03 YOU BE THE ALIGNER
Do the model’s job, by hand
Statistical MT spent decades engineering word alignments by hand-crafted rules and generative models. Here’s the whole problem on one sentence: click an English word, then its Urdu translation. Find all seven pairs — then reveal what RNNsearch actually learned.
SOURCE · ENGLISH — click one
TARGET · URDU — click its translation
You just built a hard alignment: one word ↔ one word. The paper’s move was softer and stranger — a whole probability distribution over the source, freshly drawn for every output word.
04 SOFT SEARCH
A private context for every word
Before emitting Urdu word yi, the decoder re-searches all English annotations and mixes a context vector ci — its own blend, different from every other word’s. Click through the Urdu sentence and watch each word pull from exactly the English it needs.
THE URDU OUTPUT — CLICK ANY WORD (in generation order)
α WEIGHTS OVER THE SOURCE — for that word only
THE BLEND IT DRINKS
Where do those α weights come from? From a tiny learned network that scores every decoder-state × source-word pair. One formula — unpack it number by number.
05 THE ALIGNMENT SCORE
A learned comparator
The paper’s equation (5): eij = vaT tanh(Wa si−1 + Ua hj) — a one-hidden-layer MLP that compares “what the decoder still needs” (s) against “what each source word offers” (h), then reads out one number. Softmax over the source turns numbers into α. Below: real arithmetic on hand-set 4-dim vectors, every step visible.
SCORING s = [0.6, 1.8, 0.2, 0.9] AGAINST EVERY ANNOTATION — computed live
SOFTMAX OVER THE FOUR SCORES → α (for the target word “علاقہ”)
Score, softmax, blend — three steps. Now assemble the full machine that uses them: RNNsearch, exactly as drawn in the paper. Click any block.
06 RNNSEARCH
The paper’s machine
Everything is still 2014 technology: GRUs everywhere, an RNN encoder, an RNN decoder. The only new part is the bridge in the middle. Press GENERATE and watch it translate English into Urdu live — each pass lights the α weights on the annotation cells, re-weighs the blend lines, and emits one word. Hover any block to trace its wires; click for its exact job.
THE MACHINE, RUNNING — ENGLISH IN, URDU OUT, ONE WORD PER PASS
THE URDU OUTPUT — the h-cells and blend lines light with each word’s α
c_i — THE BLEND THE CURRENT WORD DRINKS
WHAT JUST HAPPENED
The annotations deserve their own look — the paper’s quiet masterstroke was making each source word’s memory bidirectional.
07 ANNOTATIONS
Read both ways
A forward-only encoder hits the classic wall: by the time it reaches “purred”, the beginning has faded. The paper’s fix — run a second RNN backwards, and give each word the concatenation hj = [→ ; ←]. Toggle the directions and click words to see what each annotation carries.
Now the payoff picture: all α weights at once, output words down, source words across — the alignment matrix that made attention famous.
08 THE ALIGNMENT MATRIX
Figure 3, live
Every output word’s α distribution, stacked: this IS what attention looks like from outside. Hover cells for exact α; toggle hard/soft and feel what statistical MT had to engineer by hand — and what this paper simply learned. The boxed block is the famous adjective flip.
αi,j — URDU OUTPUT ↓ · ENGLISH SOURCE →
hover a cell to read its α
WHY SOFT BEAT HARD
The paper devotes a section to this: soft alignments handle the messy cases gracefully — an Urdu word can lean on two English ones at once, “The” needs no NULL token or forced partner, and nothing breaks when no clean 1:1 pairing exists. Hard argmax looks tidy on simple sentences and shatters on real ones. Soft is also differentiable — the whole matrix trains by gradient descent, end to end, with no alignment labels at all. The alignments you’re hovering were never annotated by anyone.
Attention fixed more than accuracy — it fixed training. The same weights that guide reading also carve a shortcut for gradients through the recurrence.
09 THE GRADIENT HIGHWAY
Distance-one gradients
Because ci touches every annotation directly — ci = Σ αijhj — the gradient from any output word reaches any source word in one step: ∂ci/∂hj = αij. The vanishing-gradient chains of Lectures 02–03 simply… route around. Drag the distance and compare the paths.
Did it actually work? The paper’s answer is one picture — BLEU against sentence length — and one table. Both, live.
10 RESULTS
The long-sentence rescue
Drag along the paper’s Figure 1: RNNencdec (rose) climbs, peaks, and falls off a cliff as sentences grow; RNNsearch (teal) just keeps its level. Below it, Table 1’s exact numbers — including the honest gap to the phrase-based champion of the era.
Every paper should end the way this one does: with its costs stated plainly — and then the legacy, which not even its authors could have priced.
11 THE LEDGER & THE LEGACY
What it cost, what it started
Scoring every target–source pair is O(Tx×Ty) — fine for sentences, questionable beyond (the same quadratic that Lecture 11’s Table 1 will meet again). Rare and unknown words remained unsolved; Moses still won the BLEU table, 33.30 vs 28.45.
Months later, Luong & Manning simplified the comparator: replace the MLP with a dot product of state and annotation. Same softmax blend, no extra network — the exact scoring the Transformer will keep.
Parikh et al. and Cheng et al. turned attention inward: let a sequence attend to itself, decomposing it into pairs. The retrieval trick stops needing two sequences.
“Attention Is All You Need” (Lecture 11): keep the scoring, the softmax, the blend — delete the RNNs around them. The patch becomes the whole model, and then GPT (Lecture 12) deletes the encoder too.
Soft search over a set, differentiable end to end, trained with no labels beyond the task itself. Translation was merely the first customer — vision, speech, protein folding and code all adopted the same three steps: score, softmax, blend.
seq2seq bottleneck (07) → ATTENTION (you are here) → Transformer (09) → GPT (10). Every rung after this one is a repackaging of the equation you hand-computed in §05.
12 CODE LAB
The 2014 mechanism, runnable
Three tabs: additive attention from absolute scratch in NumPy (§05’s exact arithmetic), the PyTorch module shaped like the paper, and the modern Keras built-in that returns the §08 score matrix.
13 MASTERY CHECK