CAS logo CAS NLP LABInteractive Tutorial Series ← Series

⌗  arXiv 1409.0473 · SUBMITTED 1 SEP 2014 · ICLR 2015 (ORAL) ↗

The paper that taught machines to look

Neural Machine Translation
by Jointly Learning to Align and Translate

Dzmitry Bahdanau · Jacobs University Bremen, Germany
Kyunghyun Cho · Yoshua Bengio · Université de Montréal, Canada

arXiv:1409.0473 · ICLR 2015 (oral presentation)

OPEN THE PAPER (PDF) ↗

2014’s translation models had a fatal habit: squeeze the entire source sentence into one fixed-length vector, then generate the whole translation from that crush. Long sentences collapsed. Three researchers made the smallest possible fix — let the decoder search the source again before every output word, and blend what it finds. They called the weights an alignment. The field never stopped using the word attention. This lecture rebuilds that paper piece by piece — the gap, the search, the score, the matrix — every mechanism live.

SOURCE WORDS→ BiGRU ANNOTATIONS hⱼ→ SCORE e_ij = a(s, h_j)→ SOFTMAX → α_ij→ c_i = Σ α_ij h_j→ NEXT WORD
13Sectionsgap → mechanism → results
11Live Demosincl. the famous matrix, live
+8.9BLEU vs seq2seq26.75 vs 17.82 at 50 words
8Quiz Checksthe mastery gate
THE ONE-SENTENCE SUMMARY

Attention replaces one context vector with one per output word: before emitting each Urdu word, the decoder scores every English word’s annotation against its own state, softmaxes the scores into alignment weights α, and reads a private blend c_i = Σ αij hj. The bottleneck dies; long sentences survive; three years later the same equation becomes the entire Transformer.

SEQ2SEQ + ATTENTION, ONE PICTURE

This is the paper’s whole idea in one view: the encoder turns English into annotations, and the decoder consults them again before every Urdu word — the arcs below ARE the alignment weights α, thick where attention is strong. Hover any word.

Watch the arcs: Urdu is an SOV language, so was established travels to the end — its arcs reach back six words. in 1994 flips to 1994 میں. And The? No arc leaves it strongly — soft attention spreads it thin. Nobody programmed word order; the model learned it from parallel text. §08 turns this map into the full matrix.

THE PAPER’S MODEL, MAPPING ITS ALIGNMENT — ENCODER → ANNOTATIONS → ATTENTION → DECODER

URDU OUTPUT ↓ · ENGLISH SOURCE →

hover any word — its alignment weights light up

First, feel the disease before the cure: what exactly was wrong with 2014’s state of the art, and why everyone knew it was wrong only when sentences got long.

02 THE GAP

One vector for a whole sentence

Lecture 07’s encoder–decoder reads the English sentence left to right and hands the decoder one final hidden state — a single fixed-length vector c, however long the input. The encoder-decoder papers of 2014 (Cho et al., Sutskever et al.) both worked this way, and both worked… until they didn’t. The Bahdanau paper opens with the conjecture: “the use of a fixed-length vector is a bottleneck.” Drag the slider and watch why.

SENTENCE LENGTH3
3 words81530

THE SOURCE SENTENCE · ITS URDU TRANSLATION

…ALL OF IT, INTO THIS — THE ONLY THING THE DECODER EVER SEES

c

WHAT THE VECTOR MUST REMEMBER

c isn’t just “the gist” — to translate, the decoder needs from it: every content word, the tense, the number agreement, the word order, the entities, the discourse state — a lossless summary of an arbitrary sentence in a fixed number of floats. The paper’s measured verdict: RNNencdec’s BLEU “deteriorates rapidly” past ~30 words, and its 50-word translations visibly “drift and lose meaning”. A 1000-dimensional vector is huge — but “fixed” is the killer, not “small.”

h_T = c  \u2014  one vector, no matter how long the sentence (2014)

The fix will be: look again. But before the machine does it, try doing its job yourself — aligning two languages by hand teaches more than any equation.

03 YOU BE THE ALIGNER

Do the model’s job, by hand

Statistical MT spent decades engineering word alignments by hand-crafted rules and generative models. Here’s the whole problem on one sentence: click an English word, then its Urdu translation. Find all seven pairs — then reveal what RNNsearch actually learned.

SOURCE · ENGLISH — click one

TARGET · URDU — click its translation

You just built a hard alignment: one word ↔ one word. The paper’s move was softer and stranger — a whole probability distribution over the source, freshly drawn for every output word.

Where do those α weights come from? From a tiny learned network that scores every decoder-state × source-word pair. One formula — unpack it number by number.

05 THE ALIGNMENT SCORE

A learned comparator

The paper’s equation (5): eij = vaT tanh(Wa si−1 + Ua hj) — a one-hidden-layer MLP that compares “what the decoder still needs” (s) against “what each source word offers” (h), then reads out one number. Softmax over the source turns numbers into α. Below: real arithmetic on hand-set 4-dim vectors, every step visible.

SCORING s = [0.6, 1.8, 0.2, 0.9] AGAINST EVERY ANNOTATION — computed live

SOFTMAX OVER THE FOUR SCORES → α (for the target word “علاقہ”)

Score, softmax, blend — three steps. Now assemble the full machine that uses them: RNNsearch, exactly as drawn in the paper. Click any block.

06 RNNSEARCH

The paper’s machine

Everything is still 2014 technology: GRUs everywhere, an RNN encoder, an RNN decoder. The only new part is the bridge in the middle. Press GENERATE and watch it translate English into Urdu live — each pass lights the α weights on the annotation cells, re-weighs the blend lines, and emits one word. Hover any block to trace its wires; click for its exact job.

THE MACHINE, RUNNING — ENGLISH IN, URDU OUT, ONE WORD PER PASS

THE URDU OUTPUT — the h-cells and blend lines light with each word’s α

c_i — THE BLEND THE CURRENT WORD DRINKS

WHAT JUST HAPPENED

Click any block for its exact job — or hover to trace its wires.

1,000 + 1,000units per encoder direction; annotations are 2,000-dim; decoder GRU 1,000; maxout 500; embeddings 620.
348M wordsWMT’14 En–Fr after data selection (from 850M); 30,000-word vocabulary per side; no lowercasing.
~5 daystraining per model with Adadelta, minibatches of 80 sentences; beam search at test time.

The annotations deserve their own look — the paper’s quiet masterstroke was making each source word’s memory bidirectional.

07 ANNOTATIONS

Read both ways

A forward-only encoder hits the classic wall: by the time it reaches “purred”, the beginning has faded. The paper’s fix — run a second RNN backwards, and give each word the concatenation hj = [→ ; ←]. Toggle the directions and click words to see what each annotation carries.

Look ahead to Lecture 11: the Transformer’s encoder keeps this exact property — every word’s representation sees both sides — but gets it from one attention step instead of two RNNs. Bidirectionality was worth a whole architecture change in 2014; self-attention made it free in 2017.

Now the payoff picture: all α weights at once, output words down, source words across — the alignment matrix that made attention famous.

08 THE ALIGNMENT MATRIX

Figure 3, live

Every output word’s α distribution, stacked: this IS what attention looks like from outside. Hover cells for exact α; toggle hard/soft and feel what statistical MT had to engineer by hand — and what this paper simply learned. The boxed block is the famous adjective flip.

αi,j — URDU OUTPUT ↓ · ENGLISH SOURCE →

hover a cell to read its α

WHY SOFT BEAT HARD

The paper devotes a section to this: soft alignments handle the messy cases gracefully — an Urdu word can lean on two English ones at once, “The” needs no NULL token or forced partner, and nothing breaks when no clean 1:1 pairing exists. Hard argmax looks tidy on simple sentences and shatters on real ones. Soft is also differentiable — the whole matrix trains by gradient descent, end to end, with no alignment labels at all. The alignments you’re hovering were never annotated by anyone.

Attention fixed more than accuracy — it fixed training. The same weights that guide reading also carve a shortcut for gradients through the recurrence.

09 THE GRADIENT HIGHWAY

Distance-one gradients

Because ci touches every annotation directly — ci = Σ αijhj — the gradient from any output word reaches any source word in one step: ∂ci/∂hj = αij. The vanishing-gradient chains of Lectures 02–03 simply… route around. Drag the distance and compare the paths.

DISTANCE: OUTPUT WORD → SOURCE WORD10
WITHOUT ATTENTION — through the recurrence
—
WITH ATTENTION — direct, α-weighted
—

Did it actually work? The paper’s answer is one picture — BLEU against sentence length — and one table. Both, live.

10 RESULTS

The long-sentence rescue

Drag along the paper’s Figure 1: RNNencdec (rose) climbs, peaks, and falls off a cliff as sentences grow; RNNsearch (teal) just keeps its level. Below it, Table 1’s exact numbers — including the honest gap to the phrase-based champion of the era.

SENTENCE LENGTH10
—RNNencdec-50 BLEU (approx from Fig 1)
—RNNsearch-50 BLEU (approx from Fig 1)

Every paper should end the way this one does: with its costs stated plainly — and then the legacy, which not even its authors could have priced.

11 THE LEDGER & THE LEGACY

What it cost, what it started

The stated costs

Scoring every target–source pair is O(Tx×Ty) — fine for sentences, questionable beyond (the same quadratic that Lecture 11’s Table 1 will meet again). Rare and unknown words remained unsolved; Moses still won the BLEU table, 33.30 vs 28.45.

2015 · Luong

Months later, Luong & Manning simplified the comparator: replace the MLP with a dot product of state and annotation. Same softmax blend, no extra network — the exact scoring the Transformer will keep.

2016 · Self-attention

Parikh et al. and Cheng et al. turned attention inward: let a sequence attend to itself, decomposing it into pairs. The retrieval trick stops needing two sequences.

2017 · The deletion

“Attention Is All You Need” (Lecture 11): keep the scoring, the softmax, the blend — delete the RNNs around them. The patch becomes the whole model, and then GPT (Lecture 12) deletes the encoder too.

The idea, distilled

Soft search over a set, differentiable end to end, trained with no labels beyond the task itself. Translation was merely the first customer — vision, speech, protein folding and code all adopted the same three steps: score, softmax, blend.

Where you are

seq2seq bottleneck (07) → ATTENTION (you are here) → Transformer (09) → GPT (10). Every rung after this one is a repackaging of the equation you hand-computed in §05.

12 CODE LAB

The 2014 mechanism, runnable

Three tabs: additive attention from absolute scratch in NumPy (§05’s exact arithmetic), the PyTorch module shaped like the paper, and the modern Keras built-in that returns the §08 score matrix.

ATTENTION · THREE WAYS

13 MASTERY CHECK

Eight questions — one per mechanism.

YOUR SCORE

0 / 8