⚡ GRU (2014) — The LSTM, on a Diet
Gated Recurrent Unit
Lecture 03 ended with a question: the LSTM's three gates and two states cure forgetting — but are all of them necessary? Cho et al.'s 2014 answer: no. One merged state, two gates, one convex blend — ~25% fewer parameters, usually the same accuracy. This deep dive takes the GRU apart with the same rigor: block architecture, live traces, hand-computed math.
THE SAME PROBE SENTENCE, TWO LAMPS PER WORD
“The keys to the cabinet are on the table.” — the subject–verb trap from Lecture 03. A GRU solves it with two dials instead of three: the update gate stays shut on the plural channel (remember by not writing), and the reset gate filters what each new draft may consult.
On the right, watch the cell read that sentence. Bright rose = overwriting now; bright sky = consulting the past.
WHY SIMPLIFY? ↓LIVE · TWO GATES PER WORD
h_t = (1−z_t)⊙h_{t−1} + z_t⊙h̃_tNotice the rose lamp go dark on filler words — that is memory being kept by simply not writing.
Start with the motivation, in hard numbers: gating works, but the LSTM charges four weight matrices for it. What does the fourth one buy?
02 WHY SIMPLIFY
The LSTM's bill
Every gated layer pays for its gates in weight matrices: vanilla RNN h·(h+x+1), GRU 3·h·(h+x+1), LSTM 4·h·(h+x+1). Drag the sizes and watch the premium scale.
Parameters per layer, all biases included. At 2016-era scales this decided what fit on one GPU.
The GRU's answer: merge the two states into one, fuse forget and input into a single update gate, and add one new idea — a reset gate for the candidate. Here is the whole machine.
03 THE CORE IDEA
One state, two gates, one blend
No separate cell state: the hidden state itself rides the highway, touched only by a multiply and an add. Two sigmoid gates steer it — update z_t and reset r_t. Hover the diagram.
04 STEP BY STEP · BLOCK ARCHITECTURE
Follow the signal, gate by gate
Start with the full diagram, fully lit — every wire, gate and blend at once — then tour the five moves of one GRU cell in data-flow order: merged state → reset gate → candidate → update gate → interpolation. A glowing dot rides the live data path at each step; the dialog underneath unpacks the underlying math, symbol by symbol.
Diagram conventions follow colah's blog and the original paper: Cho et al., "Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation" (2014) — the same four lines dissected in §07.
05 GATE INTUITION LAB
You be the gate
Each scenario puts you in charge of one gate's pre-activation z. Slide it, watch σ(z) open or close the gate, and get judged against what a trained GRU would do.
Gates in isolation are easy. The magic is two decisions per word, sustained over a whole sentence. Time to watch a full read.
06 LIVE CELL · READING A SENTENCE
Watch memory being managed
The same three probes as the LSTM lecture: a Winograd pronoun, a slow-burn review, and a subject–verb agreement trap. Step word by word — left, the two gates; right, the six state channels after each blend.
high → write new · low → keep old (1−z ≈ 1)
low → candidate ignores stale context
STATE h_t · six channels (teal = positive, rose = negative)
You've seen the behavior. Here is the exact machinery — four lines where the LSTM needed six.
07 THE GOVERNING EQUATIONS
Four lines. Every symbol touchable.
Hover or tap any highlighted symbol — the inspector below explains its job, shape and personality.
READING ORDER
① both gates come from the same concatenated input — one matmul, two slices, in practice.
② the candidate is the only layer that sees the filtered past r_t ⊙ h_{t−1}.
③ the blend closes the step: old scaled by (1−z), new by z.
④ parameter count: 3 × d_h × (d_h + d_x + 1) — exactly 3× a vanilla RNN layer, ¾ of an LSTM.
08 ONE TIMESTEP, BY THE NUMBERS
Hand-computed. No magic left.
Hidden size 2, input size 1, actual weight matrices. Step through all five computations and watch h_t = [0.511, −0.028] emerge — the code lab in §11 reproduces it exactly.
Forward pass understood. The backward pass is where the keep branch earns its keep — one gate fewer, same highway.
09 WHY IT TRAINS
The keep-branch highway
In a vanilla RNN, ∂h_t/∂h_{t−1} contains a weight matrix and a tanh′ — magnitudes below 1, compounding to zero. On the GRU's keep branch the factor is just (1 − z_t). Drag the keep factor and compare the two chains.
Same caveat as the LSTM, honestly: gradients into the gates themselves still pass through h and can shrink. But the memory path is safe — and the GRU buys that safety with one gate fewer. Where the LSTM holds f_t ≈ 1, the GRU simply holds z_t ≈ 0.
10 THE VERDICT
GRU vs LSTM — the honest scoreboard
Same family, same trick, different trade. Flip between the two stacks of equations, then check the head-to-head table.
| Aspect | Vanilla RNN | GRU · 2014 | LSTM · 1997 |
|---|---|---|---|
| State vectors | 1 · h only | 1 · h merged | 2 · private c_t + public h_t |
| Gates | none | 2 · update z, reset r | 3 · forget, input, output |
| Memory update | h = tanh(W·[h,x]) — full remix | (1−z)⊙h + z⊙h̃ — convex blend | f⊙c + i⊙c̃ — gated, additive |
| Gradient path | × W·tanh′ per step — vanishes | × (1−z) on the keep branch | × f only — the highway |
| Erase / write coupling | inseparable | coupled — writing z erases (1−z) | independent — f and i decide separately |
| Candidate sees | everything (no gate) | r ⊙ h — reset-filtered past | full h_{t−1} (no reset gate) |
| Memory privacy | none | none — h is fully exposed | output gate over tanh(c_t) |
| Parameters (per layer) | d·(d+h+1) | 3·d·(d+h+1) | 4·d·(d+h+1) |
| Best at | toy demos, teaching | small data, tight compute, fast iteration | very long sequences, precise memory |
One-line summary: the GRU proves that most of the LSTM's power lives in a single idea — learned, per-channel, additive updates — and that one well-placed dial (z_t) can do the work of two gates, as long as you accept that erasing and writing move together.
11 CODE LAB
From framework call to from scratch
The PyTorch tab is what you'll write at work — note what the API doesn't return. The NumPy tab is what's actually happening — one matmul, two slices, the blend. It reproduces §08's hand numbers to the third decimal.
12 SUMMARY
What to carry out of this room
The LSTM charges 4 weight matrices for 3 gates + 2 states. Cho et al. (2014) asked which of that is essential — and cut a quarter of the bill.
One state h_t replaces cell + hidden. Memory and output become the same vector: simpler, faster — but nothing is ever private.
h_t = (1−z_t)⊙h_{t−1} + z_t⊙h̃_t — z_t writes, (1−z_t) keeps, and the two always sum to 1. Forget and input, fused into one convex dial.
The GRU's one genuinely new idea: the candidate sees r_t ⊙ h_{t−1}, a filtered past — so a new clause can be drafted without stale context leaking in.
The keep branch multiplies by (1−z_t) only — no matrix, no tanh′. Hold z ≈ 0 and gradients flow hundreds of steps, exactly like the LSTM's f ≈ 1.
No consistent winner empirically. GRU for lean and fast, LSTM for long and precise — and both taught the field the gated-additive pattern transformers would later generalize.
13 MASTERY CHECK