⚡ LSTM (1997) — Memory by Design, Not Decay
Long Short-Term Memory
A fully interactive deep dive into the architecture that cured the RNN's amnesia. Lecture 02 ended on a wound — vanilla RNNs forget. Hochreiter & Schmidhuber's answer: memory that is written, kept, and erased on purpose, controlled by tiny learned switches called gates.
THE PROBLEM, IN ONE SENTENCE
“The keys to the cabinet are on the table.” The plural subject sits five words from its verb, behind a singular distractor. A vanilla RNN's memory fades every step; an LSTM protects the plural bit with a forget gate held near 1 — then spends it exactly at the verb.
On the right, watch a cell read that sentence. Three lamps per word: forget, input, output. Bright = gate open.
WHY RNNs FORGET ↓LIVE · THREE GATES PER WORD
c_t = f_t⊙c_{t−1} + i_t⊙c̃_tNotice how the forget lamp stays bright on the subject words — that is memory being chosen, not decaying.
To deserve the solution, first re-live the failure: gradients in a vanilla RNN are multiplied by the same weight at every step backwards — and repeated multiplication is a terrible memory policy.
02 THE VANISHING GRADIENT, FELT
Repeated multiplication is not a memory policy
Backprop through time multiplies the gradient by W·tanh′ once per step. Drag the sliders: the left bars are a vanilla RNN's surviving signal from word #1; the right bars are an LSTM whose forget gate simply stays open.
Bars show how much of word #1's signal is still alive after k timesteps. 24 steps shown.
VANILLA RNN · signal of word #1 after k steps
LSTM CELL · same signal, forget gate held at f
The fix is audacious: give memory its own private lane that no weight matrix is allowed to touch — only tiny learned switches may open or close it.
03 THE CORE IDEA
A conveyor of memory, guarded by gates
The cell state c_t runs straight across time like a conveyor belt. Three gates — each its own tiny sigmoid layer looking at [h_{t−1}, x_t] — are the only hands allowed to touch it. Hover the diagram.
04 STEP BY STEP · THE CLASSIC WALKTHROUGH
Follow the signal, gate by gate
The five moves every LSTM cell makes, in the exact order of the classic walkthrough — cell state → forget gate → input gate → cell update → output gate. A glowing dot rides the live data path at each step; the dialog underneath unpacks the underlying math, symbol by symbol.
Step order and diagram conventions follow “Introduction to Long Short-Term Memory (LSTM)” (Analytics Vidhya) and colah's blog; the equations are the standard Hochreiter & Schmidhuber (1997) formulation — the same six lines dissected in §07.
05 GATE INTUITION LAB
You be the gate
Each scenario puts you in charge of one gate's pre-activation z. Slide it, watch σ(z) open or close the gate, and get judged against what a trained LSTM would do.
Gates in isolation are easy. The magic is four decisions per word, sustained over a whole sentence. Time to watch a full read.
06 LIVE CELL · READING A SENTENCE
Watch memory being managed
Three classic probes: a Winograd pronoun, a slow-burn review, and a subject–verb agreement trap. Step word by word — left, the three gates; right, the six cell channels after each update.
high → previous cell content survives
high → this word enters the cell
high → h_t exposes the cell right now
CELL STATE c_t · six channels (teal = positive, rose = negative)
You've now seen the behavior. Here is the exact machinery that produces it — six lines that launched a thousand models.
07 THE GOVERNING EQUATIONS
Six lines. Every symbol touchable.
Hover or tap any highlighted symbol — the inspector below explains its job, shape and personality.
READING ORDER
① all four pre-activations come from the same concatenated input — one big matmul in practice, sliced four ways.
② only then does the cell update happen: forget, then write.
③ h_t is derived last, from the new c_t.
④ parameter count: 4 × d_h × (d_h + d_x + 1) — exactly 4× a vanilla RNN layer.
08 ONE TIMESTEP, BY THE NUMBERS
Hand-computed. No magic left.
Hidden size 2, input size 1, actual weight matrices. Step through all six computations and watch [h, c] = [0.515, −0.229], [0.732, −0.603] emerge — the code lab in §11 reproduces it exactly.
Forward pass understood. But the reason the LSTM conquered the 2010s lives in the backward pass.
09 WHY IT TRAINS
The gradient highway
In a vanilla RNN, ∂h_t/∂h_{t−1} contains a weight matrix and a tanh′ — magnitudes below 1, compounding to zero. On the cell path, ∂c_t/∂c_{t−1} = f_t. Nothing else. Drag the forget gate and compare the two chains.
Caveat, honestly: gradients into the gates themselves still pass through h_t and can vanish — LSTMs mitigate, not abolish. But the memory path is safe, and that is what carries long-range signal. Transformers later took the same idea further: residual connections are the gradient highway generalized.
10 THE FAMILY
LSTM vs GRU — same idea, leaner body
Cho et al. (2014) asked: do we need three gates and two states? The GRU says no — and often matches the LSTM with ~25% fewer parameters. Open the full GRU deep dive →
| Aspect | Vanilla RNN | GRU · 2014 | LSTM · 1997 |
|---|---|---|---|
| State vectors | 1 · h only | 1 · h merged | 2 · private c_t + public h_t |
| Gates | none | 2 · update z, reset r | 3 · forget, input, output |
| Memory update | h = tanh(W·[h,x]) — full remix | (1−z)⊙h + z⊙h̃ — convex blend | f⊙c + i⊙c̃ — gated, additive |
| Gradient path | × W·tanh′ per step — vanishes | × (1−z) on the keep branch | × f only — the highway |
| Erase / write coupling | inseparable | coupled — writing erases (1−z vs z) | independent — f and i decide separately |
| Parameters (per layer) | d·(d+h+1) | 3·d·(d+h+1) | 4·d·(d+h+1) |
| Forget-bias trick | n/a | partial (via z) | b_f = 1 — the classic init |
| Best at | toy demos, teaching | small data, tight compute | very long sequences, precise memory |
One-line summary: the LSTM wins not by remembering harder but by deciding — three learned gates turn memory from passive decay into an active, per-channel policy, and the additive cell lane keeps the gradient alive long enough to learn that policy.
11 CODE LAB
From framework call to from scratch
The PyTorch tab is what you'll write at work. The NumPy tab is what's actually happening — one matmul, four slices, and the famous line. It reproduces §08's hand numbers to the third decimal.
12 SUMMARY
What to carry out of this room
Vanilla RNNs multiply gradients by W·tanh′ every step back — memory decays exponentially. Long-range dependencies die in training, not in forward pass.
A private cell state updated only by f_t ⊙ c_{t−1} + i_t ⊙ c̃_t — multiply and add, never a matrix. Memory persists because nothing forces it to decay.
Forget (delete), input (write), output (reveal): three sigmoid layers, all reading [h_{t−1}, x_t], each learning its own policy. σ ∈ (0,1) makes them soft switches.
∂c_t/∂c_{t−1} = f_t. Keep the forget gate near 1 and gradients flow hundreds of steps — the same trick residual networks and transformers would later generalize.
4× the parameters and compute of a vanilla RNN, still strictly sequential — no parallelization across time. That last weakness is exactly what attention was invented to break.
GRU merges the two states and two of the gates: ~25% fewer parameters, usually within noise of the LSTM. Default recipe: LSTM with forget-bias 1, or GRU when small.
13 MASTERY CHECK