02 CAS · AI ENGINEERING · DEEP LEARNING · NLP SERIES
RECURRENT NEURAL NETWORKS
The first architecture that gave neural networks a working memory — one set of weights, applied over and over through time, carrying every previous word forward inside a single evolving vector.
HOW A HUMAN READS · pick a sequence
You don't read word 9 with amnesia about words 1–8. Your mental state accumulates. That is exactly what an RNN's hidden state does. Choose a sentence:
THE MEMORY TRAIL · h evolves with every word
h_t = tanh(W_hh·h_{t-1} + W_xh·x_t)Each block is the hidden state h_t after reading the word above it — the colors literally change as meaning shifts.
Before we can appreciate the solution, we must feel the pain: feed-forward networks are amnesiacs. They see a bag of words, not a sentence.
02 THE PROBLEM RNNs SOLVE
Feed-forward networks read like a bag of words
A standard MLP has three fatal blind spots on sequences: it needs a fixed-size input, it forgets word order, and it shares nothing across positions. Shuffle the sentence below and watch each architecture react.
BAG-OF-WORDS MLP · ignores order
Averages the word vectors — mean(v) is identical no matter how you scramble the input.
Waiting…
RECURRENT NETWORK · reads left → right
Folds each word into a running memory — order changes every step, so the final memory changes too.
Waiting…
Fixed-size input. An MLP's first layer is a rigid matrix — a 3-word sentence and a 300-word review simply do not fit. The RNN sidesteps this: its loop runs once per token, so any length works with zero padding or truncation, and the same weights are shared across every position.
The fix is embarrassingly simple: feed the network its own previous opinion. That feedback wire is the entire invention.
03 ARCHITECTURE
One cell, a loop — and its unrolled in time
An RNN is a cell with a self-loop: its output memory feeds back into its next computation. Unrolling the loop across time turns the recurrence into a very deep feed-forward graph — one layer per time step — which is exactly how we train it.
Drawing circles is easy. Now press play and watch real numbers flow through a real cell, step by step.
04 LIVE SIMULATOR
The forward pass, step by step
A real RNN runs below — no faked numbers. Words enter as one-hot vectors (all zeros, one 1), exactly as they did in the original RNN era. Step, play, or click a token to jump.
TENSOR INSPECTOR · live values
Press PLAY or NEXT to begin.
You just watched the forward pass. Two small equations produce that entire choreography — hover every symbol until it has no secrets left.
05 MATHEMATICAL ANATOMY
Two equations govern everything
The complete vanilla RNN is the recurrence below — nothing more. Click any symbol to inspect its shape, role, and a live miniature of the actual matrix used in the simulator above.
and at output steps: p_t = softmax(y_t) · shapes here: V = D_x = 6, D_h = 5, D_y = 4 (matching the simulator above)
h_t — hidden state
shape [D_h]06 WHY tanh AND softmax
The squashing and the currency of probability
tanh keeps memory bounded in (−1, 1) so recurrence cannot blow up, but its derivative fades to zero at the extremes — the seed of the vanishing-gradient problem. Softmax turns raw logits into a probability distribution; cross-entropy punishes confident mistakes.
Drag the pre-activation magnitude. Watch the function value (and its derivative — the learning signal) collapse in the saturation zones.
Next-word logits for context “The lion hunted its…”. Target: prey. Drag any logit and watch probability and loss react.
Perfect confidence → loss 0. Uniform guessing → loss ln(4) ≈ 1.39. Confidently wrong → loss explodes toward ∞. That explosion is the learning signal.
07 TENSOR SHAPES & CAPACITY
Shapes flow, and the size of the net never depends on T
Get the matrix shapes right once and every RNN implementation becomes readable. Then notice the deeper fact: whether the sentence has 5 words or 5,000, the parameter count is identical.
Conventions differ (row-vector form h = xW + hW vs column form h = Wh + Wx). Learn one, translate on sight — the shapes below use the column convention of our equations. And in the RNN era the input was never a dense embedding: x_t was a one-hot vector, so W_xh·x_t is literally a row lookup — the learned row is the word vector.
| Parameter | Shape | Count |
|---|---|---|
| W_xh input → hidden · one-hot row lookup | [D_h × D_x] | — |
| W_hh hidden → hidden | [D_h × D_h] | — |
| b_h hidden bias | [D_h] | — |
| W_hy hidden → output | [D_y × D_h] | — |
| b_y output bias | [D_y] | — |
| TOTAL | Σ | — |
D_h² dominates: the recurrent matrix grows quadratically with memory size — sequence length T appears nowhere in the formula. With one-hot inputs, W_xh scales with the vocabulary: every word owns one learned row.
08 TRAINING
Backpropagation through time
To train, unroll the RNN into a T-layer network, total the loss over every step, and backpropagate. The twist: because weights are shared across time, every step's gradient flows into the same matrices — and gradients must travel through the recurrence again and again.
◈ DEEP DIVE AVAILABLE — Want to see every single number of a real forward pass and full BPTT backpropagation, computed by hand on a 3-word sentiment example?
Open the step-by-step numerical walkthrough →Language modeling: each step's cross-entropy, added up. Perplexity is just exp(L/T) — average confusion per token.
A gradient walking k steps back in time is a product of k Jacobians — k copies of W_hh (modulated by tanh derivatives < 1). Small |W_hh| ⇒ exponential shrink. Large ⇒ exponential blow-up. This single equation is the story of section 09.
Each link multiplies by roughly |W_hh| × 0.4 (a typical tanh′ factor). Set the chain length with T below and see section 09's experiment.
09 THE INVISIBLE WALL
Vanishing & exploding gradients — the experiment
The gradient that reaches early words is roughly (|W_hh| × tanh′)k after walking k steps back. Slide the values and watch long-range learning die — or detonate. Then try the fixes.
Memory mechanics mastered — now the design question: how many inputs and outputs does your problem have?
10 I/O TOPOLOGIES
The architecture zoo
Same cell, five different wiring diagrams — named by how many inputs and outputs they expose. Pick a pattern and watch a sequence flow through it.
—
CLASSIC TASKS
11 COMPOUNDING CELLS
Bidirectional & stacked-deep RNNs
Two upgrades, zero new math. Bidirectional RNNs read the sequence twice — once each way — so every position sees past and future. Stacked RNNs feed one RNN's hidden sequence into the next, building hierarchical features exactly like CNN layers.
12 THE FLAGSHIP USE
Seq2seq: encoder → decoder — and its beautiful flaw
Chain two RNNs and any sequence can map to any other sequence: translation, summarization, dialogue. The decoder is trained with teacher forcing — and the way it degrades when left alone previews why attention was invented.
Reads the source, hands its final memory to the decoder as the context vector — the entire meaning of the sentence in one h.
Each step predicts the next word. In teacher-forcing mode it is fed the true previous word; in free-running mode it eats its own prediction — errors compound.
Press RUN to translate.
The whole source sentence must fit inside one fixed-size vector. Longer sentences ⇒ the context vector becomes a bottleneck and quality collapses. The 2015 fix — let the decoder look back at all encoder states with learned weights — is called attention, and it eventually became the architecture (Lecture 04).
Enough theory. Below, a genuine char-level RNN — written in raw JavaScript — trains live in your browser tab. Watch it learn to speak.
13 PROOF IT ALL WORKS
Train a real RNN right here — in your browser
No server, no GPU, no libraries: the code panel below implements every equation from sections 04–09 — forward pass, BPTT, clipping, Adagrad — in plain JavaScript. Press START and watch loss fall while the network's babble becomes words.
H = 42 hidden units · BPTT over 24 chars · Adagrad lr 0.1 · grad-clip 5 · inputs: one-hot characters (V ≈ 24) · corpus: 512 chars
Press START TRAINING — the first samples are pure noise; watch what 2,000 steps do.
▼ THE ACTUAL TRAINING LOOP · full vanilla-RNN code (forward + BPTT + Adagrad) — the same algorithm the tab is running
14 IMPLEMENTATIONS
The code lab: from scratch → frameworks
Three ways to write the same network — and all three keep the one-hot inputs of the RNN era. NumPy exposes every equation; PyTorch and TensorFlow show how production code hides the loop inside nn.RNN / SimpleRNN.
15 WHERE RNNS RUN THE WORLD
Any data with order is RNN territory
Language was the motivation, but recurrence applies wherever time or sequence exists. Click a card to see the exact RNN topology each problem uses.
16 FOUR DECADES OF MEMORY
A short history of the loop
The RNN was not one paper but a lineage — from statistical mechanics to the seq2seq systems that first made machines translate. Click each milestone.
17 THE WHOLE LECTURE, COMPRESSED
The RNN cheat sheet
A weight-sharing network applied across time, carrying a hidden state: x_t enters as a one-hot word vector, then h_t = tanh(W_hh·h_{t-1} + W_xh·x_t + b_h), y_t = W_hy·h_t + b_y.
Feed-forward nets can't take variable lengths, ignore order, and share nothing across positions. Recurrence fixes all three with one loop.
Folded = drawing. Unrolled = computation: a T-layer tied-weight network that backprop trains directly.
Total cross-entropy over steps, then BPTT. The recurrent gradient is a product of k Jacobians: ∏ W_hhᵀ·diag(tanh′).
|W_hh| < 1 ⇒ gradients vanish exponentially (no long-range learning). |W_hh| > 1 ⇒ explode (training diverges). Clipping, truncation, init, gates.
one-to-many · many-to-one · many-to-many (synced) · many-to-many (seq2seq) — plus bidirectional and stacked compounds.
Encoder–decoder seq2seq with teacher forcing; its fixed-vector bottleneck led directly to attention.
The sequential bottleneck (step t waits for t−1) blocked massive parallelism — Transformers replaced RNNs in NLP. But recurrence remains the natural tool for streaming and online decisions, and its ideas — state, gates, decoding — echo through modern AI.
A complete step-by-step numerical walkthrough: every matrix multiplication, every tanh, every gradient chain on a 3-word sentiment example. One-hot inputs, hidden states, softmax, cross-entropy loss, and full BPTT with weight updates.
Open the walkthrough →18 MASTERY CHECKPOINT
Prove the loop stuck
Eight questions spanning intuition, architecture, math, dynamics, and code. Explanations appear after every answer.
“Memory, shared through time, trained by its own errors — master this loop and Transformers will read like a review, not a revelation.”
← BACK TO CAS-NLP SERIES HUB