🏗 Depth — the Other Dimension

Deep & Stacked Recurrent Networks

Lectures 02–03 and the GRU deep dive mastered the horizontal dimension — memory flowing through time. This lecture adds the vertical one: stack recurrent cells so that layers read the readings of other layers. Deep RNN, deep LSTM, deep GRU — one skeleton, three cells: one climb per timestep, one memory per layer.

⇈ One Climb per Timestep ≣ One Memory per Layer ▶ Follow-the-Dot Walkthrough ∂ Hand-Computed 2-Layer Step ⚖ Depth vs Width

2013–15The deep eraGraves · Pascanu · Sutskever

2–4Layersthe practical sweet spot

×LParametersno weight sharing across layers

2Gradient pathstime + depth — both can vanish

THE SAME PROBE SENTENCE, THREE LAMPS PER WORD

“The keys to the cabinet are on the table.” — the agreement trap one last time. On the right, watch a 3-layer stack read it: for every word, three lamps fire in sequence — layer 1 perceives the word, layer 2 reads layer 1's pattern, layer 3 reads layer 2's.

The staggered light-up is the climb: bottom to top, inside a single timestep.

WHY GO DEEP? ↓

LIVE · THREE LAYERS PER WORD

h_t^[l] = Cell^[l](h_{t−1}^[l], h_t^[l−1])

Teal → sky → violet: the signal climbs the stack before the next word arrives.

First, the honest motivation: a single unidirectional layer already sees the whole past — so what could a second layer possibly add?

02 THE MOTIVATION

One layer reads words. A stack reads the reading.

Depth does not widen the temporal receptive field — it deepens the computation per timestep. Same sentence, three lenses: flip through what each layer of a trained stack gets to work with.

Hierarchy, not historyLayer 1 already sees the whole past. Depth buys successive re-processing: features, then features of features — the same reason CNNs stack convolutions.

Specialization emergesTrained stacks self-organize: lower layers track surface form (POS, morphology), upper layers track structure and semantics — measured in papers, visible in §06.

The priceLayers are sequential: L cells must fire per word, so latency scales ×L, parameters scale ×L, and the gradient gains a second vanishing path (§09).

The sweet spot2–4 layers. Beyond that, returns collapse — the field's answer to "deeper" was eventually a different architecture, not a taller LSTM.

Convinced depth is worth its bill? Then here is the machine — every wire of it, before we animate the climb.

03 THE BLUEPRINT

Anatomy of a stack

One input at the bottom, three complete recurrent cells above it, one output at the top — and a private memory wire per layer. Hover the diagram.

Vertical = same timestepThe climbing wire carries h_t^[l] — layer l's output at time t becomes layer l+1's input at the SAME t. No time travel involved.

Horizontal = per-layer memoryEvery layer keeps its own h_{t−1}^[l] (and c_{t−1}^[l] for LSTM). Three layers = three independent memories ticking in parallel.

No weight sharing across layersWeights are shared across TIME within a layer — never across layers. W^[1] reads words; W^[2] reads layer 1. Different jobs, different matrices.

Only the top emitsy_t = h_t^[L]. Lower layers are scaffolding: their states matter as inputs upward and memories forward — nothing else leaves the stack.

04 STEP BY STEP · BLOCK ARCHITECTURE

Follow the signal up the stack

Start with the full diagram, fully lit — spine, cells, memories, output — then tour one complete climb in data-flow order: input → layer 1 fires → the climb → layers 2→3 → output + handoff. A glowing dot rides the live data path at each step; the dialog underneath unpacks the math, symbol by symbol.

OVERVIEW · FULL DIAGRAM

Architecture references: Pascanu et al., "How to Construct Deep Recurrent Neural Networks" (2014) · Sutskever et al., "Sequence to Sequence Learning with Neural Networks" (2014, 4-layer LSTM) · Graves et al., "Speech Recognition with Deep Recurrent Neural Networks" (2013).

05 CELL FLAVORS + THE PRICE OF DEPTH

Same skeleton, three cells

The stacking recipe never changes — only the inside of the box does. Flip between stacked RNN, GRU and LSTM; then price a whole stack with the sliders. Watch the input fan-in jump from x to h above layer 1.

LAYERS · L3

HIDDEN SIZE · h128

INPUT SIZE · x100

Stacked RNN1 matrix / layer

—

Stacked GRU3 matrices / layer

—

Stacked LSTM4 matrices / layer

—

Statics understood. Now watch the thing move: a 2-layer LSTM reading two sentences, both layers' memories exposed.

06 LIVE STACK · READING A SENTENCE

Watch two layers think at two speeds

A 2-layer LSTM reads the series' two probes. Left, layer 1 — fast, surface-level channels reacting to every word. Right, layer 2 — slower channels integrating layer 1's digest into meaning. Step word by word.

TOKEN STREAM · click any word to jump

LAYER 1 · h_t^[1] — surface features (fast)

mean |h|

0.00

LAYER 2 · h_t^[2] — integrated meaning (slow)

mean |h|

0.00

READY

You've seen the behavior at two depths. Here is the exact machinery — one new symbol, the superscript l, and everything else you already know.

07 THE GOVERNING EQUATIONS

One new symbol. Every symbol touchable.

Hover or tap any highlighted symbol — the inspector below explains its job, shape and personality.

(h_t,c_t)^[l]=Cell^[l](h_t^[l−1],h_{t−1}^[l],c_{t−1}^[l];W^[l])

h_t^[0]=x_t · y_t=h_t^[L] · Cell∈ { RNN, GRU, LSTM }

READING ORDER

① the superscript ^[l] is the only new symbol — everything else (gates, states, candidates) now exists once per layer.

② each layer has TWO inputs: the vertical h_t^[l−1] from below (same t) and its own horizontal h_{t−1}^[l] (previous t).

③ the bottom layer's vertical input is the embedding: h_t^[0] = x_t.

④ the output is read off the top: y_t = h_t^[L]. Lower states never leave the stack.

08 ONE TIMESTEP, TWO LAYERS, BY THE NUMBERS

Hand-computed. Up the stack.

Hidden size 2 per layer, input size 1, actual weight matrices. Step through the full climb and watch y_t = [0.7215, −0.0432] emerge — the NumPy tab in §11 reproduces it to the fourth decimal.

Forward pass understood. The backward pass is where depth collects its tax: the gradient must now survive the climb DOWN as well as the trip through time.

09 WHY DEEP STACKS ARE HARD TO TRAIN

Gradients vanish in two directions

Gates protect the horizontal path through time. But the error at the top must also flow down through L nonlinear layers per timestep — a second compounding path. Drag the per-layer factor and depth, then toggle the standard cure.

Per-layer gradient factor (no residual)0.60

Depth · layers L4

residual / skip connection h^[l] + x^[l]

Honest footnote: with gated cells the per-layer factor is learned, not doomed — but early in training it is whatever the init gives you, which is why deep stacks historically shipped with residual connections (He et al., 2016, imported into RNNs) and layer normalization between layers.

10 THE VERDICT

Depth vs width — the honest scoreboard

Same parameter budget, two ways to spend it: one wide layer, or three narrow ones. Flip through the options, then check the head-to-head table.

Go deep when…the task is hierarchical — parsing, translation, speech — and you can afford ×L latency: depth buys abstraction per timestep that width cannot mimic.

Stay wide when…latency matters (on-device, real-time), the task is shallow (tagging, short classification), or training stability is the priority — one layer, one short gradient path.

Empirically…the 2013–2016 benchmarks (Graves, Sutskever) won with 2–4 layers + residual + inter-layer dropout. Beyond ~4 layers, returns collapsed — the field jumped to attention instead of stacking higher.

Aspect	Shallow & wide · 1×512	Deep & narrow · 3×256
Parameters (LSTM, x=300)	1,665,024	1,620,992 — same budget
Transforms per timestep	1 cell firing	3 sequential cell firings
Temporal receptive field	whole past	whole past — unchanged
What is learned	one flat representation	hierarchy: surface → phrase → meaning
Inference latency	minimal — one cell per word	×3 — layers are sequential
Gradient paths	time only	time + depth — needs residual / norm
States carried	1 × (h, c)	3 × (h, c) — one pair per layer
Regularization slot	output dropout only	inter-layer dropout (PyTorch dropout=)
Best at	tagging, short text, on-device	translation, speech, parsing — the 2015 recipe

One-line summary: stacking turns one reader into a reading committee — each layer interprets the layer below, at the price of ×L parameters, ×L latency, and a second gradient path that only residual connections fully tame.

11 CODE LAB

Depth: one argument, or from scratch

PyTorch hides the stack behind num_layers; Keras makes you say return_sequences=True so upper layers get fed; the NumPy tab shows what both are actually doing — and reproduces §08 to the fourth decimal.

STACKED RECURRENCE · THREE WAYS

12 SUMMARY

What to carry out of this room

01 · The idea

Feed one recurrent layer's output as the next layer's input — same timestep. Layers read readings: words → patterns → patterns of patterns.

02 · The wiring

Vertical wires carry h_t^[l−1] (same t, one layer up); horizontal wires carry each layer's own h_{t−1}^[l]. L layers = L independent memories and weight sets.

03 · What depth buys

Not a longer receptive field — a unidirectional layer already sees the whole past. Depth buys more nonlinear computation per timestep: hierarchy and specialization.

04 · The bill

Parameters ×L (nothing shared across layers), latency ×L (layers are sequential), and a second vanishing-gradient path — cured with residual connections and layer norm.

05 · The flavors

The skeleton is cell-agnostic: stacked RNN (1 matrix/layer), GRU (3), LSTM (4). Frameworks hide it behind num_layers + inter-layer dropout.

06 · The verdict

2–4 layers was the 2013–2016 sweet spot that won translation and speech. Deeper stopped paying — and the field's next move was a different architecture entirely. That is the next lecture.

13 MASTERY CHECK