⇄ Reading Both Ways
Bidirectional Recurrent Networks
Every lecture so far read the sentence the way you hear it — left to right, past only. This lecture adds the second lane: the same sentence read right to left, from the future. Two recurrent lanes over one sequence, concatenated at every position: each word understood in the light of its whole sentence.
THE SAME PROBE SENTENCE, TWO LAMPS PER WORD
“The animal didn't cross the street because it was too tired.” — who is it? On the right, watch a bidirectional layer read it: for every word, two lamps fire — the teal lamp is the forward lane (everything the past knows), the sky lamp is the backward lane (everything the future knows).
At “it”, the sky lamp burns brightest — the answer arrived from the right.
WHY TWO DIRECTIONS? ↓LIVE · TWO LANES PER WORD
y_t = [ h→_t ; h←_t ]teal = past lane · sky = future lane — both lit at every position.
First, the honest motivation: a left-to-right reader is genuinely blind on real sentences — and you can watch the blindness.
02 THE MOTIVATION
One direction is blind. The future settles it.
Same left context, opposite futures — the classic Winograd pair. Pick a sentence, then limit what the reader may see: left only (every RNN so far), right only, or both. Watch the probability bars move.
Convinced the second lane earns its keep? Then here is the machine — every wire of it, before we animate the two readings.
03 THE BLUEPRINT
Anatomy of two lanes
One input at the bottom, forked. Two cells fed from opposite directions of time. One concat at the top — the only place the lanes ever meet. Hover the diagram.
04 STEP BY STEP · BLOCK ARCHITECTURE
Follow the signal, both ways
Start with the full diagram, fully lit — fork, two lanes, two time-sources, concat, output — then tour one word's double reading in data-flow order: fork → from the past → from the future → both fire → the concat. A glowing dot rides the live data path at each step; the dialog underneath unpacks the math, symbol by symbol.
Architecture references: Schuster & Paliwal, "Bidirectional Recurrent Neural Networks" (1997) · Bahdanau et al., "Neural Machine Translation by Jointly Learning to Align and Translate" (2015, BiGRU encoder) · Huang et al., "Bidirectional LSTM-CRF Models for Sequence Tagging" (2015).
05 CELL FLAVORS + THE PRICE OF TWO LANES
One trick, three cells
Bidirectionality is cell-agnostic — mirror any cell you like. Flip between BiRNN, BiGRU and BiLSTM; then price a whole bidirectional stack with the sliders. Watch the fan-in jump to 2h above layer 1: the next layer reads the concat.
Statics understood. Now watch the thing move: a BiLSTM reading two probes, both lanes' memories exposed side by side.
06 LIVE BI-CELL · READING A SENTENCE
Watch the future decide
A BiLSTM reads the series' two probes. Left, the forward lane — what the past knows at each word. Right, the backward lane — what the future knows. Step word by word and watch the backward row solve the pronoun before the forward row can.
FORWARD h→ · knows the past
BACKWARD h← · knows the future
You've seen the behavior in both lanes. Here is the exact machinery — one sign flip, t−1 becomes t+1, and everything else you already know.
07 THE GOVERNING EQUATIONS
One sign flip. Every symbol touchable.
Hover or tap any highlighted symbol — the inspector below explains its job, direction and personality.
READING ORDER
① the two equations are the SAME recurrence mirrored — the only change is the state index: t−1 (past) becomes t+1 (future).
② both lanes read the same x_t; neither ever sees the other's state.
③ the lanes never share weights: W→ compresses the past, W← the future — hence exactly ×2 parameters.
④ the output concatenates — width 2h, never an addition. The next layer reads 2h-wide vectors.
08 ONE WORD, TWO DIRECTIONS, BY THE NUMBERS
Hand-computed. Both lanes.
One-dimensional states, actual weights, one tanh per direction. Step through both lanes and watch y_t = [0.8977 ; 0.4930] emerge — the NumPy tab in §11 reproduces it to the fourth decimal.
Forward pass understood — in both directions. The backward pass mirrors too: error flows left in one lane and right in the other.
09 BPTT, MIRRORED
Gradients also travel both ways
Backpropagation through time runs opposite to each lane's reading direction: the forward lane's error flows toward the past, the backward lane's toward the future. Drag the gated keep-factor and compare the three chains.
Honest footnote: bidirectionality is not a gradient cure — each lane vanishes on its own schedule, which is why the lanes are gated (GRU/LSTM) in practice. What two lanes DO buy: every position receives error from both sides, so the shorter of the two paths to any word is often much shorter than the only path a unidirectional net has.
10 THE VERDICT
Bi vs uni — the honest scoreboard
Direction is decided by one question: does the future exist at read time? Flip through the options, then check the head-to-head table.
| Aspect | Unidirectional → | Bidirectional ⇄ |
|---|---|---|
| Context per position | the past only | past + future — the whole sentence |
| Parameters | 1× | 2× — two weight sets, zero sharing |
| Compute | 1× | 2× — but the two lanes parallelize |
| Output width | h | 2h — the concat [h→ ; h←] |
| Streaming / real-time | ✓ — emits as words arrive | ✗ — needs the full sequence first |
| Text generation | ✓ — the decoder's only option | ✗ — the future doesn't exist yet |
| Gradient paths | time, one direction | time, mirrored — error from both sides |
| Classic use | language modeling, decoders | NER (BiLSTM-CRF), tagging, QA encoders, ELMo |
One-line summary: bidirectionality turns one reader into two readers meeting at every word — double the parameters, double the compute, the whole sentence in every vector — legal only when the future already exists.
11 CODE LAB
Bidirectional: one flag, one wrapper, or from scratch
PyTorch hides the second lane behind bidirectional=True; Keras wraps any recurrent layer in Bidirectional; the NumPy tab shows what both are actually doing — and reproduces §08 to the fourth decimal.
12 SUMMARY
What to carry out of this room
Run two independent recurrent layers over the same sequence in opposite directions; concatenate their states at every position. Each word is understood in the light of its whole sentence.
x_t forks to both lanes unchanged. h→ flows left→right from the past; h← flows right→left from the future. The lanes meet only at the concat — never in the weights, never in the state.
Ambiguity resolution: pronouns (Winograd), word senses, entity boundaries. “It was too tired” is unsolvable left-to-right and trivial right-to-left.
Exactly ×2 parameters and ×2 compute per layer, a 2h-wide output that compounds when stacked — and a hard requirement: the full sequence up front. No streaming, no generation.
BiRNN (Schuster & Paliwal, 1997), BiGRU (Bahdanau's attention encoder, 2015), BiLSTM (BiLSTM-CRF for NER; ELMo, 2018). One trick, any cell.
Encoders go bidirectional; decoders stay unidirectional. The rule outlived the RNN: BERT reads both ways, GPT reads one. Direction is about whether the future exists — not about the architecture.
13 MASTERY CHECK