CASCenter for Advanced Studies · AI Engineering Program

◆ DEEP DIVE TUTORIAL · RNN · NUMERICAL WALKTHROUGH

RNN Forward Pass & Backprop by Hand

A complete numerical example: the sentence "was not bad" enters as one-hot vectors, and we compute every single value — forward and backward — with real numbers you can verify on a calculator.

Forward pass: 3 timesteps, every a_t and h_t with 4-decimal precision

Softmax output & cross-entropy loss with exact numbers

Backprop: δ_t at every timestep with intermediate products shown

Total gradients for all 5 parameter tensors, summed across time

Gradient descent update with η = 0.1 — the new weights

No black boxes. Every number below was computed by a Python script — you can reproduce every step.

01 THE EXAMPLE

Sentence: "was not bad"

Three words. Target: POSITIVE (because "not bad" is good). The model must read one word at a time and build a hidden memory.

VOCABULARY · 3 WORDS

Index	Word	One-hot
0	was	[1, 0, 0]
1	not	[0, 1, 0]
2	bad	[0, 0, 1]

Vocab size |V| = 3. Hidden dim H = 2. Classes C = 2 (negative, positive).

Initial Parameters (the weights we will learn from)

W_xh (2 × 3)

[[ 0.2, 0.1, -0.1],
[-0.1, 0.2, 0.1]]

W_hh (2 × 2)

[[ 0.3, 0.2],
[-0.1, 0.3]]

W_hy (2 × 2)

[[ 0.2, -0.1],
[ 0.1, 0.3]]

Biases

b_h = [ 0.1, -0.1]
b_y = [ 0.0, 0.0]

Parameters are set. Now we process each word — one timestep at a time, with every number shown.

02 FORWARD PASS

Every Number, Every Step

We initialize h₋₁ = [0, 0]. At each timestep: a_t = W_xh·x_t + W_hh·h_t−1 + b_h, then h_t = tanh(a_t).

◈ LIVE DEMO

Watch the network think

Step through the forward pass one timestep at a time. See values flow through weights, hidden states update, and the final prediction emerge. Every number is real — from the example above.

Step 1 of 4 — Initialization

FOLDED RNN CELL

INPUT x_t

[0, 0, 0]

—

W_xh

PRE-ACTIVATION a_t

[0.0000, 0.0000]

tanh

HIDDEN h_t

[0.0000, 0.0000]

W_hh

+ b_h

W_hy

LOGIT z

[0.0000, 0.0000]

softmax

OUTPUT ŷ

[0.0000, 0.0000]

—

Computation at this step

Press Next to begin the forward pass.

Current Weight Values

W_xh

0.20	0.10	−0.10
−0.10	0.20	0.10

W_hh

0.30	0.20
−0.10	0.30

b_h

0.10

−0.10

W_hy

0.20	−0.10
0.10	0.30

b_y

0.00

TIMESTEP 0

Word: "was" · x₀ = [1, 0, 0]

Start with the zero hidden state. The one-hot x₀ selects column 0 of W_xh.

a₀ = W_xh·x₀ + W_hh·h₋₁ + b_h a₀ = [0.2, −0.1] + [0.0, 0.0] + [0.1, −0.1] a₀ = [0.3000, −0.2000]

Apply tanh to squash into (−1, 1):

h₀ = tanh(a₀) = [tanh(0.3000), tanh(−0.2000)] h₀ = [0.2913, −0.1974]

TIMESTEP 1

Word: "not" · x₁ = [0, 1, 0]

Now the recurrence kicks in. We multiply W_hh by the previous hidden state h₀.

W_hh·h₀ = [0.3·0.2913 + 0.2·(−0.1974), −0.1·0.2913 + 0.3·(−0.1974)] = [0.0874 − 0.0395, −0.0291 − 0.0592] = [0.0479, −0.0883]

Add the input contribution (column 1 of W_xh) and bias:

a₁ = [0.1, 0.2] + [0.0479, −0.0883] + [0.1, −0.1] a₁ = [0.2479, 0.0117]

Apply tanh:

h₁ = tanh(a₁) = [tanh(0.2479), tanh(0.0117)] h₁ = [0.2430, 0.0117]

TIMESTEP 2

Word: "bad" · x₂ = [0, 0, 1]

Same computation — W_hh·h₁, plus column 2 of W_xh, plus bias.

W_hh·h₁ = [0.3·0.2430 + 0.2·0.0117, −0.1·0.2430 + 0.3·0.0117] = [0.0729 + 0.0023, −0.0243 + 0.0035] = [0.0752, −0.0208]

a₂ = [−0.1, 0.1] + [0.0752, −0.0208] + [0.1, −0.1] a₂ = [0.0752, −0.0208]

h₂ = tanh(a₂) = [tanh(0.0752), tanh(−0.0208)] h₂ = [0.0751, −0.0208]

OUTPUT

Softmax Prediction

Use the final hidden state h₂ to compute logits z, then softmax.

z = W_hy·h₂ + b_y z = [0.2·0.0751 + (−0.1)·(−0.0208), 0.1·0.0751 + 0.3·(−0.0208)] = [0.0150 + 0.0021, 0.0075 − 0.0062] = [0.0171, 0.0013]

Softmax: exponentiate, sum, divide.

e^z₀ = e^0.0171 = 1.0172 e^z₁ = e^0.0013 = 1.0013 Σe^zᵢ = 1.0172 + 1.0013 = 2.0185 ŷ₀ = 1.0172 / 2.0185 = 0.5040 (negative) ŷ₁ = 1.0013 / 2.0185 = 0.4960 (positive)

Prediction: NEGATIVE (50.4% confidence) — the model is wrong! True label is POSITIVE.

The model predicted negative but the true label is positive. Time to compute the loss and blame the right weights.

03 LOSS

Cross-Entropy with Real Numbers

LOSS

How surprised is the model?

True label y = [0, 1] (positive). Model output ŷ = [0.5040, 0.4960].

L = − Σ yᵢ · log(ŷᵢ) L = − ( 0 · log(0.5040) + 1 · log(0.4960) ) L = − log(0.4960) = 0.7011

A perfect confident prediction (ŷ₁ = 0.9999) would give L ≈ 0.0001. Our model is uncertain and wrong — the loss is high.

Loss = 0.7011. Now we compute which weights are responsible — by propagating gradients backward through every timestep.

04 BACKPROPAGATION

Every Gradient, Every Product

We compute ∂L/∂W_xh, ∂L/∂W_hh, ∂L/∂W_hy, and both biases. Starting from the output and walking back to t=0.

STEP A

Output Error Signal

For softmax + cross-entropy, the gradient simplifies to ŷ − y.

∂L/∂z = ŷ − y ∂L/∂z = [0.5040, 0.4960] − [0, 1] ∂L/∂z = [0.5040, −0.5040]

STEP B

Gradients for W_hy and b_y

∂L/∂W_hy = (ŷ−y) · h₂ᵀ = [[0.5040·0.0751, 0.5040·(−0.0208)], [−0.5040·0.0751, −0.5040·(−0.0208)]] = [[0.0378, −0.0105], [−0.0378, 0.0105]]

∂L/∂b_y = [0.5040, −0.5040]

STEP C

Initial Hidden Gradient δ₂

δ₂ = W_hyᵀ · (ŷ−y) = [0.2·0.5040 + 0.1·(−0.5040), −0.1·0.5040 + 0.3·(−0.5040)] = [0.1008 − 0.0504, −0.0504 − 0.1512] = [0.0504, −0.2016]

STEP D

Gradients at t = 2 (word "bad")

First compute the tanh derivative at a₂: (1 − tanh²).

1 − tanh²(a₂) = [1 − tanh²(0.0752), 1 − tanh²(−0.0208)] = [0.9944, 0.9996]

Multiply δ₂ element-wise with the tanh derivative:

g₂ = δ₂ ⊙ (1−tanh²) = [0.0504·0.9944, −0.2016·0.9996] = [0.0501, −0.2015]

Now accumulate gradients. For W_xh, g₂ multiplies with x₂ᵀ = [0, 0, 1] — only column 2 survives:

∂L/∂W_xh at t=2 = [[0, 0, 0.0501], [0, 0, −0.2015]]

∂L/∂W_hh at t=2 = g₂ · h₁ᵀ = [[0.0122, 0.0006], [−0.0490, −0.0023]]

∂L/∂b_h at t=2 = [0.0501, −0.2015]

STEP E

Backprop to t = 1 (word "not")

Propagate the gradient backward through W_hhᵀ:

δ₁ = W_hhᵀ · g₂ = [0.3·0.0501 + (−0.1)·(−0.2015), 0.2·0.0501 + 0.3·(−0.2015)] = [0.0150 + 0.0201, 0.0100 − 0.0604] = [0.0352, −0.0504]

Apply tanh derivative at a₁:

1 − tanh²(a₁) = [0.9410, 0.9999] g₁ = δ₁ ⊙ (1−tanh²) = [0.0331, −0.0504]

∂L/∂W_xh at t=1 = [[0, 0.0331, 0], [0, −0.0504, 0]]

∂L/∂W_hh at t=1 = g₁ · h₀ᵀ = [[0.0096, −0.0065], [−0.0147, 0.0100]]

∂L/∂b_h at t=1 = [0.0331, −0.0504]

STEP F

Backprop to t = 0 (word "was")

δ₀ = W_hhᵀ · g₁ = [0.3·0.0331 + (−0.1)·(−0.0504), 0.2·0.0331 + 0.3·(−0.0504)] = [0.0099 + 0.0050, 0.0066 − 0.0151] = [0.0150, −0.0085]

1 − tanh²(a₀) = [0.9151, 0.9610] g₀ = δ₀ ⊙ (1−tanh²) = [0.0137, −0.0082]

∂L/∂W_xh at t=0 = [[0.0137, 0, 0], [−0.0082, 0, 0]]

∂L/∂W_hh at t=0 = [[0, 0], [0, 0]] (because h₋₁ = 0)

∂L/∂b_h at t=0 = [0.0137, −0.0082]

TOTALS

Sum Gradients Across All Timesteps

Add the contributions from t=0, t=1, and t=2 for each parameter.

∂L/∂W_xh = [[0.0137+0+0, 0+0.0331+0, 0+0+0.0501], [−0.0082+0+0, 0−0.0504+0, 0+0−0.2015]] = [[0.0137, 0.0331, 0.0501], [−0.0082, −0.0504, −0.2015]]

∂L/∂W_hh = [[0+0.0096+0.0122, 0−0.0065+0.0006], [0−0.0147−0.0490, 0+0.0100−0.0023]] = [[0.0218, −0.0060], [−0.0636, 0.0076]]

∂L/∂b_h = [0.0137+0.0331+0.0501, −0.0082−0.0504−0.2015] = [0.0969, −0.2601]

Gradients computed. One final step: apply gradient descent and see the new weights.

05 PARAMETER UPDATE

Gradient Descent: New Weights

Learning rate η = 0.1. We subtract 10% of each gradient from its parameter.

W_xh BEFORE

[[ 0.2, 0.1, −0.1],
[−0.1, 0.2, 0.1]]

W_xh AFTER (new)

[[0.1986, 0.0967, −0.1050],
[−0.0992, 0.2050, 0.1202]]

W_hh BEFORE

[[ 0.3, 0.2],
[−0.1, 0.3]]

W_hh AFTER (new)

[[0.2978, 0.2006],
[−0.0936, 0.2992]]

W_hy BEFORE

[[ 0.2, −0.1],
[ 0.1, 0.3]]

W_hy AFTER (new)

[[0.1962, −0.0989],
[0.1038, 0.2989]]

BIASES BEFORE

b_h = [ 0.1, −0.1]
b_y = [ 0.0, 0.0]

BIASES AFTER (new)

b_h = [0.0903, −0.0740]
b_y = [−0.0504, 0.0504]

OBSERVATION

Notice W_hy[1,0] increased from 0.1 → 0.1038, and W_xh[1,2] (the weight for "bad"→hidden[1]) increased from 0.1 → 0.1202. The model is learning to boost the positive signal. After many iterations, it would correctly predict POSITIVE.

The complete loop: forward → loss → backprop → update. Let us test your understanding with a quick quiz.

06 KNOWLEDGE CHECK

Verify Your Understanding

Instant feedback — no grades, just clarity. Every answer references the actual numbers above.

RNN Forward Pass & Backprop by Hand

Sentence: "was not bad"

Initial Parameters (the weights we will learn from)

Every Number, Every Step

Watch the network think

Word: "was" · x₀ = [1, 0, 0]

Word: "not" · x₁ = [0, 1, 0]

Word: "bad" · x₂ = [0, 0, 1]

Softmax Prediction

Cross-Entropy with Real Numbers

How surprised is the model?

Every Gradient, Every Product

Output Error Signal

Gradients for W_hy and b_y

Initial Hidden Gradient δ₂

Gradients at t = 2 (word "bad")

Backprop to t = 1 (word "not")

Backprop to t = 0 (word "was")

Sum Gradients Across All Timesteps

Gradient Descent: New Weights

Verify Your Understanding

What is h₀ after reading "was"?

What was the model's prediction for "was not bad"?

What is the cross-entropy loss L?

Why is ∂L/∂W_hh at t=0 equal to [[0,0],[0,0]]?

After the update, what happened to b_h[1]?

RNN Forward Pass & Backprop by Hand

Initial Parameters (the weights we will learn from)

Word: "was" · x₀ = [1, 0, 0]

Word: "not" · x₁ = [0, 1, 0]

Word: "bad" · x₂ = [0, 0, 1]

Softmax Prediction

How surprised is the model?

Output Error Signal

Gradients for Why and by

Initial Hidden Gradient δ₂

Gradients at t = 2 (word "bad")

Backprop to t = 1 (word "not")

Backprop to t = 0 (word "was")

Sum Gradients Across All Timesteps

What is h₀ after reading "was"?

What was the model's prediction for "was not bad"?

What is the cross-entropy loss L?

Why is ∂L/∂Whh at t=0 equal to [[0,0],[0,0]]?

After the update, what happened to bh[1]?

Gradients for W_hy and b_y

Why is ∂L/∂W_hh at t=0 equal to [[0,0],[0,0]]?

After the update, what happened to b_h[1]?