CASDeep Learning Series

Lecture 03, Foundations part 3

Gradient descent, one weight at a time

This is the gradient of the weight furthest from the loss in a 3-layer network. It looks hard. By the end of this lecture you will write it yourself, explain every factor, and shrink it to one short line.

  • Loss
  • Layer 3
  • Layer 2
  • Layer 1

It shrinks to

Part 1

Why we subtract the slope

A neural network is a machine with knobs. The knobs are its weights and biases. The loss L is one number that says how wrong the machine is. Training means turning the knobs until that number is small.

Take one knob w. The derivative \dfrac{\partial L}{\partial w} tells us what happens to the loss when we move that knob a tiny amount:

\Delta L \approx \frac{\partial L}{\partial w}\,\Delta w

We choose \Delta w, and we want \Delta L to be negative. Here is the clever choice. Make the step equal to the slope with its sign flipped, scaled by a small positive number \eta, the learning rate:

\Delta w = -\eta\,\frac{\partial L}{\partial w}

Put that step into the first line:

\Delta L \approx \frac{\partial L}{\partial w}\left(-\eta\,\frac{\partial L}{\partial w}\right) = -\eta\left(\frac{\partial L}{\partial w}\right)^{2}

A square is never negative and \eta is positive, so the right side is never positive. The minus sign is the reason the loss goes down. It works whether the slope is positive or negative, and that gives the only rule in this lecture:

w_{\text{new}} = w_{\text{old}} - \eta\,\frac{\partial L}{\partial w}

The word approximately matters. The first line only holds for small steps, so a large \eta can break the promise. Try it on the right.

Roll down a bowl Loss L(w)=\tfrac12(w-2)^2, so the slope is L'(w)=w-2

Part 2

Meet the network

We use the smallest network that still shows everything: 2 inputs, a hidden layer of 2 neurons, a second hidden layer of 2 neurons, and 1 output. We count layers that have weights, so this is a 3-layer network.

Lectures 01 and 02 used a 4–3–2–1 network with 26 parameters. This one has 15, few enough to derive every single one here. Nothing is lost: a weight in layer 1 still reaches the loss through more than one path, and that is the hard case.

LayerGoes fromWeightsBiases
Layer 1Input to Hidden 142
Layer 2Hidden 1 to Hidden 242
Layer 3Hidden 2 to Output21

How to read the symbols

w_{jk}^{(l)}
  • (l) is the layer the weight belongs to.
  • j is the neuron the wire goes to.
  • k is the neuron the wire comes from.

So w_{21}^{(2)} is the layer 2 wire going to neuron 2 from neuron 1. Each neuron does two small jobs. It adds up its inputs into a sum z, then passes that sum through an activation function f to give its output a. We use the simplest activation there is, the linear one, where the output is just the sum:

z_j^{(l)} = \sum_{k} w_{jk}^{(l)}\,a_k^{(l-1)} + b_j^{(l)}, \qquad a_j^{(l)} = f\!\left(z_j^{(l)}\right) = z_j^{(l)}

Two naming shortcuts: the inputs are a_k^{(0)} = x_k, and the output is a_1^{(3)} = \hat y. The loss for one example is

L = \tfrac12\,(\hat y - y)^2

The \tfrac12 is there only to cancel a 2 when we differentiate.

Why linear? It removes the curved part of the math, so you see the chain rule itself, and the numbers stay simple enough to check by hand. We still write the step a = f(z), so nothing has to be relearned for other activations. One honest limit: stacked linear layers can only ever draw a straight-line relationship, so real networks use a non-linear activation such as sigmoid or ReLU. Part 12 shows the single change that needs.

Part 3

Forward pass: get the numbers first

Gradients are built from values the forward pass produces, so we run it first and keep every a. Follow the network from left to right. These numbers are small on purpose: check them with a pencil.

The prediction is off, and the loss measures by how much. Now we ask each of the 15 knobs one question: if I turn you, what happens to L?

Part 4

The only five derivatives you need

The network is a chain of tiny steps. Each step has its own easy derivative. Every long formula later is just these five, multiplied.

  1. Loss with respect to the prediction

    The power rule brings down the 2, and it cancels the \tfrac12. If the prediction is too high, this is positive.

  2. Activation with respect to its sum

    With a linear activation, any change in z passes straight through to a, so this factor is 1. We keep writing it in every chain so you can see where it sits. This is the one factor that changes if you swap the activation.

  3. Sum with respect to a weight

    In z, only one term contains this weight, and it is multiplied by the incoming activation. So a weight learns in proportion to the signal on its wire.

  4. Sum with respect to a bias

    The bias is simply added, so its slope is 1. A bias behaves like a weight whose input is always 1.

  5. Sum with respect to the previous activation

    This one lets us cross backwards over a wire. The wire's weight is the exchange rate between the two neurons.

Part 5

Layer 3: three factors

Start next to the loss. Changing w_{1k}^{(3)} changes the sum, the sum changes the prediction, and the prediction changes the loss:

Step 1: write the chain

One factor for each arrow:

Step 2: replace each factor

Each one is a derivative from Part 4. The middle factor is the linear slope, 1:

Step 3: give the repeated part a name

The first two factors do not depend on which weight we picked. Every parameter in this layer shares them, so we name them \delta^{(3)}, the error signal of the output neuron. It means \partial L/\partial z_1^{(3)}: how much the loss cares about this neuron's sum. Here it is simply the prediction error.

Step 4: put in the numbers

Read the result. Gradient equals the error signal at the neuron in front of the wire, times the activation behind the wire. Every layer ends in this same shape.

Part 6

Layer 2: five factors

Go one layer back. Now the weight has further to travel. It changes z_j^{(2)}, which changes a_j^{(2)}, and only then does it enter the Layer 3 sum:

Step 1: write the chain

Step 2: replace each factor

Step 3: spot what we already know

Look at the first two factors. That is \delta^{(3)} from Part 5, so we reuse it. This reuse is what makes backpropagation fast: the calculation for a layer starts from the answer for the layer after it.

The bracket is the error signal of neuron j in Hidden 2:

In words: take the error ahead and send it back across the wire. The wire's weight decides how much of the error arrives here.

Step 4: put in the numbers

Part 7

Layer 1: two paths, seven factors each

Here is the new idea. A neuron in Hidden 1 feeds both neurons in Hidden 2. If a_j^{(1)} changes, the effect reaches the loss by two separate roads.

When a change travels by several roads, the total effect is the sum of the roads. This is the multivariable chain rule: multiply along a path, add across paths.

Step 1: write the complete chain

That is the equation from the top of this page, now for any j and k. Each path has seven factors, one for each step.

Step 2: replace the first four factors of each path

In each path, the first four factors are exactly \delta_m^{(2)} from Part 6:

Step 3: pull out what both paths share

The last two factors are the same in both paths, so they move outside:

Step 4: replace the small derivatives

The bracket is the error signal of neuron j in Hidden 1:

Seven-factor paths have become a single product, with the same shape as Layer 3.

Step 5: put in the numbers

Look at the signs. The two paths can pull in opposite directions. A negative weight flips the sign of the error passing through it, so one path may say "go up" while the other says "go down". That is why we must add every path: the sum is the true effect on the loss.

Part 8

Pick any parameter and watch its derivation

Choose a parameter, then step through its full derivation line by line. The lit wires show every path it takes to the loss.

    Part 9

    All 15 update rules

    Here is every parameter in the network. Each one has its complete chain rule, its simplified form, and its real update for one step with learning rate .

    Part 10

    Did it work? Two independent checks

    Check 1: is every gradient right?

    We can estimate a gradient with no calculus at all. Nudge a parameter up and down by a tiny \varepsilon, run the forward pass twice, and measure the slope:

    \frac{\partial L}{\partial \theta} \approx \frac{L(\theta+\varepsilon)-L(\theta-\varepsilon)}{2\varepsilon}, \qquad \varepsilon = 10^{-5}

    Here \theta stands for any one parameter. The two gradient columns below come from different methods, and they should agree.

    Check 2: did the loss go down?

    All 15 parameters moved at the same moment, using gradients from the old weights. Part 1 predicted the change. With many parameters, the squares simply add up:

    The actual change differs from the prediction because the prediction treats the loss as a straight line near the current weights, and it is really a curve. Lower \eta with the slider in Part 2 and the two numbers move closer together.

    Part 11

    Now repeat it

    One step is a small move. Training is the same step repeated: forward pass, backward pass, update. Run it and watch the wires change. Then compare learning rates: try 0.1, then 0.5, then 0.85. A small rate slides down, a larger one bounces before it settles, and one that is too large blows up.

    Loss after each step
    Step
    Prediction \hat y
    Loss L

    Part 12

    The whole lecture in five lines

    Stack the neurons of a layer into vectors and everything we derived collapses. These five lines work for a network of any size. The symbol \odot means multiply element by element.

    See it once with this network's numbers. For Layer 2, a column of error signals times a row of activations makes the whole gradient matrix, with each entry in the same place as its weight:

    The same five lines in NumPy

    def train_step(x, y, W, b, eta):
        # forward: keep every activation
        a = [x]
        for Wl, bl in zip(W, b):
            a.append(Wl @ a[-1] + bl)          # linear: a = z
    
        # backward: start at the output, reuse delta layer by layer
        delta = a[-1] - y
        gW, gb = [None] * 3, [None] * 3
        for l in (2, 1, 0):
            gW[l] = np.outer(delta, a[l])
            gb[l] = delta
            if l > 0:
                delta = W[l].T @ delta
    
        # update: only after all gradients exist
        for l in range(3):
            W[l] -= eta * gW[l]
            b[l] -= eta * gb[l]

    Three mistakes to avoid

    • Updating too early. If you change W^{(3)} before computing \delta^{(2)}, the gradient uses the wrong weight. Compute everything first, then update everything.
    • Thinking the 1 is always there. \partial a/\partial z = 1 is true only for a linear activation. For any other activation, that factor is a real number you must include.
    • Forgetting a path. A hidden neuron with two outgoing wires gets two terms in its bracket. Multiply along a path, add across paths.

    Moving to a real activation

    Every derivation in this lecture stays the same. Only the factor f'(z) changes. Where we wrote 1, a sigmoid network puts a(1-a), as you saw in Lecture 02, and a ReLU network puts 1 when z>0 and 0 otherwise. In the code, multiply delta by that slope each time it moves back a layer.

    Beyond one example

    We trained on a single example to keep the math visible. With a dataset, compute these gradients for each example in a batch and average them before updating. If you drop the \tfrac12 from the loss, \partial L/\partial\hat y becomes 2(\hat y-y), and the rest stays the same.

    Test yourself

    How many paths lead from w_{12}^{(1)} to the loss, and how many factors does each have?

    Two paths, one through each Hidden 2 neuron. Each has seven factors. The gradient is the sum of the two products.

    A gradient is -0.04 and \eta = 0.5. Does the weight go up or down, and by how much?

    Up by 0.02, since w_{\text{new}} = w - 0.5(-0.04) = w + 0.02. A negative slope means raising the weight lowers the loss.

    An input x_1 is exactly 0. What happens to w_{11}^{(1)} in this step?

    Nothing. Its gradient is \delta_1^{(1)}x_1 = 0. A wire carrying no signal cannot be blamed for the error. Set x_1 to 0 with the slider and check the numbers in Part 9.

    Why is the bias gradient always equal to \delta?

    Because \partial z/\partial b = 1. The chain ends with a factor of 1, so only the error signal is left.

    Now scroll back to the top. That equation is two paths of seven small derivatives, and you know every one of them.

    Back to the equation