Why
In , the second and third terms behave like constant numbers when differentiating wrt . Constants have slope 0. Only responds!
Foundations · Part 4 · Chain-Rule Gradient Math
Final gradient formulas only stick when you see every intermediate derivative resolved without skipping steps. Build physical intuition for partial derivatives first, then derive all 26 weights and biases of a 4 → 3 → 2 → 1 linear network down to a single unified rule.
Part 0 · Before Any Algebra
Calculus symbols look scary until you translate each derivative into one physical question: “If I freeze every other knob on the machine and nudge just this one knob by a tiny amount, how much does the output wiggle?”
Everyday Analogy — A Recording Studio Mixer & Connected Gears
Imagine a mixing board with 26 volume sliders. You tape down 25 sliders so nothing else moves, then slide knob up by 0.1. If your input signal is equal to 1.4 volts, turning that gain knob up by 0.1 increases the sum by 0.1 × 1.4 = 0.14. The ratio output change ÷ knob change is literally the signal sitting on that wire: . And if that sum then drives a chain of three connected gears, the end-to-end sensitivity is simply Gear 1 ratio × Gear 2 ratio × Gear 3 ratio.
Careful Index Rule: Throughout this page, denotes the weight going from neuron i (in layer ) to neuron j (in layer ). So neuron j in layer l computes . Count the trainable knobs: Layer 1 has 4×3 weights + 3 biases = 15; Layer 2 has 3×2 weights + 2 biases = 8; Layer 3 has 2×1 weights + 1 bias = 3 — 26 parameters in total.
In , the second and third terms behave like constant numbers when differentiating wrt . Constants have slope 0. Only responds!
The in exists solely to cancel the exponent 2 when you apply the power rule! Notice the tangent slope equals signed error : positive when , negative when .
Two derivatives simplify to 1 throughout this entire lecture—never confuse why they happen:
Because the bias has no input wire multiplying it, its “effective input” is always the fixed constant 1.
Changing cannot touch directly—it must travel through three relay stages. Multiply all local magnification factors along the route:
Section 1 · The Dependency Pattern
Suppose we want the derivative of loss with respect to an output-layer weight:
The weight does not directly produce the loss. It changes the pre-activation sum , that sum determines output activation , and only touches the loss . The forward dependency chain is:
To find total sensitivity, walk backwards along the arrows and multiply each adjacent link:
This exact domino structure is the pattern we will reuse for every single weight in the network—even deep inside Layer 1.
Section 2 · Layer 3 (3 Parameters)
Our single output neuron receives signals from the two neurons of Hidden Layer 2 plus its own bias:
Using the power rule and inner chain factor 1:
Since is linear, shifting shifts 1-for-1:
Treat and as frozen constants (derivative = 0):
Applying the exact same 3-factor chain to and :
Section 3 · Layer 2 (8 Parameters)
Move one layer backward. Hidden Layer 2 has two neurons, each fed by all three neurons of Hidden Layer 1:
Consider . Notice that connects only to the single output neuron , giving one 5-step path:
Now resolve every single term from right-hand output back to the weight—no shortcuts:
| Derivative Link | Result | Why? |
|---|---|---|
| Loss slope computed at the output | ||
| Linear output activation () | ||
| Coefficient multiplying inside | ||
| Linear activation of Hidden Layer 2 neuron 1 | ||
| Incoming input signal feeding weight |
Take (connecting Layer 1 neuron 1 to Layer 2 neuron 2). Its signal flows out through , whose bridge weight to is instead of :
Section 4 · Layer 1 (15 Parameters)
Physical Intuition — One Warehouse, Two Delivery Trucks
Hidden Layer 1 neuron 1 () sends copies of its output to BOTH neurons in Layer 2 ( and ). Think of nudging as loading +1 extra crate onto your factory belt: Truck 1 carries part of that change through Layer 2 neuron 1 to the loss, while Truck 2 simultaneously carries another part through Layer 2 neuron 2. Total impact on the loss is Truck 1 impact + Truck 2 impact!
Focus on . Look at the first neuron equation and follow its forward split:
By the multivariable chain rule, we compute :
Section 5 · Why This Compresses Into One Rule
Once we know how to resolve individual chain rules by hand, define the local error signal to reuse downstream path sums automatically across every neuron and every layer.
Start at the end: the single output neuron’s δ equals .
For linear activations (), each neuron sums downstream δ signals weighted by its outgoing connections.
Every parameter gradient in all 3 layers is just incoming activation × target neuron’s δ.
Output Layer 3 (2 Weights + 1 Bias)
Hidden Layer 2 (6 Weights + 2 Biases)
Hidden Layer 1 (12 Weights + 3 Biases)
Section 6 · Check Your Understanding
Questions 1 to 3 test core differentiation intuition directly; Questions 4 to 7 verify indices, multi-path cancellation, and bias behavior.
If you are teaching or revising this at a whiteboard, avoid expanding all 26 derivatives separately. Derive the 3 anchor examples carefully without skipping steps— (output 3-factor chain), (hidden Layer 2 single-route 5-factor chain), and (hidden Layer 1 two-path split)—and then introduce so students watch all 26 formulas collapse into one general backpropagation rule.