LECTURE 04 · FOUNDATIONS · 4 → 3 → 2 → 1 LINEAR NETWORK · 26 EXACT DERIVATIVES

Foundations · Part 4 · Chain-Rule Gradient Math

How Every Derivative Is Resolved Using the Chain Rule

Final gradient formulas only stick when you see every intermediate derivative resolved without skipping steps. Build physical intuition for partial derivatives first, then derive all 26 weights and biases of a 4 → 3 → 2 → 1 linear network down to a single unified rule.

Architecture: 4 → 3 → 2 → 1 Activation: a_j^(l) = z_j^(l) Loss: L = 1/2 (ŷ − y)^2 Convention: W_ij^(l) means i → j

Part 0 · Before Any Algebra

What Does a Partial Derivative Actually Mean?

Calculus symbols look scary until you translate each derivative into one physical question: “If I freeze every other knob on the machine and nudge just this one knob by a tiny amount, how much does the output wiggle?”

Everyday Analogy — A Recording Studio Mixer & Connected Gears

Imagine a mixing board with 26 volume sliders. You tape down 25 sliders so nothing else moves, then slide knob up by 0.1. If your input signal is equal to 1.4 volts, turning that gain knob up by 0.1 increases the sum by 0.1 × 1.4 = 0.14. The ratio output change ÷ knob change is literally the signal sitting on that wire: . And if that sum then drives a chain of three connected gears, the end-to-end sensitivity is simply Gear 1 ratio × Gear 2 ratio × Gear 3 ratio.

Exact Network Setup (Used Throughout This Lecture)
Architecture, Linear Activation, Loss, and Index Rule

Careful Index Rule: Throughout this page, denotes the weight going from neuron i (in layer ) to neuron j (in layer ). So neuron j in layer l computes . Count the trainable knobs: Layer 1 has 4×3 weights + 3 biases = 15; Layer 2 has 3×2 weights + 2 biases = 8; Layer 3 has 2×1 weights + 1 bias = 3 — 26 parameters in total.

Intuition Lab 1 · Frozen Constants

Why

In , the second and third terms behave like constant numbers when differentiating wrt . Constants have slope 0. Only responds!

0.80
1.40
Nudge W by +0.10 to observe Δz ÷ ΔW.
Intuition Lab 2 · Error Slope

Why

The in exists solely to cancel the exponent 2 when you apply the power rule! Notice the tangent slope equals signed error : positive when , negative when .

1.40
1.00
L = 0.080 | Slope dL/da = e = +0.400
Intuition Lab 3 · Two “Free Pass” Derivatives of 1

Why Both Bias and Linear Activation Give 1

Two derivatives simplify to 1 throughout this entire lecture—never confuse why they happen:

1) Linear Activation Derivative = 1
2) Bias Derivative = 1

Because the bias has no input wire multiplying it, its “effective input” is always the fixed constant 1.

Intuition Lab 4 · Why We Multiply

The Chain Rule as Connected Stages

Changing cannot touch directly—it must travel through three relay stages. Multiply all local magnification factors along the route:

Exact derivative vs tiny-step test will appear here.

Section 1 · The Dependency Pattern

First Understand the Chain Rule

Suppose we want the derivative of loss with respect to an output-layer weight:

The weight does not directly produce the loss. It changes the pre-activation sum , that sum determines output activation , and only touches the loss . The forward dependency chain is:

Dependency Chain

To find total sensitivity, walk backwards along the arrows and multiply each adjacent link:

Fundamental Chain-Rule Pattern

This exact domino structure is the pattern we will reuse for every single weight in the network—even deep inside Layer 1.

Section 2 · Layer 3 (3 Parameters)

Output Layer: Resolve Every Derivative

Our single output neuron receives signals from the two neurons of Hidden Layer 2 plus its own bias:

Step 1 — Differentiate Loss wrt Output Activation

Using the power rule and inner chain factor 1:

Step 2 — Differentiate Activation wrt Pre-Activation

Since is linear, shifting shifts 1-for-1:

Step 3 — Differentiate Output Sum wrt First Weight

Treat and as frozen constants (derivative = 0):

Step 4 — Multiply All Three Derivatives

Repeat for the Second Output Weight and the Output Bias

Applying the exact same 3-factor chain to and :

Pattern Alert: The partial derivative of any weighted sum with respect to one of its weights is simply the incoming signal connected to that weight. With respect to its bias , the partial derivative is always 1.
Live Output Layer Cards (Click to Inspect on Diagram)

Section 3 · Layer 2 (8 Parameters)

Hidden Layer 2: The Important Chain-Rule Expansion

Move one layer backward. Hidden Layer 2 has two neurons, each fed by all three neurons of Hidden Layer 1:

Consider . Notice that connects only to the single output neuron , giving one 5-step path:

5-Link Dependency Path
Full 5-Factor Chain Rule Expansion

Now resolve every single term from right-hand output back to the weight—no shortcuts:

Derivative Link Result Why?
Loss slope computed at the output
Linear output activation ()
Coefficient multiplying inside
Linear activation of Hidden Layer 2 neuron 1
Incoming input signal feeding weight
Multiply All 5 Factors

Why Does Neuron 2 Use Instead?

Take (connecting Layer 1 neuron 1 to Layer 2 neuron 2). Its signal flows out through , whose bridge weight to is instead of :

All 8 Layer-2 Gradients (6 Weights + 2 Biases — Click Any Card to Inspect)

Section 4 · Layer 1 (15 Parameters)

Hidden Layer 1: Why We Add Two Paths

Physical Intuition — One Warehouse, Two Delivery Trucks

Hidden Layer 1 neuron 1 () sends copies of its output to BOTH neurons in Layer 2 ( and ). Think of nudging as loading +1 extra crate onto your factory belt: Truck 1 carries part of that change through Layer 2 neuron 1 to the loss, while Truck 2 simultaneously carries another part through Layer 2 neuron 2. Total impact on the loss is Truck 1 impact + Truck 2 impact!

Focus on . Look at the first neuron equation and follow its forward split:

Interactive Multivariable Chain Rule — Branching Path Visualizer

Path 1 (via Layer 2 Neuron 1)

Path 2 (via Layer 2 Neuron 2)

Total gradient = P1 + P2

Write Out Both 7-Factor Paths Without Skipping Any Step

By the multivariable chain rule, we compute :

Path 1 — Route W_11^(1) → z_1^(1) → a_1^(1) → z_1^(2) → a_1^(2) → z_1^(3) → a_1^(3) → L
Path 2 — Route W_11^(1) → z_1^(1) → a_1^(1) → z_2^(2) → a_2^(2) → z_1^(3) → a_1^(3) → L
Add Path 1 + Path 2 and Factor Out Common e and x_1
Watch What Changes vs What Stays Fixed: Notice how every weight entering Layer 1 neuron () from input () shares the exact same downstream multiplier bracket for neuron , and swaps only the input factor (or for the bias)!
All 15 Layer-1 Gradients (12 Weights + 3 Biases — Click Any Card to Inspect)

Section 5 · Why This Compresses Into One Rule

The General Backpropagation Rule & 26-Parameter Explorer

Once we know how to resolve individual chain rules by hand, define the local error signal to reuse downstream path sums automatically across every neuron and every layer.

Rule 1 · Output Error

Start at the end: the single output neuron’s δ equals .

Rule 2 · Recursive Hidden δ

For linear activations (), each neuron sums downstream δ signals weighted by its outgoing connections.

Rule 3 · Every Weight & Bias

Every parameter gradient in all 3 layers is just incoming activation × target neuron’s δ.

Why Does Every Weight Gradient Collapse to a_i^(l-1) · δ_j^(l)?
Interactive 26-Parameter Chain-Rule & Numerical Verifier

Output Layer 3 (2 Weights + 1 Bias)

Hidden Layer 2 (6 Weights + 2 Biases)

Hidden Layer 1 (12 Weights + 3 Biases)

Selected Parameter Chain Decomposition

Section 6 · Check Your Understanding

7 Mastery Questions to Lock It In

Questions 1 to 3 test core differentiation intuition directly; Questions 4 to 7 verify indices, multi-path cancellation, and bias behavior.

Teaching Recommendation for Lectures & Exam Prep

If you are teaching or revising this at a whiteboard, avoid expanding all 26 derivatives separately. Derive the 3 anchor examples carefully without skipping steps— (output 3-factor chain), (hidden Layer 2 single-route 5-factor chain), and (hidden Layer 1 two-path split)—and then introduce so students watch all 26 formulas collapse into one general backpropagation rule.