The chain rule and backpropagation

You will be able to

Multiply local derivatives along a computational path.
Check an analytical gradient against a finite difference.

Differentiate the graph

For u=wx, v=u² and L=(v−1)², the derivative dL/dw multiplies dL/dv, dv/du and du/dw. Backpropagation applies this chain rule from outputs toward inputs, reusing intermediate values. If a variable affects several paths, their derivative contributions add.

Forward values, backward sensitivities

The forward pass computes predictions and loss. The backward pass calculates derivatives of that loss with respect to parameters. Automatic differentiation follows the implemented operations; it cannot correct a wrong objective. Finite differences are useful for gradient checks in small, smooth cases, with a moderate step to balance truncation and rounding.

dL/dw = 2(v−1) × 2u × x = 4x²w((wx)²−1)

Worked example

At x=2,w=1: u=2, v=4, L=9. The local derivatives are 6,4,2, so dL/dw=48. At w=0.5 the prediction is 1 and the loss gradient is zero.

A common trap

Backpropagation computes gradients; the optimiser uses those gradients to change parameters. They are different stages.

Predict, then experiment

At x=2,w=1, should increasing w increase or decrease loss locally? Before touching the controls, say or write your prediction.

Try it yourself

Trace a composite gradient

How does a change in w affect the final loss?

Weight w

Input x

Graph: u=wx, prediction=u², L=(prediction−1)². Bars show u, prediction and loss. Readout gives the analytical dL/dw; no automatic differentiation library is used.

Your experiment

Set x=2,w=1. Read gradient 48. Compare w=0.9 and 1.1. Then set w=0.5 and inspect why both loss and gradient are zero.

Change one parameter at a time. Compare what you observed with your prediction and record an explanation in your notes.

Check your understanding

Try from memory first. Hints and feedback are part of learning, not a penalty. These checks are for self-study, not a graded qualification.

Reference answers (after your attempt)

add — Correct: the multivariable chain rule sums paths.
48 — 2(4−1)×2(2)×2=48.

Apply it: Check the composite derivative

Use paper, your notes, or a local notebook/editor. Code examples are Python 3 starting points to run yourself; this page does not execute code or automatically assess the practical task.

Evaluate the graph at w±0.0001 with x=2.
Compute a central-difference derivative.
Compare to 48 and explain why too-small steps can be unhelpful.

Reference code / starting point

def loss(w, x=2):
    return ((w*x)**2 - 1)**2
w, eps = 1.0, 1e-4
print((loss(w+eps)-loss(w-eps))/(2*eps))

Reference solution & review criteria

The central difference approaches 48. Finite step introduces truncation error; tiny steps amplify floating-point cancellation. Avoid nonsmooth activation boundaries when checking a smooth formula.

I attempted the task and compared my reasoning with the solution or rubric.

Your notebook

Prediction, observations, reasoning and questionsAdd this lesson to my review list

Notes and progress stay in this browser. They are not sent to the server; ordinary web-server request logs still operate. Export a backup from the course page.

Review list

Completion requires both correct checks and your practical-task acknowledgement.

Go deeper

Dive into Deep Learning: forward and backward propagation

Sources support further study; the worked examples and questions here are original. Browser experiments are small, disclosed models—not evidence of production-model quality or hardware performance.