Gradient descent · sinusoidal.space

You will be able to

Compute one gradient-descent step.
Explain learning-rate sensitivity and the difference between batch and stochastic updates.

The update rule

Gradient descent subtracts a positive learning rate η times the loss gradient. For the quadratic L(w)=(w−3)², error from the optimum obeys eₜ₊₁=(1−2η)eₜ. The error shrinks when |1−2η|<1, which means 0<η<1 for this particular scaling.

Real training adds variation

Full-batch gradients average all training examples. Stochastic or minibatch gradients estimate that gradient from fewer examples, introducing variability. Momentum and adaptive methods change the update, not the need for sound evaluation. Neural objectives are not generally convex, and a rule valid for this quadratic is not a universal learning-rate range.

wₜ₊₁ = wₜ − η∇L(wₜ); here ∇L(w)=2(w−3)

Worked example

Starting at w=0 with η=0.1 gives gradient −6 and next value 0.6. At η=0.5, the same quadratic reaches 3 in one step. At η=1, it oscillates between 0 and 6.

A common trap

Increasing the learning rate does not necessarily learn faster; it can create oscillation or divergence.

Predict, then experiment

Which of η=0.1,0.5,1 and 1.2 converge from zero for this loss? Before touching the controls, say or write your prediction.

Try it yourself

Watch gradient descent

When does a downhill step overshoot?

Initial parameter w

Learning rate η

Exact iterations of L(w)=(w−3)², gradient 2(w−3), for 16 full-batch steps. Solid curve is parameter w; dashed is the optimum 3. This quadratic is not a neural-training benchmark.

Can you make it happen?

Start at 0 or below and choose a learning rate that gives loss below 0.001 after 16 updates.

Predict a setting before you try it. This optional challenge checks the current parameters, not lesson completion.

A hint

For this quadratic, the error is multiplied by 1−2η each step. Very small steps are slow; steps above 1 are unstable.

One solution & why it works

Start 0 and learning rate 0.5 reach w=3 in one step. Many other rates between 0 and 1 also meet the 16-step target.

Your experiment

Try those four rates at start 0. Compare the exact first step and final loss. Explain the convergence using the error multiplier 1−2η.

Change one parameter at a time. Compare what you observed with your prediction and record an explanation in your notes.

Check your understanding

Try from memory first. Hints and feedback are part of learning, not a penalty. These checks are for self-study, not a graded qualification.

Reference answers (after your attempt)

makes error grow in magnitude — Correct: e changes by −1.4 each step.
0.6 — 0−0.1×(−6)=0.6.

Apply it: Implement and check an optimiser

Use paper, your notes, or a local notebook/editor. Code examples are Python 3 starting points to run yourself; this page does not execute code or automatically assess the practical task.

Run 16 updates of the supplied quadratic.
Log parameter and loss, not just iteration count.
Try a divergent rate and explain why early improvement is insufficient.

Reference code / starting point

w, rate = 0.0, 0.1
for step in range(16):
    gradient = 2 * (w - 3)
    w -= rate * gradient
    print(step + 1, w, (w - 3)**2)

Reference solution & review criteria

For η=0.1, error contracts by 0.8 each step. Compare trajectories for η=1.2. Use deterministic tests on a known objective before trusting a custom optimiser.

I attempted the task and compared my reasoning with the solution or rubric.

Your notebook

Prediction, observations, reasoning and questionsAdd this lesson to my review list

Notes and progress stay in this browser. They are not sent to the server; ordinary web-server request logs still operate. Export a backup from the course page.

Review list

Completion requires both correct checks and your practical-task acknowledgement.

Go deeper

Sources support further study; the worked examples and questions here are original. Browser experiments are small, disclosed models—not evidence of production-model quality or hardware performance.