Regularisation and model selection

You will be able to

Explain regularisation as a constraint or penalty on fitting.
Select model capacity using validation rather than final test data.

Fit versus generalisation

High capacity can fit quirks of training examples. Underfitting misses useful structure; overfitting models variation that does not transfer. These are statements about training and held-out behaviour, not merely whether the curve looks complicated.

Trade-offs require evidence

An L2 penalty adds λ times the squared parameter norm to the training objective. Early stopping chooses an earlier checkpoint based on validation behaviour; dropout introduces stochastic masking during training. These techniques have different mechanisms and do not guarantee generalisation. Fit preprocessing on train and keep final test data out of hyperparameter selection.

L_total = L_data + λΣⱼwⱼ² (stated penalty convention)

Worked example

For one weight w=2 and λ=0.1, this convention adds 0.4 to the loss and 0.4 to its derivative. Some libraries use a half factor or different optimiser weight-decay behaviour; verify the convention.

A common trap

Weight decay and adding an L2 penalty are not interchangeable under every optimiser; avoid treating the names as a universal identity.

Predict, then experiment

Does introducing more bends prove the model will generalise better? Before touching the controls, say or write your prediction.

Try it yourself

A tiny piecewise-linear network

Can two nonlinear units make a bend?

Second hinge k

Second unit weight a

y=ReLU(x+1)+a ReLU(x−k). Solid is y; dashed is first unit. Parameters are manually adjusted, not learned. This network is only a two-unit illustration.

Your experiment

Move the hinge and its weight to make several plausible shapes. Explain what additional data and held-out objective would be needed to choose among them.

Change one parameter at a time. Compare what you observed with your prediction and record an explanation in your notes.

Check your understanding

Try from memory first. Hints and feedback are part of learning, not a penalty. These checks are for self-study, not a graded qualification.

Reference answers (after your attempt)

Validation — Correct: preserve a separate final assessment.
0.4 — 0.1×2²=0.4.

Apply it: Plan a model-selection experiment

Use paper, your notes, or a local notebook/editor. Code examples are Python 3 starting points to run yourself; this page does not execute code or automatically assess the practical task.

Choose two model capacities and a regularisation grid.
Fix grouping, preprocessing and validation metric.
Define the final-test procedure before seeing its results.

Reference solution & review criteria

Compare baselines and model/penalty choices on fixed validation data (or training-only cross-validation). Select once, refit under a declared policy, then evaluate the untouched final test with counts, uncertainty and limitations.

I attempted the task and compared my reasoning with the solution or rubric.

Your notebook

Prediction, observations, reasoning and questionsAdd this lesson to my review list

Notes and progress stay in this browser. They are not sent to the server; ordinary web-server request logs still operate. Export a backup from the course page.

Review list

Completion requires both correct checks and your practical-task acknowledgement.

Go deeper

Sources support further study; the worked examples and questions here are original. Browser experiments are small, disclosed models—not evidence of production-model quality or hardware performance.