Nonlinear features and neural networks

You will be able to

Explain why stacked affine maps alone cannot create nonlinear features.
Construct a small ReLU piecewise-linear function.

Stacking linear maps is not enough

An affine map is Wx+b; W is a matrix of weights and b is a bias vector. Compose two affine maps without an activation and the result is still affine. A nonlinear activation between them enables different local behaviour in different input regions.

Hinges build shapes

ReLU(u)=max(0,u) is zero on one side and linear on the other. Weighted sums of shifted ReLUs produce piecewise-linear functions. The lab manually combines two such units; training would fit their weights and biases using a loss. More capacity can model useful structure or memorise irrelevant patterns.

h=ReLU(W₁x+b₁); ŷ=W₂h+b₂

Worked example

y=ReLU(x+1)−ReLU(x) is zero below −1, ramps from 0 to 1 on [−1,0], and is constant 1 above 0. Two hinges create a saturating shape.

A common trap

A multilayer network without nonlinear activations is still an affine map, regardless of how impressive its layer count looks.

Predict, then experiment

With the second hinge at zero and weight −1, what happens for x above zero? Before touching the controls, say or write your prediction.

Try it yourself

A tiny piecewise-linear network

Can two nonlinear units make a bend?

Second hinge k

Second unit weight a

y=ReLU(x+1)+a ReLU(x−k). Solid is y; dashed is first unit. Parameters are manually adjusted, not learned. This network is only a two-unit illustration.

Your experiment

Set second hinge k=0 and weight a=−1. Describe the three regions of the function, then move the hinge to 1 and explain the changed plateau.

Change one parameter at a time. Compare what you observed with your prediction and record an explanation in your notes.

Check your understanding

Try from memory first. Hints and feedback are part of learning, not a penalty. These checks are for self-study, not a graded qualification.

Reference answers (after your attempt)

one affine layer — Correct: substitute one affine expression into the other.
0 — max(0,−3)=0.

Apply it: Construct a hinge function

Use paper, your notes, or a local notebook/editor. Code examples are Python 3 starting points to run yourself; this page does not execute code or automatically assess the practical task.

Evaluate ReLU(x+1)−ReLU(x) at x=−2,−0.5,2.
Name the breakpoints.
Explain which parameters would be fitted during training.

Reference solution & review criteria

Outputs are 0,0.5,1; breakpoints are −1 and 0. Learned weights and biases set slopes, locations and combination; the manual lab is not evidence of training.

I attempted the task and compared my reasoning with the solution or rubric.

Your notebook

Prediction, observations, reasoning and questionsAdd this lesson to my review list

Notes and progress stay in this browser. They are not sent to the server; ordinary web-server request logs still operate. Export a backup from the course page.

Review list

Completion requires both correct checks and your practical-task acknowledgement.

Go deeper

Sources support further study; the worked examples and questions here are original. Browser experiments are small, disclosed models—not evidence of production-model quality or hardware performance.