Numerical stability · sinusoidal.space

You will be able to

Explain overflow, cancellation and finite precision.
Use a max-subtracted softmax and stable logit loss.

Arithmetic has limits

Floating-point numbers have finite range and spacing. Adding a tiny value to a large one may lose the tiny contribution; subtracting nearly equal approximations may lose relative accuracy. Numerical stability concerns how implementation error behaves, while conditioning concerns sensitivity of the underlying problem. They are related but not identical.

Preserve the result, change the calculation

Softmax is unchanged if a common constant is subtracted from every logit. Subtracting the maximum makes all exponent arguments nonpositive, preventing positive exponential overflow for finite logits. This does not remove all underflow or rounding, but at least one exponent is one. Stable binary loss uses max(z,0)−yz+log(1+exp(−|z|)).

softmax(z)ᵢ = exp(zᵢ−m)/Σⱼexp(zⱼ−m), m=maxⱼzⱼ

Worked example

For logits [1000,1001], naive exponentials overflow in binary64. Subtract 1001 to use [−1,0], giving approximately [0.2689,0.7311]. Exact probabilities are unchanged by the shift.

A common trap

Changing precision, operation order or algebra can change rounding behaviour even when exact-real arithmetic says two expressions are equivalent.

Predict, then experiment

Should a common shift of 1000 change the exact probabilities for a fixed gap? Before touching the controls, say or write your prediction.

Try it yourself

The same probabilities, safer arithmetic

Does a common logit shift change softmax?

Common logit shift

Logit gap

Logits are [shift, shift+gap]. Solid bars use max-subtracted softmax. Naive exp/sum is shown in the readout and may overflow. JavaScript uses binary64; other precisions have different limits.

Can you make it happen?

Find a common shift where naive softmax produces a non-finite result, while stable softmax stays finite.

Predict a setting before you try it. This optional challenge checks the current parameters, not lesson completion.

A hint

Probabilities should not depend on a common shift, but direct exponentials can overflow or underflow.

One solution & why it works

Shift 1000 and gap 1 make direct exponentials overflow in binary64. Max subtraction gives finite probabilities, approximately [0.2689, 0.7311]. Shift −1000 also exposes underflow.

Your experiment

Set gap 1. Compare shifts 0,1000 and −1000. Observe the stable output remaining constant while the naive calculation overflows or underflows.

Change one parameter at a time. Compare what you observed with your prediction and record an explanation in your notes.

Check your understanding

Try from memory first. Hints and feedback are part of learning, not a penalty. These checks are for self-study, not a graded qualification.

Reference answers (after your attempt)

preserves exact probabilities and reduces overflow risk — Correct: it changes the arithmetic range, not the mathematical result.
-1 — 1000−1001=−1.

Apply it: Implement stable probabilities

Use paper, your notes, or a local notebook/editor. Code examples are Python 3 starting points to run yourself; this page does not execute code or automatically assess the practical task.

Calculate naive and stable softmax for [1000,1001].
Verify the stable probabilities sum to one.
Test equal logits, large negative values and a large gap.

Reference code / starting point

from math import exp
z = [1000.0, 1001.0]
m = max(z)
e = [exp(v-m) for v in z]
p = [v/sum(e) for v in e]
print(p, sum(p))

Reference solution & review criteria

Use the maximum subtraction, then normalise exponentials. Equal logits give equal probabilities; an extreme negative gap may round a small probability to zero. Document precision and tolerances.

I attempted the task and compared my reasoning with the solution or rubric.

Your notebook

Prediction, observations, reasoning and questionsAdd this lesson to my review list

Notes and progress stay in this browser. They are not sent to the server; ordinary web-server request logs still operate. Export a backup from the course page.

Review list

Completion requires both correct checks and your practical-task acknowledgement.

Go deeper

Sources support further study; the worked examples and questions here are original. Browser experiments are small, disclosed models—not evidence of production-model quality or hardware performance.