You will be able to
- Calculate precision and recall from confusion counts.
- Design evaluation for class imbalance and distribution shift.
Count the right errors
A true positive is a positive prediction for a positive case. False positives are positive predictions for negative cases; false negatives are missed positives. Precision divides true positives by all predicted positives. Recall divides them by all actual positives. Accuracy can be high while a rare important class is always missed.
Evaluate the intended deployment
Report counts, class support, threshold and split policy alongside metrics. Check subgroup performance and plausible distribution shift; a random split of historical data may not reflect future sensors or users. Small test sets create uncertainty. Choose thresholds and models with validation data and leave the final test untouched until decisions are fixed.
Worked example
TP=8, FP=2, FN=4 gives precision 0.8 and recall 8/12≈0.667. On 100 cases with only 5 positives, an always-negative classifier has 95% accuracy and 0% recall.
A common trap
There is no universally best metric: error costs, population, class prevalence and the intended use determine what matters.
Predict, then experiment
Can a model make the same probability prediction but incur different decision costs in different applications? Before touching the controls, say or write your prediction.
Try it yourself
From a logit to probability
How does confidence affect cross-entropy?
Curve is sigmoid(z). The circle marks the adjustable logit; target is 0 or 1. Binary cross-entropy uses natural logarithms and a numerically stable logit formula.
Your experiment
Inspect probabilities at logits −1,0,+1. Write the decisions at thresholds 0.3 and 0.7 and explain which mistake type each policy may increase.
Change one parameter at a time. Compare what you observed with your prediction and record an explanation in your notes.
Check your understanding
Try from memory first. Hints and feedback are part of learning, not a penalty. These checks are for self-study, not a graded qualification.
Reference answers (after your attempt)
- 95% accuracy but zero positive recall — Correct: all negatives are right, all positives missed.
- 0.75 — 6/(6+2)=0.75.
Apply it: Write an evaluation card
Use paper, your notes, or a local notebook/editor. Code examples are Python 3 starting points to run yourself; this page does not execute code or automatically assess the practical task.
- State population, split policy, label definition and threshold.
- Include confusion counts, precision/recall and an appropriate baseline.
- List uncertainty, subgroup and shift checks.
Reference solution & review criteria
A useful card names recording/session grouping, positive definition, validation-selected threshold and held-out support counts. Compare a constant baseline; disclose small samples, label noise and populations not evaluated.
Your notebook
Notes and progress stay in this browser. They are not sent to the server; ordinary web-server request logs still operate. Export a backup from the course page.
Completion requires both correct checks and your practical-task acknowledgement.
Go deeper
Sources support further study; the worked examples and questions here are original. Browser experiments are small, disclosed models—not evidence of production-model quality or hardware performance.