You will be able to
- Calculate normalised attention weights.
- Distinguish keys, values, masks and the meaning of attention scores.
Score, normalise, combine
A query q is compared with keys kⱼ. Dot-product scores become softmax weights, which combine values vⱼ. Keys decide relevance under the learned scoring rule; values supply the retrieved content. Standard scaled dot-product attention divides by √dₖ, where dₖ is key dimension, before softmax.
What the tiny lab leaves out
The lab has scalar keys [−1,0,1] and values [2,0,−1], with temperature controlling concentration. Real transformers learn projections and use multiple heads, masks, residual paths and feed-forward layers. Causal masking excludes future positions before normalisation. Attention weights alone are not a complete causal explanation of a model prediction.
Worked example
When q=0 all three scalar scores are zero, so weights are 1/3 each and retrieved value is (2+0−1)/3=1/3. Positive q favours key +1, whose value is −1—not the key itself.
A common trap
A highly weighted key is not necessarily a highly positive output: its associated value may be negative or multidimensional.
Predict, then experiment
As q becomes very positive at low temperature, does the output approach +1 or −1? Before touching the controls, say or write your prediction.
Try it yourself
Retrieve by weighted similarity
Which key does the query favour?
One scalar query; keys [−1,0,1], values [2,0,−1]. Scores=qk; softmax(scores/temperature). Bars are attention weights, not evidence of a trained or interpretable model.
Can you make it happen?
Give the key −1 more than 90% of the attention weight.
Predict a setting before you try it. This optional challenge checks the current parameters, not lesson completion.
A hint
A negative query favours a negative key. Temperature controls how sharply the score differences are expressed.
One solution & why it works
Query −3 and temperature 1 give the key −1 approximately 95.0% weight. This selects its value 2 most strongly; it does not train a model.
Your experiment
Compare query −3,0,+3 at temperature 1, then 0.2. Identify which key is favoured and the value it contributes. Explain why the weight sum stays one.
Change one parameter at a time. Compare what you observed with your prediction and record an explanation in your notes.
Check your understanding
Try from memory first. Hints and feedback are part of learning, not a penalty. These checks are for self-study, not a graded qualification.
Reference answers (after your attempt)
- values — Correct: keys score relevance, values carry retrieved content.
- 0.25 — 0.25×2+0.5×0+0.25×(−1)=0.25.
Apply it: Calculate masked attention
Use paper, your notes, or a local notebook/editor. Code examples are Python 3 starting points to run yourself; this page does not execute code or automatically assess the practical task.
- Use equal logits for three values [2,0,−1].
- Mask out the third key before softmax.
- Calculate the remaining weights and output.
Reference solution & review criteria
The masked weight is zero and the others become [0.5,0.5], giving output 1. Masking after softmax without renormalising would give a different, incorrectly normalised operation.
Your notebook
Notes and progress stay in this browser. They are not sent to the server; ordinary web-server request logs still operate. Export a backup from the course page.
Completion requires both correct checks and your practical-task acknowledgement.
Go deeper
Sources support further study; the worked examples and questions here are original. Browser experiments are small, disclosed models—not evidence of production-model quality or hardware performance.