You will be able to
- Derive the relative-angle identity for one rotated pair.
- Distinguish mathematical invariance from model extrapolation.
Position changes the basis
RoPE rotates coordinate pairs of queries and keys by position-dependent angles. For one frequency ω, q at p becomes R(pω)q and k at s becomes R(sω)k. Orthogonal rotations preserve lengths. Their dot product can be written using only the relative position difference s−p when unrotated content vectors are held fixed.
Many pairs, different scales
A full rotary representation applies different frequencies to many pairs. Modern extensions may alter frequencies or scaling for context length or multidimensional positions. The one-pair lab verifies an algebraic property; real hidden states can themselves change with sequence context, so a shared shift is not a universal guarantee of identical full-model predictions.
Worked example
If p=2,s=7 and ω=0.3, the relative angle is 1.5 rad. Changing to p=12,s=17 leaves that relative angle unchanged for the fixed q and k.
A common trap
RoPE does not guarantee useful behaviour at arbitrarily long contexts; frequency choices, training and task evidence still matter.
Predict, then experiment
At fixed offset and frequency, should moving the query position change the displayed dot product? Before touching the controls, say or write your prediction.
Interactive experiment
Position as relative rotation
Does moving both positions by the same amount change the score?
Fixed content vectors q=[1,0.5], k=[0.4,1]. The plot shows their rotated coordinates on shared axes; the readout gives the dot product.
Your experiment
Use offset 5 and ω=0.3. Change query position from 2 to 12, then change offset to 6. Explain why only the second change affects the relative score.
Change one parameter at a time. Compare what you observed with your prediction and record an explanation in your notes.
Check your understanding
Try from memory first. Hints and feedback are part of learning, not a penalty. These checks are for self-study, not a graded qualification.
Reference answers (after your attempt)
- fixed unrotated content vectors — Correct: the algebra rotates the same q and k.
- 1.5rad — 5×0.3=1.5 radians.
Apply it: Verify an invariant
Use paper, your notes, or a local notebook/editor. Code examples are Python 3 starting points to run yourself; this page does not execute code or automatically assess the practical task.
- Write the one-pair rotation matrix and its transpose.
- Use R(a)ᵀR(b)=R(b−a).
- State the assumptions under which equal scores follow.
Reference solution & review criteria
R(θ)=[[cos θ,−sin θ],[sin θ,cos θ]]. Its transpose is R(−θ). Holding q,k,ω and s−p fixed preserves the pairwise dot product, not necessarily the entire context-dependent model.
Your notebook
Notes and progress stay in this browser. They are not sent to the server; ordinary web-server request logs still operate. Export a backup from the course page.
Completion requires both correct checks and your practical-task acknowledgement.
Go deeper
Sources support further study; the worked examples and questions here are original. Browser experiments are small, disclosed models—not evidence of production-model quality or hardware performance.