A sinusoidal encoding adds clocks to a token vector. Rotary position embedding uses clocks to turn pairs of coordinates inside the query and key themselves.
Two arrows, one reference frame
For a two-dimensional vector, rotation preserves length but changes direction. Let R(θ) be the usual rotation matrix for Cartesian coordinates. Rotate query q by pω and key k by rω. Their dot product becomes:
The absolute positions disappear from this expression. Their difference remains, intertwined with the original content vectors. The experiment holds content fixed so that this property is visible.
Interactive experiment
Position as relative rotation
Does moving both positions by the same amount change the score?
Fixed content vectors q=[1,0.5], k=[0.4,1]. The plot shows their rotated coordinates on shared axes; the readout gives the dot product.
Move the origin, then change the distance
Change query position while keeping the key offset fixed. Both arrows rotate together and their score stays constant. Change the offset: their relative orientation changes and so does the score. Increasing angular frequency makes the score vary over shorter positional distances.
A real attention head uses several pairs at different frequencies, then sums contributions and applies attention scaling and softmax. Its learned queries and keys differ by token. A bare phase curve therefore cannot predict the complete attention map.
Why multiple wavelengths help
One pair repeats after a full turn. Several frequencies provide different scales of positional variation, just as the additive sinusoidal encoding combines fast and slow clocks. RoPE changes where this representation enters the computation: the score directly compares rotated content coordinates.
Rescaling positions slows those clocks, making distant positions fit into a smaller phase range. It also makes nearby positions less distinguishable in the affected pairs. The representation remains computable beyond training length; whether a trained model uses it well requires evaluation.
From sequence positions to image coordinates
An image position has two coordinates. A simple construction assigns separate rotation pairs to horizontal and vertical offsets. A mixed pair can instead use an angle ωₓx+ωᵧy. In either case, a common translation cancels inside relative scores when the construction preserves the rotation identity.
Rethinking RoPE (April 2025) formalises multidimensional constructions through constraints on orthogonal transformations and their generators. Its framework uses commuting generators to preserve relative-position structure and explores learned basis changes. This is a principled route to extensions, not evidence that every rotation scheme improves every task.
Implementation details that change the answer
Pairing adjacent coordinates and pairing halves of a vector are different layouts. They are equivalent only if weights and vector layout are transformed consistently. Keep frequency order, coordinate order and rotation signs aligned with the checkpoint. Rotate the intended query/key channels and store cached keys using the convention expected by later decoding.
Test identity at position zero, norm preservation, and invariance under a common position shift. A numerical test establishes the rotation algebra; retrieval and reasoning evaluations establish model behaviour.
For context-extension research, LongRoPE2 combines frequency rescaling with model training. That distinction matters: extending a positional formula alone is not the complete method.