Attention alone does not assign an order to its inputs. A sinusoidal encoding gives each position a deterministic coordinate while preserving a simple algebra for offsets.
A scalar needs more than one clock
One sine wave repeats, so it cannot uniquely identify arbitrary positions. Several wavelengths repeat at different rates; taken together, their phases form a richer coordinate. The classic transformer encoding pairs sine and cosine at each angular frequency:
PE(p, 2i+1) = cos(p / 10000^(2i/d))
Here p is position, i selects a frequency pair and d is embedding dimension. The denominator grows geometrically, so early pairs turn quickly while later pairs vary slowly.
Interactive experiment
A position becomes a vector
How do fast and slow sinusoidal coordinates tile position space?
Blue cells are positive and amber cells negative; opacity gives magnitude. The outlined row is shown numerically, so the matrix does not rely on colour alone.
Why the pair matters
A sine value alone cannot distinguish phase directions. Sine and cosine together place phase on the unit circle. Advancing by offset Δ rotates each pair by an angle determined only by Δ and that pair's frequency:
This is a precise structural property: an offset acts linearly within each two-dimensional pair. It does not mean a model automatically learns every relative-position rule.
Similarity and offset
The dot product of two complete encodings is a sum of cosines of their offset, one per frequency. It depends on relative separation rather than absolute origin when all dimensions form complete pairs. Because several frequencies interfere, similarity need not decrease monotonically with distance.
Extrapolation is representation, not performance
The formula can generate coordinates beyond any training length. That is numerical extrapolation of the encoding. It does not establish that learned attention weights, data statistics or task performance extrapolate. At very large positions, finite-precision argument reduction can also erode phase accuracy.
Later methods
Learned position tables trade deterministic extension for fitted coordinates. Rotary position embeddings apply position-dependent rotations inside attention rather than adding a vector to the token representation. Both invite the same questions: where is position represented, which relations are easy to compute, and how does numerical scale behave?
Practical checks
- Keep embedding dimension even or define the final unpaired coordinate explicitly.
- Verify the exponent convention against the implementation being reproduced.
- Do not infer monotonic distance from a multi-frequency dot product.
- Separate “the encoding is defined” from “the model generalises”.