Self-attention without positional information or an order-dependent mask does not assign an order to its inputs. A sinusoidal encoding gives each position a deterministic coordinate while preserving a simple algebra for offsets.
A scalar needs more than one clock
One sine wave repeats, so it cannot uniquely identify arbitrary positions. Several wavelengths repeat at different rates; taken together, their phases form a richer coordinate. The classic transformer encoding pairs sine and cosine at each angular frequency:
PE(p, 2i+1) = cos(p / 10000^(2i/d))
Here p is position, i selects a frequency pair and d is embedding dimension. The denominator grows geometrically, so early pairs turn quickly while later pairs vary slowly.
Interactive experiment
A position becomes a vector
How do fast and slow sinusoidal coordinates tile position space?
Blue cells are positive and amber cells negative; opacity gives magnitude. The outlined row is shown numerically, so the matrix does not rely on colour alone.
Why the pair matters
A sine value alone cannot distinguish phase directions. Sine and cosine together place phase on the unit circle. Advancing by offset Δ rotates each pair by an angle determined only by Δ and that pair's frequency:
This is a precise structural property: an offset acts linearly within each two-dimensional pair. It does not mean a model automatically learns every relative-position rule.
Similarity and offset
The dot product of two complete encodings is a sum of cosines of their offset, one per frequency. It depends on relative separation rather than absolute origin when all dimensions form complete pairs. Because several frequencies interfere, similarity need not decrease monotonically with distance.
Extrapolation is representation, not performance
The formula can generate coordinates beyond any training length. That is numerical extrapolation of the encoding. It does not establish that learned attention weights, data statistics or task performance extrapolate. At very large positions, finite-precision argument reduction can also erode phase accuracy.
Later methods
Learned position tables trade deterministic extension for fitted coordinates. Rotary position embeddings apply position-dependent rotations inside attention rather than adding a vector to the token representation. Both invite the same questions: where is position represented, which relations are easy to compute, and how does numerical scale behave?
Practical checks
- Keep embedding dimension even or define the final unpaired coordinate explicitly.
- Verify the exponent convention against the implementation being reproduced.
- Do not infer monotonic distance from a multi-frequency dot product.
- Separate “the encoding is defined” from “the model generalises”.
Build the smallest useful example
With four dimensions, the two angular frequencies are 1 and 0.01 radians per position. At position zero, the encoding is [0, 1, 0, 1]. At position one it is approximately [0.84147, 0.54030, 0.01000, 0.99995]. The first pair moves appreciably; the second is a slowly turning reference.
A wavelength is 2π/ω positions, not simply the denominator in the formula. The fastest pair repeats about every 6.28 positions; the slower pair in this example repeats about every 628.32 positions. The base 10000 chooses a scale distribution rather than a universal optimal length.
Read the heatmap as a bank of clocks
Set dimension to four and move the selected position. Read adjacent numbers in pairs. Each pair has squared length one, while its angle changes. Increase dimension to sixteen: more intermediate wavelengths appear. Changing dimension changes the frequency distribution itself, so it is not equivalent to truncating a larger encoding.
Make the offset property explicit
For the ordering [sine, cosine], the rotation matrix is [[cos θ, sin θ], [−sin θ, cos θ]], with θ = ωΔ. Stating the ordering matters because the familiar matrix for [cosine, sine] has opposite off-diagonal signs.
For even dimension d, the encoding has norm √(d/2). Its cosine similarity across offset Δ is therefore (2/d) Σᵢ cos(ωᵢΔ). This supplies an offset-dependent comparison, not a distance metric with a guaranteed monotonic curve.
What adding it to an embedding does
The original construction adds this vector to a token embedding of the same dimension. The result combines content and position in one coordinate space. Once learned projections act on that sum, content–position cross terms enter attention scores; the clean dot-product identity for bare encodings does not describe the entire attention mechanism.
Further reading
Attention Is All You Need, section 3.5, introduces this construction. The Fourier essay gives the complementary view of sinusoids as coordinates for a signal rather than for position.