Position and representation · sinusoidal.space

You will be able to

Explain why sequence order needs an explicit mechanism.
Calculate one sinusoidal position-encoding pair.

Content does not determine order

A token or feature embedding represents content. A set-like processing mechanism may not distinguish reordered inputs without positional structure, a causal mask or another order-sensitive operation. Standard self-attention without positional information is permutation-equivariant, not automatically a sequence-order model.

Coordinates for position

Sinusoidal encoding uses sine/cosine pairs at different angular frequencies. For dimension d and pair index j, the denominator is 10000^(2j/d). Adding these encodings to content supplies a deterministic position-dependent signal. Multiple frequencies provide several scales, but the existence of values at unseen positions does not establish learned extrapolation.

PE(p,2j)=sin(p/10000^(2j/d)); PE(p,2j+1)=cos(p/10000^(2j/d))

Worked example

At p=0 every complete pair is [0,1]. For the first pair at p=1, values are [sin 1,cos 1]≈[0.8415,0.5403]. The angle here is in radians, not degrees.

A common trap

Having a positional formula defined for longer sequences does not prove a trained model can use those longer contexts accurately.

Predict, then experiment

Which dimensions vary fastest as position changes? Before touching the controls, say or write your prediction.

Interactive experiment

A position becomes a vector

How do fast and slow sinusoidal coordinates tile position space?

Sequence length

Dimensions

Selected position

Blue cells are positive and amber cells negative; opacity gives magnitude. The outlined row is shown numerically, so the matrix does not rely on colour alone.

Your experiment

Use 16 dimensions and 32 positions. Inspect early and late pairs, then compare lengths 32 and 64. Keep the selected position within the length.

Change one parameter at a time. Compare what you observed with your prediction and record an explanation in your notes.

Check your understanding

Try from memory first. Hints and feedback are part of learning, not a penalty. These checks are for self-study, not a graded qualification.

Reference answers (after your attempt)

the representation can be computed there, but performance still needs testing — Correct: evaluate longer contexts empirically.
1 — cos(0)=1.

Apply it: Separate three sources of order

Use paper, your notes, or a local notebook/editor. Code examples are Python 3 starting points to run yourself; this page does not execute code or automatically assess the practical task.

Describe content embeddings, position encodings and causal masks.
Give the purpose of each in an autoregressive sequence model.
State one extrapolation test.

Reference solution & review criteria

Content gives token/features, position gives location, and causal masking restricts access to future tokens. Test held-out longer lengths and tasks while documenting position scaling and any context changes.

I attempted the task and compared my reasoning with the solution or rubric.

Your notebook

Prediction, observations, reasoning and questionsAdd this lesson to my review list

Notes and progress stay in this browser. They are not sent to the server; ordinary web-server request logs still operate. Export a backup from the course page.

Review list

Completion requires both correct checks and your practical-task acknowledgement.

Go deeper

Sources support further study; the worked examples and questions here are original. Browser experiments are small, disclosed models—not evidence of production-model quality or hardware performance.