Project: read and reproduce a research claim

You will be able to

Trace a specific research claim to its assumptions and evidence.
Design a minimal reproducible check and report its limits.

Read one claim deeply

Choose a primary paper linked from the existing essays: rotary positions, selective state spaces or FFT-based computation. Write a single claim in your own words and identify whether it is an algebraic result, an accuracy experiment, a complexity argument or a hardware benchmark. Record publication/version and distinguish author claims from what you independently verified.

Choose the smallest meaningful reproduction

For an identity, test both a normal case and a counterexample outside the assumptions. For a numerical algorithm, compare against a simple reference with tolerances. For a performance claim, disclose hardware, environment, inputs, validation and timing scope. If the needed device or training budget is unavailable, report that limitation and reproduce a narrower claim rather than inventing results.

Close with evidence and boundaries

A good technical note includes a source link, assumptions, code or exact calculations, expected result, observed result, error/variability and limitations. A negative result can be valuable if the setup is correct and discrepancies are investigated. Reproduction on one shape, device or task is not universal validation; changes in context and version should be visible.

claim → assumptions → reproducible method → observed evidence → limitations

Worked example

Testing fixed-vector RoPE under a shared position shift verifies a pairwise identity. It does not reproduce long-context accuracy on a trained transformer. A traffic ratio likewise does not reproduce a measured GPU speedup.

A common trap

A citation is a pointer to evidence, not a substitute for checking what that evidence actually supports.

Predict, then experiment

If a claimed formula is algebraically shift-invariant, can a naive floating-point implementation still fail? Before touching the controls, say or write your prediction.

Try it yourself

The same probabilities, safer arithmetic

Does a common logit shift change softmax?

Common logit shift

Logit gap

Logits are [shift, shift+gap]. Solid bars use max-subtracted softmax. Naive exp/sum is shown in the readout and may overflow. JavaScript uses binary64; other precisions have different limits.

Can you make it happen?

Find a common shift where naive softmax produces a non-finite result, while stable softmax stays finite.

Predict a setting before you try it. This optional challenge checks the current parameters, not lesson completion.

A hint

Probabilities should not depend on a common shift, but direct exponentials can overflow or underflow.

One solution & why it works

Shift 1000 and gap 1 make direct exponentials overflow in binary64. Max subtraction gives finite probabilities, approximately [0.2689, 0.7311]. Shift −1000 also exposes underflow.

Your experiment

Use the stable/naive softmax lab as a miniature reproduction exercise: declare the identity, choose an extreme common shift, record both implementations, and explain the discrepancy.

Change one parameter at a time. Compare what you observed with your prediction and record an explanation in your notes.

Check your understanding

Try from memory first. Hints and feedback are part of learning, not a penalty. These checks are for self-study, not a graded qualification.

Reference answers (after your attempt)

state the limitation and narrow the reproduced claim — Correct: a narrower honest reproduction is useful.
2.5× — 20/8=2.5 under that measurement scope.

Apply it: Publish a reproducibility note

Use paper, your notes, or a local notebook/editor. Code examples are Python 3 starting points to run yourself; this page does not execute code or automatically assess the practical task.

Select and link one primary paper; quote or paraphrase one precise claim and list its assumptions.
Run a minimal reference comparison or identity check; record environment, inputs and observed evidence.
Write limitations, an alternative explanation and one next experiment. Do not fabricate inaccessible measurements.

Reference solution & review criteria

Rubric: traceable primary source/version; sharply scoped claim; reproducible method; actual observed results; numerical/timing validation where relevant; explicit assumptions; no conversion of a toy check into an untested model-quality claim.

I attempted the task and compared my reasoning with the solution or rubric.

Your notebook

Prediction, observations, reasoning and questionsAdd this lesson to my review list

Notes and progress stay in this browser. They are not sent to the server; ordinary web-server request logs still operate. Export a backup from the course page.

Review list

Completion requires both correct checks and your practical-task acknowledgement.

Go deeper

Sources support further study; the worked examples and questions here are original. Browser experiments are small, disclosed models—not evidence of production-model quality or hardware performance.