Project: a sequence-model investigation

You will be able to

Compare a fixed recurrence with content-weighted retrieval.
State what evidence is required before claiming a model advantage.

Choose a dependency

One task asks for a fading weighted sum of past values; a scalar recurrence implements this exactly. Another asks for retrieval of a value associated with a matching key; attention naturally expresses content-weighted retrieval. Neither task alone establishes that an architecture is universally better. Specify sequence length, information available at each step and whether predictions must be causal.

Match the comparison

For the fixed recurrence, compute an output using both sequential updates and its lag kernel, then verify equivalence numerically. For attention, compute scores, masking and normalised weighted values on a small supplied sequence. Explain how fixed-size state and all-token retrieval use different storage/computation assumptions. A fair trained-model comparison would also match data, parameter budget, precision and optimisation.

A useful investigation is falsifiable

Write an expected outcome and a counterexample before running code. Changing transition A based on input removes the simple fixed-kernel model. Changing positions without holding hidden content fixed invalidates the simple rotary invariant demonstration. Keep algebraic checks, synthetic task observations and empirical claims about trained models in separate categories.

fixed memory hₜ=a hₜ₋₁+xₜ versus retrieval Σⱼsoftmax(q·kⱼ)vⱼ

Worked example

For a=0.5 and x=[1,2,0], recurrence and lag-kernel evaluation both give [1,2.5,1.25]. That equivalence is a property of this fixed linear system, not proof of parity between Mamba and a transformer.

A common trap

One toy task or asymptotic complexity statement does not establish model quality, energy use or measured speed on all workloads.

Predict, then experiment

Can a fixed fading sum selectively retrieve an arbitrary past value by matching its content without changing its mechanism? Before touching the controls, say or write your prediction.

Try it yourself

Retrieve by weighted similarity

Which key does the query favour?

Query q

Temperature

One scalar query; keys [−1,0,1], values [2,0,−1]. Scores=qk; softmax(scores/temperature). Bars are attention weights, not evidence of a trained or interpretable model.

Can you make it happen?

Give the key −1 more than 90% of the attention weight.

Predict a setting before you try it. This optional challenge checks the current parameters, not lesson completion.

A hint

A negative query favours a negative key. Temperature controls how sharply the score differences are expressed.

One solution & why it works

Query −3 and temperature 1 give the key −1 approximately 95.0% weight. This selects its value 2 most strongly; it does not train a model.

Your experiment

Explore query values that favour each scalar key. Compare that selection with the uniformly fixed time-decay weights of the memory lab linked in the lesson references.

Change one parameter at a time. Compare what you observed with your prediction and record an explanation in your notes.

Check your understanding

Try from memory first. Hints and feedback are part of learning, not a penalty. These checks are for self-study, not a graded qualification.

Reference answers (after your attempt)

mathematical identity, toy observation and trained-model evidence — Correct: each supports a different strength of claim.
1.25 — h₀=1,h₁=2.5,h₂=1.25.

Apply it: Deliver a comparison memo

Use paper, your notes, or a local notebook/editor. Code examples are Python 3 starting points to run yourself; this page does not execute code or automatically assess the practical task.

Run the recurrence/kernel identity below and report maximum absolute difference.
Work one masked-attention example and state what content it retrieves.
Design, but do not fabricate, a matched trained-model study with task, budgets, metrics and limitations.

Reference code / starting point

x, a = [1.0, 2.0, 0.0], 0.5
h, recurrent = 0.0, []
for value in x:
    h = a*h + value
    recurrent.append(h)
kernel = [sum(a**(t-j)*x[j] for j in range(t+1))
          for t in range(len(x))]
print(recurrent, kernel)
print(max(abs(u-v) for u, v in zip(recurrent, kernel)))

Reference solution & review criteria

Rubric: exact fixed-system assumptions and zero initial state; correct kernel indexing; normalised masked attention; causal-access policy; storage/compute distinctions; no invented trained-model benchmark or ranking.

I attempted the task and compared my reasoning with the solution or rubric.

Your notebook

Prediction, observations, reasoning and questionsAdd this lesson to my review list

Notes and progress stay in this browser. They are not sent to the server; ordinary web-server request logs still operate. Export a backup from the course page.

Review list

Completion requires both correct checks and your practical-task acknowledgement.

Go deeper

Sources support further study; the worked examples and questions here are original. Browser experiments are small, disclosed models—not evidence of production-model quality or hardware performance.