A sequence model can remember through a state rather than by retaining every previous token. Start with one impulse and ask how its influence survives future updates.
The simplest fading memory
Consider hₜ = ρhₜ₋₁ + xₜ, with state h₋₁=0 before the first input. An impulse at time zero produces 1, ρ, ρ², …. For 0<ρ<1, the influence decays. Near one it lasts longer. At one it never decays; above one it grows, which can cause instability.
A two-dimensional state can also rotate. The matrix A=ρR(θ) turns the state by θ radians and scales it by ρ at each step. Equivalently, a complex state is multiplied by ρ exp(iθ). Decay controls persistence; phase controls oscillation.
Interactive experiment
An impulse leaves a rotating memory
How do decay and rotation control the response to one input?
State starts at [1,0], then evolves as hₜ₊₁=ρR(θ)hₜ with no further input. Solid blue and dashed amber are the two state coordinates, not the output of a trained model.
Separate persistence from what you observe
At decay 0.95, the state norm halves after about 13.51 steps. Changing rotation alters either coordinate's waveform without changing that norm envelope. A readout that observes only the first coordinate can cross zero while substantial information remains in the second. Small output is not the same as an empty state.
Unroll the recurrence into a filter
For fixed matrices, write hₜ=Ahₜ₋₁+Bxₜ and yₜ=Chₜ. Starting from h₋₁=0 and substituting earlier states gives:
With a nonzero initial state, add CA^(t+1)h₋₁ to this expression. The impulse-response coefficients CAᵏB define a causal convolution. With scalar A=0.5 and B=C=1, input [1,2,0] gives output [1,2.5,1.25] by either recurrence or convolution. This equivalence depends on fixed dynamics; input-dependent transitions need not form a stationary convolution.
Another representation changes execution
The same scalar example is multiplication by a lower-triangular matrix whose rows begin [1,0,0], [0.5,1,0] and [0.25,0.5,1]. Recurrence avoids storing that full matrix. Structured block methods can expose parallel work while retaining compact representations.
Transformers are SSMs (ICML 2024) develops structured state-space duality through semiseparable matrices and connections to variants of attention. This provides Mamba-2's algorithmic foundation. The title should not be read as saying arbitrary softmax attention is identical to our fixed linear filter.
What the newer research changes
Mamba-3 (March 2026; ICLR 2026) combines a revised discretisation, complex state dynamics and a multi-input, multi-output formulation. These link numerical modelling decisions to state tracking and execution efficiency. Our two-coordinate recurrence illustrates decay and rotation only; it is not a miniature trained Mamba-3.
Discretisation approximates how a continuous system evolves between observations. A higher formal order requires its stated assumptions: the paper distinguishes accuracy conditions on its trapezoidal parameter from the unconstrained settings preferred in its experiments. Better approximation and better task performance are related questions, not interchangeable guarantees.
What finite memory costs
A fixed-size state compresses history. It cannot be assumed to retain every detail of arbitrarily long inputs. Compare retrieval, state tracking and task accuracy alongside latency and state storage. The useful engineering question is which information the state preserves at the cost and scale your application needs.
Return to convolution for the fixed-filter view, or hardware-aware computation for why an efficient representation still needs an efficient implementation.