State-Space and Selective Sequence Models

A state-space model maintains a hidden state. The state evolves as inputs arrive. An observation rule maps that state to the output required at each step.

Classical linear state-space equations use fixed transition and input operators. Deep variants learn their parameters, stack nonlinear blocks, and adapt the mechanism for long digital sequences. What follows is not a survey of that idea in the abstract. It is a set of named systems and one theorem. The interesting question about a compact state is always empirical: what did it actually keep, and what did it actually lose?

Structured state-space layers can expose a recurrent scan for inference and a convolutional form for parallel training.

The dual view became a working layer in 2021, in the paper that introduced S4. Albert Gu and two colleagues condition the state matrix “with a low-rank correction, allowing it to be diagonalized stably and reducing the SSM to the well-studied computation of a Cauchy kernel”. What that bought is in the same abstract. 91% accuracy on sequential CIFAR-10, “with no data augmentation or auxiliary losses, on par with a larger 2-D ResNet”. Generation “60× faster” than Transformers on image and language modelling. State of the art “on every task from the Long Range Arena benchmark, including solving the challenging Path-X task of length 16k that all prior work fails on”.

An abstract praising its own contribution is one source. A program committee is another. The arXiv comments field records “ICLR 2022 (Outstanding Paper HM)”. The award was confirmed on 20 April 2022, and the citation names the contribution precisely: “The S4 model is basically a new and clever parameterization of the state space model (SSM), that can address the prohibitive computation and memory requirements of SSM, while maintaining the theoretical strengths of SSM for handling long-range dependencies.” Read that as a definition rather than as applause. The parameterisation is what the paper added. The recurrent scan and the convolutional kernel are what it made affordable. Path-X at length 16k is what affordability bought.

Here is a state-space layer with one number of state, invented for the check: h_t = 0.5 × h_{t−1} + x_t and y_t = h_t, starting from h = 0.

The families differ in access, compression, parallelism, and content dependence.

The fourth row is the newest, and its authors state the trade in their own abstract. Mamba arrived on 1 December 2023, from Albert Gu and Tri Dao. They name “a key weakness” of earlier subquadratic models: “their inability to perform content-based reasoning”. Their fix is “simply letting the SSM parameters be functions of the input”. That allows “the model to selectively propagate or forget information along the sequence length dimension depending on the current token”. The cost arrives in the next sentence. “Even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode.” Selectivity buys content sensitivity by giving up the convolutional training view. That is why the fourth row lists specialized kernels as a requirement. They report “5× higher throughput than Transformers” at inference, and linear scaling in sequence length. A Mamba-3B “outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation”.

Those benchmarks do not ask for exact recall, and exact recall is where the same design falls behind. Samy Jelassi and three co-authors measured it in a paper presented at ICML in 2024. They gave pretrained Mamba models and Pythia Transformers of similar sizes a phone book in the prompt, then asked for the number of one name on it. Once the phone book reached 70 entries, even the smallest Transformer, at 410M parameters, beat the largest Mamba, at 2.8B. That is the measurement behind the fourth row's last line.

A row in a comparison table is a claim until somebody else builds on it. Mistral AI shipped Codestral Mamba on 16 July 2024. Apache 2.0, 7,285,403,648 parameters, designed with help from Albert Gu and Tri Dao, tested on in-context retrieval up to 256k tokens. Mistral gave the reason for the architecture in its own words: “Unlike Transformer models, Mamba models offer the advantage of linear time inference and the theoretical ability to model sequences of infinite length.” Two organisations, one property, each stating it in its own words. Linear-time inference is the reason the fourth row exists.

These examples share long streams, limited inference memory, or expensive dense attention. The first of them stopped being hypothetical on 19 February 2025. Arc Institute and NVIDIA released Evo 2 that day, a genome foundation model built on StripedHyena 2, a multi-hybrid architecture that mixes several input-dependent convolution operators with attention instead of stacking dense attention alone. Its longest convolutions are parameterised so that they can also run as a recurrence, the property state-space layers have, which makes it a close cousin of the models in this lesson rather than one of them. Arc Institute's news release states the capability that drove the choice: “The model can process genetic sequences of up to 1 million nucleotides at once, enabling it to understand relationships between distant parts of a genome.” Training took over 2,000 NVIDIA H100 GPUs. The model reached over 90% accuracy in classifying BRCA1 variants as benign or potentially pathogenic. Dense attention alone was not set aside on aesthetic grounds. A million-position window is simply the unit this problem arrives in.

A speed comparison is meaningful only when quality, length, hardware, and implementation maturity are controlled.

NVIDIA ran that comparison properly in 2024, on Mamba-based language models. Matched architectures at 8B parameters. A task list broad enough to expose where the architecture loses, rather than a single throughput headline. The five steps below reconstruct that discipline. Note what it costs. A matched 8B comparison is an expensive way to answer a question that an unmatched benchmark answers for free and wrongly.

Different designs choose different continuous-time parameterizations, discretizations, state dimensions, mixing layers, and selectivity mechanisms. Results from one codebase do not transfer automatically to another.

Hybrid architectures can combine state-space blocks with attention or convolution. The design should reflect which interactions need exact content lookup and which can be summarized through state. AI21 Labs made that split concrete rather than conceptual. Jamba, published in 2024, interleaves Transformer and Mamba layers with mixture-of-experts: 12B active parameters out of 52B total. Its model card states the payoff in hardware terms: “It supports a 256K context length, and can fit up to 140K tokens on a single 80GB GPU.”

The figure survived the trip from paper to product. Amazon Web Services documents the Jamba 1.5 family on Amazon Bedrock, available from 23 September 2024, as having “a 256k effective context window, one of the longest on the market”. Attention where exact lookup is required. Mamba layers where history can be summarised. The context window is the thing a customer can actually buy.