Deep architectures
State-Space and Selective Sequence Models
Explain state-space sequence models, selective updates, parallel training views, and the trade-offs behind modern linear-time architectures.
By the end you can
- Describe a state-space model as input-driven state evolution plus observation
- Explain how convolutional and recurrent views can represent the same linear system
- Distinguish fixed dynamics from input-selective state updates
- Identify workloads and failure modes that favor or weaken state-space architectures
A sequence can be processed as a dynamical system
A state-space model maintains a hidden state. The state evolves as inputs arrive. An observation rule maps that state to the output required at each step.
Classical linear state-space equations use fixed transition and input operators. Deep variants learn their parameters, stack nonlinear blocks, and adapt the mechanism for long digital sequences. What follows is not a survey of that idea in the abstract. It is four named systems and one theorem. The interesting question about a compact state is always empirical: what did it actually keep, and what did it actually lose?
The core question is what compact state should survive after each new observation.
Visual
One mechanism, two computational views
Structured state-space layers can expose a recurrent scan for inference and a convolutional form for parallel training.
The dual view became a working layer in 2021, in the paper that introduced S4. Albert Gu and two colleagues condition the state matrix “with a low-rank correction, allowing it to be diagonalized stably and reducing the SSM to the well-studied computation of a Cauchy kernel”. What that bought is in the same abstract. 91% accuracy on sequential CIFAR-10, “with no data augmentation or auxiliary losses, on par with a larger 2-D ResNet”. Generation “60× faster” than Transformers on image and language modelling. State of the art “on every task from the Long Range Arena benchmark, including solving the challenging Path-X task of length 16k that all prior work fails on”.
An abstract praising its own contribution is one source. A program committee is another. The arXiv comments field records “ICLR 2022 (Outstanding Paper HM)”. The award was confirmed on 20 April 2022, and the citation names the contribution precisely: “The S4 model is basically a new and clever parameterization of the state space model (SSM), that can address the prohibitive computation and memory requirements of SSM, while maintaining the theoretical strengths of SSM for handling long-range dependencies.” Read that as a definition rather than as applause. The parameterisation is what the paper added. The recurrent scan and the convolutional kernel are what it made affordable. Path-X at length 16k is what affordability bought.
- 1
Input enters
The current element contributes to the evolving state.
- 2
State transitions
Past information is retained, transformed, or forgotten.
- 3
Observation is produced
A projection reads the state for the current output.
- 4
Equivalent kernel is formed
Under linear time-invariant assumptions, the recurrence can be represented as a convolution.
Comparison
Attention, recurrence, and state-space layers route history differently
The families differ in access, compression, parallelism, and content dependence.
The fourth row is the newest, and its authors state the trade in their own abstract. Mamba arrived on 1 December 2023, from Albert Gu and Tri Dao. They name “a key weakness” of earlier subquadratic models: “their inability to perform content-based reasoning”. Their fix is “simply letting the SSM parameters be functions of the input”. That allows “the model to selectively propagate or forget information along the sequence length dimension depending on the current token”. The cost arrives in the next sentence. “Even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode.” Selectivity buys content sensitivity by giving up the convolutional training view. That is why the fourth row lists specialized kernels as a requirement. They report “5× higher throughput than Transformers” at inference, and linear scaling in sequence length. A Mamba-3B “outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation”.
A row in a comparison table is a claim until somebody else builds on it. Mistral AI shipped Codestral Mamba on 16 July 2024. Apache 2.0, 7,285,403,648 parameters, designed with help from Albert Gu and Tri Dao, tested on in-context retrieval up to 256k tokens. Mistral gave the reason for the architecture in its own words: “Unlike Transformer models, Mamba models offer the advantage of linear time inference and the theoretical ability to model sequences of infinite length.” Two organisations, one property, stated independently. Linear-time inference is the reason the fourth row exists.
Self-attention
Each position retrieves directly from visible representations.
- Flexible content-based access
- Explicit pairwise interactions
- Large score or cache cost
- No forced fixed-size state during training
Gated recurrence
History is compressed into a state through nonlinear transitions.
- Streaming by construction
- Constant-size recurrent state
- Sequential dependency
- Can struggle with very long credit paths
Structured state space
A designed state evolution captures long filters efficiently.
- Long convolutional receptive field
- Parallel training implementations
- Compact recurrent inference
- Structure constrains possible dynamics
Selective state space
Input-dependent parameters decide what enters or persists.
- Content-aware updates
- Linear sequence scaling
- Requires specialized kernels
- Selectivity does not guarantee exact recall
Example
Where state-space designs are plausible candidates
These examples share long streams, limited inference memory, or expensive dense attention. The first of them stopped being hypothetical on 19 February 2025. Arc Institute and NVIDIA released Evo 2 that day, a genome foundation model built on a sub-quadratic architecture called StripedHyena 2 rather than dense attention. Arc Institute's news release states the capability that drove the choice: “The model can process genetic sequences of up to 1 million nucleotides at once, enabling it to understand relationships between distant parts of a genome.” Training took over 2,000 NVIDIA H100 GPUs. The model reached over 90% accuracy in classifying BRCA1 variants as benign or potentially pathogenic. Dense attention was not rejected on aesthetic grounds. A million-position window is simply the unit this problem arrives in.
- Genomic sequence modeling: motifs interact across long distances, and Evo 2's training corpus — over 9.3 trillion nucleotides drawn from more than 128,000 genomes — sits far past convenient dense-attention lengths.
- Audio streaming: a recurrent state supports low-latency processing without retaining every past frame.
- Industrial telemetry: long sensor histories may contain slow trends, periodic components, and sparse regime changes.
- Long document modeling: selective updates can retain salient state while processing tokens in linear sequence time.
- Edge forecasting: compact state can reduce memory compared with a growing key–value cache.
Steps
Evaluate a state-space candidate fairly
A speed comparison is meaningful only when quality, length, hardware, and implementation maturity are controlled.
NVIDIA ran that comparison properly in 2024, on Mamba-based language models. Matched architectures at 8B parameters. A task list broad enough to expose where the architecture loses, rather than a single throughput headline. The five steps below reconstruct that discipline. Note what it costs. A matched 8B comparison is an expensive way to answer a question that an unmatched benchmark answers for free and wrongly.
1. Define memory demands
Separate trend tracking, exact retrieval, copying, counting, and long-range composition.
2. Match model budgets
Control parameter count, training tokens, optimization effort, and output head.
3. Measure both modes
Benchmark parallel training and recurrent streaming inference.
4. Stress the state
Vary delay, distractors, event rarity, and sequence length beyond training.
5. Inspect operating cost
Include kernel availability, cache size, compilation, and deployment hardware.
Key idea
Selectivity changes the model, not the laws of information
Input-dependent gates can decide which observations influence the state and how quickly older content decays. That improves content sensitivity over a fixed linear filter. It does not repeal the bottleneck. And the bottleneck is not a suspicion — it is a theorem.
The proof is about copying, and it was presented at ICML in 2024. Samy Jelassi and three co-authors open their abstract with it: “We start with a theoretical analysis of the simple task of string copying and prove that a two layer transformer can copy strings of exponential length while GSSMs are fundamentally limited by their fixed-size latent state.” Two layers of attention are enough. No amount of selectivity inside a fixed-size state is. They also find that pretrained transformers dramatically outperform state space models at copying and at retrieving from context.
That is a proof about idealised models, so the obvious objection is scale. NVIDIA's controlled 8B study answers it independently. Pure SSMs “lag behind Transformers on tasks which require strong copying or in-context learning abilities (e.g., 5-shot MMLU, Phonebook) or long-context reasoning”. A finite state can still overwrite or entangle details, and the tasks on which it does so are now named. Test selective sequence models on exact recall, order, rare events, and adversarial distractors. The failure mode is predictable, which means it is measurable, which means there is no excuse for discovering it in production.
Linear-time processing does not imply lossless memory.
Analogy
A control room with a carefully designed dashboard
A control room cannot preserve every raw sensor reading, so it updates a dashboard containing trends, alarms, and accumulated quantities chosen to support future decisions.
Every gauge on a dashboard carries a fixed human meaning, while learned state dimensions carry none. The central trade-off survives the difference: efficient history processing requires deciding what information the state can preserve. The analogy also predicts the copying result. Ask a control room to recite the exact sequence of readings from an arbitrary hour last week and it cannot, however well designed its gauges are. Not because the operators were careless. Because a dashboard is a fixed-size summary and a transcript is not.
A compact state is powerful only when its update rule retains the evidence the task will later need.
State-space models are a family, not one universal recipe
Different designs choose different continuous-time parameterizations, discretizations, state dimensions, mixing layers, and selectivity mechanisms. Results from one codebase do not transfer automatically to another.
Hybrid architectures can combine state-space blocks with attention or convolution. The design should reflect which interactions need exact content lookup and which can be summarized through state. AI21 Labs made that split concrete rather than conceptual. Jamba, published in 2024, interleaves Transformer and Mamba layers with mixture-of-experts: 12B active parameters out of 52B total. Its model card states the payoff in hardware terms: “It supports a 256K context length, and can fit up to 140K tokens on a single 80GB GPU.”
The figure survived the trip from paper to product. Amazon Web Services documents the Jamba 1.5 family on Amazon Bedrock, available from 23 September 2024, as having “a 256k effective context window, one of the longest on the market”. Attention where exact lookup is required. Mamba layers where history can be summarised. The context window is the thing a customer can actually buy.
Use the family label only after documenting the actual state update and mixing path.
Key takeaways
- State-space models process sequences through learned state evolution and observation rules.
- Structured linear dynamics can expose both convolutional training and recurrent inference views. That is the S4 parameterisation, which ICLR 2022 named an Outstanding Paper Honorable Mention.
- Input-selective updates make state retention content-dependent, as Albert Gu and Tri Dao's Mamba does, without eliminating the finite-state bottleneck.
- Long streams and streaming workloads may suit compact state better than exact arbitrary retrieval. Evo 2 processes up to 1 million nucleotides at once. But at copying, GSSMs are fundamentally limited by their fixed-size latent state — that is a proof, not a benchmark.
- Evaluation should stress delay, distractors, ordering, rare events, and extrapolation to longer sequences. Copying, in-context learning and long-context reasoning are where NVIDIA's controlled 8B study found the deficit.
- Hybrid designs can reserve attention for exact lookup while using state-space blocks for efficient continuous processing, as Jamba does with 12B active parameters of 52B total and a 256K context length.