Transformers and language models
State space models and hybrids
State space models carry a compact running state along a sequence, so their cost grows linearly with length. Hybrids mix these blocks with occasional attention layers to keep sharp recall.
Attention compares every token with every other token, so its cost grows with the square of the length. A state space model (SSM) takes a different route. It passes a fixed-size state along the sequence, one token at a time, so cost grows linearly. The trade-off is that pure SSMs have been weaker at exact recall, so hybrids mix SSM blocks with a few attention layers.
The big idea
An SSM processes a sequence with a structured recurrence, or an equivalent convolution. At each step, it updates a state of fixed size. The output for each token depends on that state.
An analogy: a reader who keeps a short running summary of a book, instead of rereading every earlier page for each new sentence. The summary is cheap to keep up. The analogy breaks down at detail. A compressed summary loses precise facts from far back, while attention can look up an exact earlier token.
Walking through a hybrid stack
Imagine a 12-layer model. Layers 3, 6, 9 and 12 are full attention. The other 9 layers are SSM blocks.
- The 9 SSM layers keep a fixed-size state, so their cost is linear in the length.
- The 3 attention layers can look up any earlier token directly. Their cache grows with the sequence, the same as in any transformer.
- Residual connections and normalization work as in a standard transformer stack.
The mix sets the cost profile. Where the time goes depends on the ratio of the two block types and on the length, so measure your own workload rather than assuming.
When to use which
| Goal | Lean toward |
|---|---|
| Strong in-context recall, well-understood tooling | Standard transformer |
| Very long sequences where linear cost matters | SSM or hybrid, after task evals |
| Mixed quality and throughput needs | Hybrid, tuned on your tasks |
Results vary by task. Pin your own evaluations before you bet a product on a pure SSM.
Common mistakes
- Treating SSMs as drop-in replacements. Recall-heavy tasks can regress.
- Assuming linear cost means cheap in practice. Constant factors and hardware support matter.
- Ignoring the attention layers in a hybrid. They still have a growing cache.
Check yourself
- Why does an SSM layer's state not grow with the sequence length? Answer: It keeps a fixed-size state rather than a stored key and value for every token.
- Why add attention layers to an SSM stack? Answer: To let the model look up specific earlier tokens directly, which a compressed state can lose.
Key takeaways
- SSMs carry a fixed-size state, so their cost grows linearly with length.
- Pure SSMs can lose exact recall, which hybrids address with attention layers.
- Attention layers in a hybrid still keep a cache that grows with the sequence.
- Judge the architecture on your own task evaluations.
Further learning
Next, see how images and audio share the same stack as text: multimodal transformers.
Back to the overview: modern LLM architectures.
Related
Modern LLM architectures
Mixture of experts, GQA and MLA, long context, state space models and multimodal transformers.
Self-attention
The operation at the core of every modern language model. Each position builds a weighted average of the others, with weights set by query-key matches.
Long-context techniques
Long context needs positions that stretch past the training length, attention that stays affordable, and training data that teaches the model to use distant text. A large context number on a spec sheet guarantees none of these.