Modern LLM architectures

The decoder-only stack from the transformer architecture page is still the spine of modern models. Most recent changes graft onto it: sparse expert layers (MoE), smaller KV caches (GQA and MLA), longer context, recurrent alternatives (state space models), and other input types such as images. This topic maps those changes, so model cards and papers become readable.

You do not need to implement a large model from scratch. You do need to read a release card. Check the active and total parameters, the number of key-value heads, the native context length, and how images or audio are wired in. Those details set the serving cost, and they decide whether a fine-tuning recipe carries over.

Where it shows up

Every architecture change trades memory, compute or quality against the others. The cache size, the all-to-all traffic of expert layers and the quadratic cost of attention are the constraints that persist. The specific designs come and go.

Concepts

Training-time levers

Some changes happen during training rather than in the architecture. Multi-token prediction trains the model to predict several future tokens from a shared trunk, using separate heads. Those extra heads can be dropped at inference, or reused for speculative decoding. Other levers include carefully curated mid-training data mixes, long-context annealing and synthetic data. None of them rename the architecture, but each changes what the model can do.

Reading a model card

Use this checklist on any new open release:

1. Parameters. Total and active, if it is an MoE model.
2. Shape. Layers, width, query heads and key-value heads. These give you the cache size.
3. Vocabulary. The size of the embedding table.
4. Context. The trained length, and whether long-context training was done.
5. License. Whether you are allowed to ship it.

Then estimate the weight memory as about 2 bytes per parameter in 16-bit precision, and add the cache for the number of sessions you expect. If the estimate says three GPUs before a single long chat fits, trust the estimate.

Failure modes

• MoE expert collapse. The router ignores most experts, and quality suffers quietly.
• GQA over-shrinking. Too few key-value heads can hurt tasks that need exact recall.
• Lost in the middle. A model can pass needle tests and still miss facts in the middle of a long prompt.
• Projector underfitting. The language model ignores the image tokens.

Each failure looks like "the model is weak" until you measure routing histograms, cache sizes and modality ablations.

Key takeaways

• MoE raises capacity while keeping active compute modest, but it needs careful routing and serving.
• GQA and MLA shrink the KV cache, which often decides how much context fits.
• Long context needs positional scaling and long training data, not only a bigger number.
• State space models and hybrids trade quadratic attention for linear-time state in some layers.
• Multimodal models project images and audio into the same transformer stream.

Further learning

Mixtral of ExpertsWe introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). Fo…arXiv.org DeepSeek-V3 Technical ReportWe present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Mul…arXiv.org GQA: Training Generalized Multi-Query Transformer ModelsMulti-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not be desirable to train a separate mod…arXiv.org