Transformers and language models
Multimodal transformers
Images and audio become sequences of vectors that share one transformer with text. Early fusion trains them together, while late fusion attaches a vision encoder to a frozen language model.
A language model sees only token ids. To handle an image, the model needs the image turned into vectors that behave like tokens. Once they do, one transformer can attend across text and image positions. The main design question is when the two streams meet.
Turning images into tokens
There are three common routes:
- Patches (ViT). Split the image into patches. Flatten each patch and project it into a vector, which becomes one token.
- A pretrained vision encoder. A model such as a CLIP-style encoder produces features, and a projection maps them to the language model's width.
- Concatenation. Place the image tokens among the text tokens, with attention masks that say which positions can see which.
Audio follows the same pattern. It becomes discrete tokens from a neural codec, or continuous features from an encoder, and a projection maps those into the shared stream.
Walking through a patch count
Take a 224 × 224 image split into 16 × 16 patches:
patches per side = 224 / 16 = 14
patches total = 14 × 14 = 196
values per patch = 16 × 16 × 3 colors = 768Each patch's 768 values pass through a linear projection into the model's width, such as 4,096. The image now adds 196 tokens to the sequence. Higher-resolution images add more, and those tokens count against the context window.
Early fusion versus late fusion
| Early fusion | Late fusion | |
|---|---|---|
| Where the image enters | The visual tokens join the stream at the first layer | A frozen language model gets vision through cross-attention or adapters |
| Training | Trained jointly with text | The base model's weights stay unchanged |
| Strength | Scales well with joint training and deep interleaving | Keeps a tested language model intact |
Early fusion scales better when you can train jointly. Late fusion is the choice when you must not touch the base model's weights.
Trade-off
Native multimodal models can interleave images and text deeply and serve one stack. Calling a separate image model as a tool keeps the best image generator and makes costs easier to track. The product constraint usually decides this before the architecture does.
Common mistakes
- Underfitting the projection. If the projection is weak, the language model learns to ignore the image tokens.
- Forgetting the token cost. Image tokens take context-window space and compute.
- Evaluating only on text. A multimodal model needs image-based tests.
Check yourself
- How many patch tokens does a 224 × 224 image give with 16 × 16 patches? Answer: 14 × 14 = 196.
- Which fusion keeps the base language model unchanged? Answer: Late fusion with adapters, since the base weights stay frozen.
Key takeaways
- Images and audio become token-like vectors that share a transformer with text.
- Patches are the simplest route. Each patch becomes one token.
- Early fusion trains jointly. Late fusion attaches to a frozen model.
- Image tokens cost context and compute, so count them.
Further learning
For multimodal product patterns, see multimodal. Back to the overview: modern LLM architectures.
Related
Modern LLM architectures
Mixture of experts, GQA and MLA, long context, state space models and multimodal transformers.
Multimodal inputs and outputs
Images, PDFs, audio and video in LLM apps: what works, what it costs and how to evaluate it.
Generative models beyond text
How images, video and speech are generated. Compares the main model families, builds diffusion from first principles, and explains why diffusion inference costs so differently from chat.