Math toolkit
Linear algebra for ML
Vectors, matrices and matrix multiplication, the language every model is written in. Learn the shapes, then batching, FLOP counts and low-rank tricks.
Every neural network is a chain of linear algebra with a few nonlinearities added. A linear layer is a matrix multiply. Attention is three matrix multiplies and a softmax. Token embeddings are rows of a matrix. If you can read shapes, compute a dot product by hand and estimate the FLOPs of a matrix multiply, you can read papers, debug shape errors and reason about training cost.
This topic is the slice of linear algebra that ML uses every day. Each concept gets its own short page.
Read shapes from left to right. A batch of sequences has a batch dimension, then a token dimension, then a feature dimension. A linear layer keeps the first two and changes the last one.
Where it shows up
Attention is the clearest example. Each token is projected into a query, a key and a value, with three linear maps. Queries and keys are compared with dot products, a softmax turns the scores into weights, and the weights mix the values. Every step is a matrix multiply or a dot product. The same arithmetic also sets the cost of a transformer layer, which is why the scaling laws and serving-cost estimates use the same counting rules.
Concepts
| Step | Concept | What it answers |
|---|---|---|
| 1 | Vectors and dot products | What is a vector, and how do you measure similarity? |
| 2 | Matrices and matrix multiplication | What does a matrix do to a vector, and when can two matrices multiply? |
| 3 | Batching and broadcasting | Why does a batch become one matrix multiply, and why do some shapes fail silently? |
| 4 | Eigenvectors, SVD and low-rank | Which directions matter most, and how does LoRA use that? |
| 5 | FLOP counting for matmul | How much compute does a layer cost? |
| 6 | Orthogonality and initialization | Why does the scale of random weights matter? |
Common mistakes
- Shape errors at runtime. Most failures like
mat1 and mat2 shapes cannot be multipliedcome from the inner dimensions not matching. Write the shape beside each tensor as you code. - Silent broadcasts. A shape that broadcasts without error can still be wrong. Assert shapes at the boundaries of a function.
- Mixed weight conventions. Storing weights as
(in, out)in one place and(out, in)in another causes transposes in the wrong place. Pick one convention and document it.
Key takeaways
- Vectors are lists of numbers. The dot product measures similarity and projection.
- A matrix is a linear map. The shape rule
(m × k)(k × n) → (m × n)catches most bugs. - Batching turns many vector operations into one matrix multiply, which is why GPUs win.
- Matrix multiplies cost
2mnkFLOPs, a number you can reuse for scaling and serving estimates.
Further learning
Grant Sanderson's visual introduction to vectors is the best first hour you can spend on this material.
Related
Calculus and gradients
Derivatives, partial derivatives, the chain rule and gradients. This is the math that turns a loss number into parameter updates.
Self-attention
The operation at the core of every modern language model. Each position builds a weighted average of the others, with weights set by query-key matches.
The roofline model and arithmetic intensity
Predicting performance from FLOPs and bytes: compute-bound vs memory-bound. Start here for the map, then work the examples.