Matrices and matrix multiplication

A matrix is a rectangle of numbers with rows and columns. Multiply a matrix by a vector and you get a new vector. A linear layer in a neural network does exactly this, with learned numbers in the matrix. Once you can read the shape of a matrix product, most ML code becomes readable.

The big idea

A matrix with m rows and n columns has shape (m, n). Multiply it by a vector x of length n, and you get a vector of length m. Each output entry is the dot product of one row with x:

yᵢ = (row i of the matrix) · x

A matrix is a linear transformation. It stretches, rotates, shears or projects space, and the origin stays fixed. Each column of the matrix shows where one basis direction lands.

Walking through it

A 2×2 matrix that stretches the x-axis by 2 and halves the y-axis:

A = | 2   0   |
    | 0   0.5 |

A · [1, 0] = [2, 0]
A · [0, 1] = [0, 0.5]
A · [1, 1] = [2, 0.5]

Now a real linear layer with 3 inputs and 2 outputs:

W = | 0.5  -1   2 |     shape (2, 3)
    | 1     0   1 |
x = [2, 1, 0.5]

Row 1 dotted with x: 0.5·2 + (−1)·1 + 2·0.5 = 1 − 1 + 1 = 1. Row 2 dotted with x: 1·2 + 0·1 + 1·0.5 = 2.5.

So y = [1, 2.5]. This is nn.Linear(3, 2) without the bias. Add a bias and you get y = W x + b.

The shape rule

Multiplying a matrix of shape (m, k) by one of shape (k, n) gives shape (m, n):

(m × k) · (k × n) → (m × n)

The inner sizes must match, and they disappear from the result. For example, (4 × 3) · (3 × 2) gives (4 × 2). The two 3s must agree. The result takes the outer sizes.

Order matters too. A · B is usually not equal to B · A, and the reverse product may not even be defined.

Transpose, identity, inverse and rank

• Transpose Aᵀ swaps rows and columns, so shape (m, n) becomes (n, m). Backprop through a linear layer multiplies by the transpose of its weights.
• Identity I has 1s on the diagonal, and A · I = A. A residual connection is "add the identity path".
• Inverse A⁻¹ undoes the map: A · A⁻¹ = I. ML rarely computes inverses of large matrices. It solves systems or uses factorizations instead.
• Rank counts the independent rows. A rank-deficient matrix squashes space into a smaller subspace, and a later linear layer cannot bring back the lost information.

Common mistakes

• Reversing a product. A · B and B · A give different results, and often only one is defined.
• Guessing shapes. Write the shape beside each matrix as you build. It takes seconds and prevents most runtime errors.
• Mixing conventions. Some code stores weights as (in, out) and others as (out, in). Pick one and stick with it.

Check yourself

1. What is the shape of (5 × 2) · (2 × 7)? Answer: (5 × 7). The inner 2s match, and the outer sizes remain.
2. Can you multiply a (3 × 4) matrix by a (3 × 4) matrix? Answer: No. The inner sizes are 4 and 3, which do not match.

Key takeaways

• A matrix times a vector is one dot product per output entry.
• A linear layer is a matrix multiply, plus an optional bias.
• The shape rule (m × k)(k × n) → (m × n) catches most bugs.
• Transpose, identity, inverse and rank describe what a matrix can and cannot undo.

Further learning

Next, see how a whole batch of examples becomes one matrix multiply: batching and broadcasting.