Eigenvectors, SVD and low-rank approximation

Some directions in a matrix matter more than others. The matrix stretches those directions, and it may twist everything else. Eigenvectors and the singular value decomposition (SVD) find these directions. Keep only the strongest ones and you get a compact approximation of the matrix. That idea sits under PCA and LoRA fine-tuning.

Eigenvectors

An eigenvector of a square matrix A is a direction that A only stretches. It does not rotate it. If A v = λ v, then v is an eigenvector and λ is its eigenvalue.

Take A = | 2 0 | ; | 0 0.5 |, the matrix from the matrices page:

• A · [1, 0] = [2, 0] = 2 · [1, 0]. So [1, 0] is an eigenvector with eigenvalue 2.
• A · [0, 1] = [0, 0.5] = 0.5 · [0, 1]. So [0, 1] is an eigenvector with eigenvalue 0.5.
• A · [1, 1] = [2, 0.5]. That is not a multiple of [1, 1], so [1, 1] is not an eigenvector.

The eigenvectors are the "natural axes" of this transformation. Only square matrices have them.

The SVD

The SVD works for any matrix, square or not:

A = U Σ Vᵀ

U and V are rotations. Σ is a diagonal matrix of singular values, sorted from largest to smallest. Geometrically, the map rotates, stretches along the axes, then rotates again.

Low-rank approximation

Keep the top r singular values and set the rest to zero. The result is the best rank-r approximation of the original matrix in the L2 sense. This is the Eckart–Young result.

Two uses show up constantly:

• PCA. Project data onto the top singular vectors. These are the axes with the most variance.
• LoRA. Instead of changing a full weight matrix W by ΔW, learn a low-rank update ΔW = B A, with a small rank r.

Walking through the LoRA count

A 4096 × 4096 weight matrix has 4096 × 4096 = 16,777,216 parameters.

A rank-8 update uses two thin matrices: B of shape (4096, 8) and A of shape (8, 4096):

parameters = 4096·8 + 8·4096 = 65,536
16,777,216 / 65,536 = 256

That is 256 times fewer trainable numbers. Training only these adapters is how people fine-tune large models on a single GPU.

Common mistakes

• Expecting eigenvectors everywhere. Eigenvectors are defined for square matrices. The SVD works for any matrix.
• Reading singular values without context. Compare each value to the total. A rank-r approximation is only good if the top values hold most of the total.
• Thinking LoRA shrinks the base model. LoRA reduces the number of trained parameters. The frozen base weights are still there.

Check yourself

1. If A v = 3 v, what is the eigenvalue? Answer: 3.
2. A 1000 × 1000 matrix is replaced by two thin matrices of shapes (1000, 10) and (10, 1000). How many numbers are stored? Answer: 2 · 1000 · 10 = 20,000, compared with 1,000,000 for the full matrix.

Key takeaways

• An eigenvector is a direction that a square matrix only stretches.
• The SVD breaks any matrix into rotate, stretch, rotate.
• Keeping the top r singular values gives the best rank-r approximation.
• PCA and LoRA both rely on this low-rank structure.

Further learning

Next, see how to count the compute of these multiplies: FLOP counting for matmul.