Vectors and dot products

Almost everything in ML is stored as a list of numbers. A user's features, a word's embedding and a model's weights are all vectors. To work with them you need two moves: combine vectors, and measure how much two of them agree. The dot product does the second. It sits under retrieval, attention and every linear classifier.

The big idea

A vector is an ordered list of numbers. You can picture it as an arrow from the origin to a point. A row of features such as [age, logins, tickets] is a vector too, even though nobody draws it.

Two operations cover most of what you need:

• Add two vectors by adding matching entries: [1, 2] + [3, −1] = [4, 1].
• Scale a vector by multiplying every entry: 2 · [1, 2] = [2, 4].

Walking through the dot product

The dot product of two vectors of the same length multiplies matching entries and adds the results:

a · b = a₁b₁ + a₂b₂ + ... + aₙbₙ

Take a = [2, 3] and b = [4, −1]:

a · b = 2·4 + 3·(−1) = 8 − 3 = 5

The result also describes the angle between the vectors:

a · b = |a| · |b| · cos θ

Here θ is the angle between them. Three cases matter:

• Vectors pointing the same way give a large positive number.
• Perpendicular vectors give 0.
• Opposite vectors give a negative number.

Try a = [1, 0] and b = [1, 1]. The dot product is 1, |a| = 1 and |b| = √2, so cos θ = 1/√2 ≈ 0.707. The angle is 45°.

Why this matters in ML:

• Similarity. For unit-length vectors, the dot product is the cosine similarity. Retrieval and attention scores are built on this.
• Linear scores. A linear classifier computes w · x + b. That asks how far x points along the weight direction, then shifts by b.

Norms: measuring length

A norm is a way to measure a vector's length. Three common ones:

Divide a vector by its L2 norm to get a unit vector. For [3, 4] that is [0.6, 0.8], and 0.6² + 0.8² = 1.

L2 shows up in weight decay. L1 pushes weights to exactly zero, which is why lasso regression produces sparse models.

Common mistakes

• Comparing raw dot products across different lengths. A long vector scores high even when it points elsewhere. Normalize to unit length first to get cosine similarity.
• Mixing lengths. A dot product needs both vectors to have the same number of entries.
• Assuming the norms agree. [3, 4] has an L2 length of 5 but an L1 length of 7.

Check yourself

1. What is [1, 2] · [3, −1]? Answer: 3 − 2 = 1.
2. Two unit vectors have a dot product of 0. What is the angle between them? Answer: 90°. They are perpendicular.

Key takeaways

• A vector is an ordered list of numbers. You can add vectors and scale them.
• The dot product multiplies matching entries and sums them. It is large when vectors point the same way.
• For unit-length vectors, the dot product is cosine similarity.
• L2 measures straight-line length. L1 sums the absolute values of the entries.

Further learning

Next, apply the dot product to whole grids of numbers: matrices and matrix multiplication.