Linear regression and least squares

Linear regression predicts a continuous number, such as a price or a delivery time, as a weighted sum of features plus a bias. The hard part is choosing the weights. Least squares chooses the weights that make the average squared error as small as possible. It is the first model worth deriving by hand.

The big idea

The model is ŷ = w · x + b. The bias b can be folded into the weights by adding a constant feature equal to 1.

The loss scores how wrong the predictions are. The standard choice is mean squared error (MSE), the average of (y − ŷ)². Squaring makes big misses count more than small ones, and it keeps the math smooth.

Think of each data point as pulling the fitted line toward it. Least squares finds the line where the pulls balance. The picture breaks with outliers. One far-away point pulls very hard, because its error is squared.

Walking through three points

Fit ŷ = w · x to (1, 2), (2, 3), (3, 5). Skip the bias to keep the arithmetic short.

XᵀX = 1² + 2² + 3² = 14
Xᵀy = 1·2 + 2·3 + 3·5 = 23
w*  = 23 / 14 ≈ 1.643

The predictions are 1.643, 3.286, 4.929. The residuals (actual minus predicted) are 0.357, −0.286, 0.071. The MSE is about 0.071.

The same fit comes out of gradient descent. This script runs both ways:

import numpy as np

X = np.array([[1.0], [2.0], [3.0]])
y = np.array([2.0, 3.0, 5.0])

w_closed = np.linalg.solve(X.T @ X, X.T @ y)  # closed form: solve (XᵀX) w = Xᵀy

w = np.zeros(1)                               # gradient descent on the same loss
for _ in range(200):
    grad = (2 / len(y)) * X.T @ (X @ w - y)
    w -= 0.05 * grad

print(w_closed, w)

Output:

[1.64285714] [1.64285714]

Both methods land on 1.643. The closed form is exact in one step. Gradient descent needs repeated steps, but it still works when no closed form exists.

When the closed form stops working

The closed form needs XᵀX to be invertible, and its cost grows with the number of features. For large datasets, or when you add regularization, use gradient descent or stochastic gradient descent. The optimization topic covers those steps.

Common mistakes

• Forgetting the bias. Without it, the line is forced through the origin.
• Reading MSE in the target's units. MSE is in squared units. Report the root mean squared error if you need the same units as the target.
• Trusting one fit without looking at residuals. Squared error lets a single far point dominate. Plot the residuals before you believe the line.

Check yourself

1. With one feature and no bias, XᵀX = 4 and Xᵀy = 10. What is w? Answer: 10 / 4 = 2.5.
2. Why square the errors rather than take absolute values? Answer: Squares are smooth everywhere, which gives a clean closed-form answer. Absolute values resist outliers better, but the math is less tidy.

Key takeaways

• Linear regression predicts a number as a weighted sum of features.
• Least squares minimizes the mean of the squared errors.
• The closed form is exact for small problems. Gradient descent scales to large ones.
• Check the residuals, because one outlier can drive the fit.

Further learning

Next, the same weighted sum becomes a probability for classification: logistic regression and log-odds.