Math toolkit
Loss functions and loss landscapes
A loss function scores how wrong a model is. Its graph over the parameters is the loss landscape. Convex landscapes have one bottom, while deep networks have many valleys and flat saddle regions.
Training is a search for parameter values that make the loss small. The loss is the score. The loss landscape is the shape of that score over all the parameters. Knowing which losses are common, and what shape they make, tells you how hard the search will be.
The big idea
A loss function L(θ) takes the parameters and returns one number. Lower is better. Each common task uses a different loss:
| Task | Loss | Why this one |
|---|---|---|
| Regression | Mean squared error | The maximum-likelihood choice under Gaussian noise |
| Regression with outliers | Mean absolute error | Less sensitive to a few extreme errors |
| Classification | Cross-entropy | The maximum-likelihood choice for categories |
| Ranking or contrastive learning | Pairwise or InfoNCE | Pulls related pairs together and pushes others apart |
Think of walking downhill in fog. You can feel the slope under your feet, which is the gradient, but you cannot see the whole map. Optimization is that walk. The analogy breaks in high dimensions, where the bottom is not one point. Many different parameter settings can fit the data about equally well.
Convex versus non-convex
Concept
A function is convex if the straight line between any two points on its graph lies above the graph. Convex functions are bowl-shaped and have one global minimum. Non-convex functions can have several local minima, saddle points and flat regions.
Linear and logistic regression with standard losses are convex in their weights, once the features are fixed. Gradient descent then finds the global best answer. Deep networks are far from convex. They still train well, because high-dimensional landscapes usually contain many good solutions, and the common obstacle is saddle points rather than bad local minima.
Walking through a saddle
Take f(x, y) = x² − y². At the origin, the gradient is zero, which looks like a bottom. It is not one:
f(0.1, 0) = 0.01 moving along x raises the loss
f(0, 0.1) = −0.01 moving along y lowers the lossThe origin is a saddle point. The gradient is zero, but there is a downhill direction. Stochastic noise and momentum help an optimizer escape saddles.
Common mistakes
- Assuming a deep network's loss is convex. The guarantees for convex problems do not apply.
- Reading a zero gradient as a minimum. Maxima and saddle points also have zero gradient.
- Choosing a loss out of habit. Squared error on class labels is a poor fit, as the logistic regression page explains.
Check yourself
- Is the squared-error loss of linear regression convex in its weights? Answer: Yes. With fixed features, it is a bowl with one global minimum.
- At a point where the gradient is zero, what are the possibilities? Answer: A local minimum, a local maximum, or a saddle point.
Key takeaways
- The loss scores the model. The loss landscape is its shape over the parameters.
- Convex losses have one bottom, and gradient descent finds it.
- Deep network landscapes are non-convex, but they still train.
- A zero gradient does not guarantee a minimum.
Further learning
Next, take the first steps down the landscape: gradient descent and the learning rate.
Related
Optimization and gradient descent
How models learn, from loss landscapes and gradient descent to learning rates, mini-batches, momentum and Adam.
Linear and logistic regression
The two models under almost everything in ML. They fit lines, predict probabilities and stay honest through regularization.
Gradient descent and the learning rate
Gradient descent moves the parameters a small step against the gradient. The learning rate sets the step size, and a wrong value makes training either crawl or blow up.