Optimization and gradient descent

A model is a function with adjustable knobs, called parameters. A loss scores how wrong the function is on the data. Optimization turns the knobs to push that score down. Almost all of modern ML uses variants of gradient descent: compute the gradient of the loss, step a little in the opposite direction, and repeat. This topic covers the geometry of that walk and the hyperparameters that decide whether it works.

You need gradients for the update direction, and probability for why the loss is a negative log-likelihood. Here the question is: given the gradient, how do you walk downhill without wasting steps or falling off a cliff?

Where it shows up

Every model you train, from a logistic regression to a large transformer, runs some form of this loop. The loop is the same. What changes is the size of the parameter vector, the batch size and the optimizer. Understanding each piece lets you read a training log and know which knob to turn.

The learning rate is the clearest example. Set it too high, and each step overshoots the bottom, so the loss grows instead of falling. Set it too low, and the loss falls so slowly that the run looks stuck. Both look like "training is broken", and the fix is different for each.

Concepts

Common mistakes

• Blaming local minima. A flat loss usually means the learning rate, the features or the code is wrong. Check those first.
• Judging a run by training loss alone. A run can fit the training data and fail on new data. Watch the validation loss.
• Tuning the optimizer before the data. Scaling and splits matter more than the choice between optimizers.

Key takeaways

• Training minimizes a loss by stepping against its gradient.
• The learning rate is the most important hyperparameter. Too large diverges, and too small stalls.
• Mini-batch SGD trades gradient noise for speed, and it often generalizes better.
• Momentum and Adam fix ravines and uneven gradients. AdamW is a practical default for neural nets.
• For convex problems such as logistic regression, use a well-tested library solver. Gradient descent is for models that have no closed form.

Further learning

StatQuest walks through gradient descent with small numbers and clear diagrams:

3Blue1Brown shows what gradient descent looks like inside a neural network:

An overview of gradient descent optimization algorithms (Sebastian Ruder)Gradient descent is the preferred way to optimize neural networks and many other machine learning algorithms but is often used as a black box. This post explores how many of the most popular gradient-based optimizatio…ruder.io Dive into Deep Learning: Optimization12. Optimization Algorithms — Dive into Deep Learning 1.0.3 documentationd2l.ai