Maximum likelihood

Training a model means choosing its parameters. Maximum likelihood estimation (MLE) asks one question: which parameter values make the data we actually saw most probable? The answer to that question gives you the two losses you will use most.

Walking through a coin

You flip a coin 10 times and get 7 heads. Which heads probability p makes that most likely?

The probability of exactly this sequence is p⁷ · (1 − p)³. Taking logs gives:

log L(p) = 7 · ln(p) + 3 · ln(1 − p)

d/dp:  7/p − 3/(1 − p) = 0
       7(1 − p) = 3p
       p = 0.7

The MLE is the observed fraction of heads, 7 out of 10. Nothing else was needed.

Why squared error is MLE

Assume each label is the model's prediction plus Gaussian noise: yᵢ = f_θ(xᵢ) + εᵢ, with εᵢ ~ Normal(0, σ²). The log-likelihood of the data is:

log likelihood = constant − (1 / 2σ²) · Σᵢ (yᵢ − f_θ(xᵢ))²

Maximizing that means minimizing the sum of squared errors. Squared loss is not an arbitrary default. It is MLE when the noise is Gaussian.

Why cross-entropy is MLE

For a classifier, the model gives a probability p_θ(y | x) for each class. Maximizing the log-likelihood of the labels:

maximize  Σᵢ log p_θ(yᵢ | xᵢ)
equals    minimize  −Σᵢ log p_θ(yᵢ | xᵢ)     (cross-entropy)

Softmax is how the model produces that categorical distribution. See softmax and temperature.

Why we use logs

Multiplying many probabilities gives a number too small for a float32 to hold. The log turns that product into a sum:

0.01 · 0.02 · 0.03 = 0.000006      (6 × 10⁻⁶)

ln 0.01 + ln 0.02 + ln 0.03 = −4.605 − 3.912 − 3.507 = −12.024
exp(−12.024) ≈ 6.0 × 10⁻⁶

Same answer, and the sum still works when you have hundreds of terms.

Common mistakes

• Maximizing likelihood with no safeguard. MLE can overfit small datasets. Priors and penalties help. A prior turns MLE into MAP, and L2 weight decay is a Gaussian prior on weights.
• Multiplying raw probabilities. Long products underflow to zero. Sum logs instead.
• Treating squared loss as a free choice. It is MLE only under Gaussian noise. If the noise is clearly not Gaussian, the loss needs a different justification.

Check yourself

1. Why maximize the log-likelihood instead of the likelihood? Answer: the log does not move the maximizer, and sums are numerically stable.
2. What is the MLE of p after 3 heads in 10 flips? Answer: 0.3, the observed fraction.

Key takeaways

• MLE picks the parameters that make the observed data most probable.
• Maximize the log-likelihood. The argmax is the same and the arithmetic is stable.
• Squared error is MLE under Gaussian noise. Cross-entropy is MLE for categorical outputs.
• Plain MLE can overfit. A prior turns it into MAP, which acts as regularization.

Further learning

Next, see how a softmax turns raw scores into the probabilities that cross-entropy needs, in softmax and temperature.