Entropy and cross-entropy

A fair coin is harder to predict than a biased one. Entropy puts a number on that difficulty. Cross-entropy uses the same idea to score a model against the true labels, and it is the loss you use for classification.

Two examples show the scale:

• A fair coin, p = [0.5, 0.5], has H = ln 2 ≈ 0.693 nats, which is 1 bit.
• A biased coin, p = [0.9, 0.1], has H ≈ 0.325 nats. It is less uncertain, so it has less entropy.

Cross-entropy

Cross-entropy H(p, q) = −Σᵢ pᵢ log qᵢ is the average surprise when outcomes come from the true distribution p, but you believe q.

When the true label is one-hot, p puts all its weight on the correct class. The sum then reduces to:

cross-entropy = −log q(true class)

That is the classification loss. Low loss means the model gave the right class a high probability.

Walking through one prediction

The true class is 0, so p = [1, 0, 0]. The model says q = [0.659, 0.242, 0.099].

H(p)    = 0                      (a one-hot label has no uncertainty)
H(p, q) = −ln 0.659 ≈ 0.417 nats
KL(p‖q) = H(p, q) − H(p) = 0.417

If the model became more confident and gave the true class 0.9, the loss would fall to −ln 0.9 ≈ 0.105.

KL divergence

KL divergence measures how far q is from p:

KL(p ‖ q) = Σᵢ pᵢ log(pᵢ / qᵢ) = H(p, q) − H(p)

KL is never negative, and it is zero only when p = q. It is not symmetric: KL(p‖q) is generally not equal to KL(q‖p). With one-hot labels, minimizing cross-entropy is the same as minimizing KL to the label distribution.

Perplexity for language models

For a language model, perplexity is exp of the average per-token cross-entropy. A perplexity of 20 means the model is as uncertain as picking uniformly among 20 tokens at each step. Lower is better, but only when the tokenizer and data are the same.

Common mistakes

• Comparing perplexity across tokenizers. Different tokenizers split text differently, so the numbers do not compare.
• Mixing nats and bits. They measure the same thing in different units. A 1-bit entropy is 0.693 nats.
• Treating KL as a distance. It is not symmetric, so it cannot be a true distance.

Check yourself

1. Why is cross-entropy the classification loss? Answer: with a one-hot label it reduces to −log of the probability given to the true class.
2. What is the cross-entropy when the model gives the true class probability 1? Answer: 0, because −log 1 = 0.

Key takeaways

• Entropy is the average surprise of samples from a distribution.
• With one-hot labels, cross-entropy is −log p(true class).
• KL divergence is cross-entropy minus entropy, and it is not symmetric.
• Perplexity is exp of the average per-token cross-entropy.

Further learning

Josh Starmer's explainers on entropy and cross-entropy are patient and concrete:

Next, see how sample size turns a measured metric into a range in uncertainty and sample size.