Softmax and temperature

A classifier's last layer outputs raw scores, called logits. Logits can be negative and do not add up to anything. A probability distribution needs values between 0 and 1 that sum to 1. Softmax converts one into the other.

Adding temperature

Temperature T divides the logits before softmax:

pᵢ = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)

• T = 1 is standard softmax.
• T → 0 makes the distribution collapse onto the largest logit, which is greedy choice.
• T → ∞ makes the distribution approach uniform.

Walking through three temperatures

Take logits [2.0, 1.0, 0.1].

At T = 1, the exponentials are 7.389, 2.718 and 1.105. Their sum is 11.212:

p ≈ [7.389, 2.718, 1.105] / 11.212 ≈ [0.659, 0.242, 0.099]

At T = 0.5, divide each logit by 0.5 to get [4.0, 2.0, 0.2]. The exponentials are 54.598, 7.389 and 1.221, which sum to 63.208:

p ≈ [0.864, 0.117, 0.019]      sharper: the top class dominates

At T = 2, divide each logit by 2 to get [1.0, 0.5, 0.05]. The exponentials are 2.718, 1.649 and 1.051, which sum to 5.418:

p ≈ [0.502, 0.304, 0.194]      flatter: the classes are closer

Each row adds up to 1, and only the spread changed. The ranking of the classes did not change.

Why we subtract the max

Large logits overflow exp. Subtracting the largest logit first fixes that:

softmax(z) = exp(z − max(z)) / Σⱼ exp(zⱼ − max(z))

The shift cancels in the ratio, so the answer is the same.

Common mistakes

• Reading a softmax score as a true probability. A score of 0.9 does not mean 90% of such predictions are right. That is a calibration question, and softmax does not answer it.
• Computing softmax naively. Large logits overflow. Use the max-subtraction form.
• Expecting retraining when you change temperature. Temperature changes the output distribution at inference. The weights stay the same.

Check yourself

1. What happens to the distribution as T goes to 0? Answer: it collapses onto the largest logit, so the top class gets probability 1.
2. Does changing temperature change which class is ranked first? Answer: no. Dividing by a positive T keeps the order of the logits.

Key takeaways

• Logits are raw scores. Softmax turns them into a categorical distribution.
• Temperature divides the logits. Low T sharpens the distribution and high T flattens it.
• Subtract the max before exponentiating to avoid overflow.
• Sharpness and calibration are different properties.

Further learning

Next, measure how surprised a model is by the true answer in entropy and cross-entropy. For how LLM decoding uses temperature and top-k or nucleus sampling, see sampling.