Temperature

After the transformer produces a logit for every vocabulary token, something still has to choose. Temperature is the first knob in that choice. The same weights can sound pedantic or inventive depending only on this step. Prompts shape what the model wants to say. Temperature shapes how much variance you allow when it says it.

The big idea

Temperature T divides logits before softmax:

p_i = exp(z_i / T) / Σ_j exp(z_j / T)

Greedy decoding always takes argmax(p), the single most likely token. It is boring and repetitive on open-ended prose, and often fine for classification-style outputs. Setting T at or near 0 is how you get there in an API.

Sampling is per token, not per sentence. A single unlucky draw early in a JSON object (a wrong brace or key) can doom the rest of the string even if later tokens are greedy. That is one reason constrained decoding beats "hope plus temperature 0" for structured work.

Walking through three cities

Toy vocabulary with logits for the next city token: Paris=2.0, Lyon=1.0, cheese=0.1.

Softmax at T=1:

exp(2.0)=7.389, exp(1.0)=2.718, exp(0.1)=1.105
sum=11.212
P(Paris)=0.659, P(Lyon)=0.242, P(cheese)=0.099

At T=0.5, divide logits first: 4.0, 2.0, 0.2:

exp(4)=54.60, exp(2)=7.39, exp(0.2)=1.22
sum=63.21
P(Paris)=0.864, P(Lyon)=0.117, P(cheese)=0.019

At T=2.0, logits become 1.0, 0.5, 0.05:

P(Paris)≈0.506, P(Lyon)≈0.307, P(cheese)≈0.187

Same model, three different risks of emitting cheese.

import numpy as np

logits = np.array([2.0, 1.0, 0.1])
names = ["Paris", "Lyon", "cheese"]

def pick(temperature, seed=0):
    rng = np.random.default_rng(seed)
    if temperature <= 0:
        return names[int(np.argmax(logits))]
    z = logits / temperature
    p = np.exp(z - z.max())
    p /= p.sum()
    return names[int(rng.choice(len(p), p=p))]

print(pick(0.5), pick(2.0), pick(0))

Paris Lyon Paris

Use greedy (or T≈0) for enums, labels, SQL fragments, JSON fields a validator will check, and logprob scoring. Use higher T when diversity is a product feature, or when you will aggregate several samples. Most business apps should spend most traffic on the greedy side. Isolate creative surfaces so a marketer's temperature experiment cannot perturb the invoice extractor.

Common mistakes

• Raising temperature to fix a vague prompt. Evals get noisier. The ambiguity stays.
• One global T for extraction and brainstorming. Split the endpoints.
• Fighting a reasoning model's docs. Those models often ignore or fix temperature.

Check yourself

• At T=0.5, why does P(Paris) rise from 0.659 to 0.864? Dividing by 0.5 stretches the logits, so the leader takes more of the softmax mass.
• Why can T=0 still produce a bad JSON object? Greedy picks the top token. If the top token is a wrong key, later tokens follow that mistake.

Key takeaways

• Temperature divides logits before softmax. Low T peaks. High T flattens.
• Sampling is per token. One bad early token can ruin a structured string.
• Keep factual paths near T=0. Keep creative paths on a separate config.

Further learning

Next, see how the long tail is cut in top-k, top-p and min-p.