Sampling and decoding

After the transformer produces a logit for every vocabulary token, something still has to choose. That choice is decoding (sampling). The same weights can sound pedantic or inventive depending only on this step. Most production bugs blamed on "randomness" are either uncontrolled sampling or the false belief that temperature 0 means bit-identical replays.

Prompts shape what the model wants to say. Sampling shapes how much variance you allow when it says it. Teams that only iterate on prompt wording while leaving temperature on a tribal default are tuning with one hand. Teams that twist sampling knobs to paper over a vague prompt usually make evals noisier without fixing the root ambiguity.

logits z  →  (optional temperature)  →  softmax  →  truncate  →  sample

Concepts in this section

Read them in order. Each one builds on the one before.

Starting points by task

Default to provider defaults until evals say otherwise. When you tune, change one knob, re-run the same eval set, and record the setting next to the prompt version. Never ship a "creativity slider" wired straight to temperature without measuring factual regressions.

For JSON, enums and tool arguments, sampling noise is usually the enemy. Prefer provider structured-output modes, temperature 0 on the extraction head, and schema validation with a single repair pass. Mix factual and creative behavior in one endpoint with one global T and the invoice extractor gets spicy.

Common mistakes

• Believing temperature 0 is bit-identical. Store the output if you need a replay.
• Tuning T and top-p together from folklore. Change one knob. Re-run the eval.
• Best-of-n without a checker. A vote can lock in a popular wrong answer.
• Leaving max tokens unset. A runaway completion is a cost incident.

Key takeaways

• Decoding maps logits → probabilities → one token. Temperature reshapes the distribution.
• Top-p adapts truncation to confidence. Prefer it over a fixed top-k when you truncate at all.
• Temperature 0 ≈ greedy, not bit-identical determinism across infrastructure.
• Set max tokens and handle stop reasons. Use penalties sparingly.
• Self-consistency and best-of-n trade dollars for reliability when answers are checkable.

Further learning

The Curious Case of Neural Text Degeneration (arXiv:1904.09751)Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that …arXiv.org

Self-Consistency Improves Chain of Thought Reasoning (arXiv:2203.11171)Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks. In this paper, we propose a new decoding strategy, self-consistency, to replace t…arXiv.org

Start with the first concept page: temperature.