Decoding controls

Temperature and truncation decide which token can appear. The rest of the decoding surface decides how long generation runs, when it stops, how much it repeats, and what you can read about the model's confidence.

The big idea

Hitting max_tokens mid-sentence is a product bug waiting to happen. Check the stop reason and continue, summarize, or raise max_tokens for that path. Prefer structured outputs over brittle stop-sequence parsers when you need JSON.

Setting temperature to 0 (or using a greedy path) removes sampling noise. It does not guarantee identical strings across calls:

• Batching different request shapes can change floating-point reduction order on GPUs
• Speculative decoding, mixture-of-experts routing and provider-side optimizations can introduce nondeterminism
• Model versions silently change under aliases

If you need reproducibility for compliance or debugging, store the output (and model version, seed if offered, and prompt). Do not plan to regenerate the same answer later and diff it. Some APIs expose a seed parameter that improves repeatability when supported. It is still not a cryptographic guarantee.

Walking through logprobs

Some APIs return logprobs: log probabilities of sampled (and optionally top-n alternative) tokens. Uses:

• Confidence proxies for classification ("positive" vs "negative" with token logprobs)
• RAG abstention — if the model assigns low mass to grounded answer tokens, escalate
• Eval analysis — compare probability of the gold token under two prompts

Logprobs are not calibrated probabilities of correctness. A fluent wrong answer can have high token probability. Treat them as features, not truth.

For classification endpoints, consider reading logprobs of label tokens at T=0 instead of generating a sentence and parsing it. Fewer failure modes, cheaper outputs, easier thresholds.

Treat the whole sampling config as part of the prompt version:

prompt_version: billing-v3
model: …
temperature: 0.2
top_p: default
max_tokens: 800

When someone "just tweaks temperature in prod," that is an unreviewed model change. Gate it like a code change: same suite, same judge, compare fail sets.

Common mistakes

• Planning to regenerate a "deterministic" answer later and diff it. Store the original.
• Ignoring the stop reason. A cut-off sentence looks like a finished answer.
• Treating logprobs as calibrated truth. Fluency is not correctness.

Check yourself

• Temperature is 0 and today's answer differs from last week's. Is the sampler broken? Not necessarily. GPU reductions, speculative decoding, MoE routing or a model alias can all move the string.
• The JSON ends mid-key. What do you read first? The stop reason. If it was max_tokens, the object was cut off.

Key takeaways

• Always set max tokens and handle the stop reason.
• Temperature 0 ≈ greedy, not bit-identical determinism across infrastructure.
• Use penalties sparingly for repetition. Prefer structured outputs for JSON.
• Logprobs are useful signals, not calibrated truth. Store them next to the prompt version.

Further learning

Next, see how several samples can be combined in self-consistency and best-of-n.