Top-k, top-p and min-p

Even at temperature 1, the vocabulary may contain 100k+ entries. Almost all of that mass is irrelevant at any given step. The tail is long and noisy. Truncation deletes it, then renormalizes so the remaining tokens sum to 1. That is what top-k, top-p and min-p do.

The big idea

• Top-k: keep only the k highest-probability tokens. Simple. Bad when the model is very sure (k is too wide) or very unsure (k is too narrow).
• Top-p (nucleus sampling): keep the smallest set whose cumulative probability is at least p (often 0.9–0.95). Adapts to confidence. This is the method from "The Curious Case of Neural Text Degeneration."
• Min-p: keep tokens with probability at least min_p × p_top. Common in open-source stacks. Relative to the mode.

Providers differ on defaults and on whether you can combine these. A practical rule: change temperature or top-p, not both at once, unless you are deliberately sweeping a grid with evals.

If you set top-p=0.9 and temperature=1.5, you first flatten the distribution (more tail mass), then nucleus keeps 90% of that flatter mass. That is a different set than nucleus at T=1. Sweeping both knobs without an eval grid produces folklore.

Walking through a five-token cut

Five-token toy distribution after softmax at T=1: [0.40, 0.25, 0.15, 0.12, 0.08].

Top-p=0.8 keeps tokens until cumulative ≥ 0.8 → first three (0.40+0.25+0.15=0.80), drops the last two, renormalizes to [0.50, 0.3125, 0.1875].

Top-k=2 would keep only [0.40, 0.25] → [0.615, 0.385] after renorm. A stricter cut.

When the model is nearly one-hot ([0.98, 0.01, …]), top-k=50 still wastes nothing important after renorm, but top-p=0.9 collapses to essentially one token. Nucleus adapts. Top-k does not.

import numpy as np

def sample(logits, temperature=1.0, top_p=1.0, rng=None):
    rng = rng or np.random.default_rng()
    if temperature <= 0:
        return int(np.argmax(logits))
    z = logits / temperature
    probs = np.exp(z - z.max())
    probs /= probs.sum()
    order = np.argsort(probs)[::-1]
    cutoff = int(np.searchsorted(np.cumsum(probs[order]), top_p)) + 1
    keep = order[:cutoff]
    kept = probs[keep] / probs[keep].sum()
    return int(rng.choice(keep, p=kept))

logits = np.array([2.0, 1.0, 0.1])
rng = np.random.default_rng(0)
print(sample(logits, temperature=1.0, top_p=0.8, rng=rng))

0

Index 0 is Paris in the temperature toy. Top-p=0.8 can drop cheese before the draw.

Classical NLP often used beam search: keep the top-B partial hypotheses at each step. It helps when you need globally likely sequences under a clean likelihood (machine translation). For open-ended chat it tends to produce bland, repetitive text, one motivation for nucleus sampling. Hosted chat APIs almost never expose beam width.

Common mistakes

• Tuning T and top-p together from a tweet. You cannot tell which knob moved the eval.
• A fixed top-k on a nearly one-hot step. Harmless after renorm, but it does not adapt when the model is unsure.
• Truncating to "add safety." Truncation is not a guardrail. It only shortens the tail.

Check yourself

• Top-p=0.8 on [0.40, 0.25, 0.15, 0.12, 0.08]: which tokens survive? The first three, because 0.40+0.25+0.15=0.80.
• Why does nucleus collapse on [0.98, 0.01, …] at p=0.9? The top token already holds 98% of the mass, so the nucleus is one token.

Key takeaways

• Truncation deletes the tail and renormalizes.
• Top-p adapts to confidence. Prefer it over a fixed top-k when you truncate at all.
• Change temperature or top-p, then re-run the same eval. Do not twist both by habit.

Further learning

The Curious Case of Neural Text Degeneration (arXiv:1904.09751)Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that …arXiv.org

Next, see the other knobs in decoding controls.