The next-token loop

A decoder-only language model is a function: given the tokens so far, score every vocabulary entry for "what comes next." Generation is a loop that samples one token, appends it, and repeats. Chat, tools and agents are this loop with different strings in the prompt and different stop rules.

Walking through one reply

1. Encode the prompt as token IDs.
2. Run the transformer to get a logit for every vocabulary entry.
3. Turn those logits into one token via sampling.
4. Append that token and repeat until a stop condition.

P(next_token | tokens_so_far)

Stop conditions include an end-of-turn token, a max length, a stop string, or a tool-call boundary. If the stop was tool-use, your app runs the tool, appends the result as tokens, and resumes the loop. A "stuck" agent is often an unbounded decode. A slow first paint is the first full pass over the prompt. A slow stream is one-token-at-a-time throughput.

def generate(prompt_ids, max_new, score_fn, sample_fn, stop_id):
    """Toy loop: score, sample, append. score_fn returns logits for the next position."""
    ids = list(prompt_ids)
    for _ in range(max_new):
        next_id = sample_fn(score_fn(ids))
        ids.append(next_id)
        if next_id == stop_id:
            break
    return ids


print(generate([10, 20], max_new=3, score_fn=lambda ids: [0, 0, 99], sample_fn=lambda logits: logits.index(max(logits)), stop_id=99))

Output:

[10, 20, 2, 2, 2]

The toy scorer always prefers token 2 (the highest logit). A real model changes the logits as the sequence grows. The product wrapper is the same: bound max_new, honor stop_id, and do not let the loop run forever.

Why this picture matters

You do not ship the scorer. You do ship the bounds. Long outputs are slow because each token waits for the one before it. Stream them, and ask for brevity. Private docs, live prices and account state are not in the frozen weights — they must be inserted as tokens or fetched with tools.

Hallucinations are not a separate module. Next-token prediction produces fluent continuations whether or not a fact is grounded. Fine-tuning makes the model helpful; it does not install a truth database. Grounding (RAG, tools, citations) and refusal policies are how products constrain the dream.

Common mistakes

• Unbounded decode. Set a max token count and a stop rule on every call.
• Waiting for the full answer before showing anything. Tokens arrive one at a time; stream them.
• Treating fluency as truth. A fluent continuation is what the loop is for, grounded or not.

Check yourself

1. Why is a 400-token answer slower to appear than a 40-token one, even after the first token? Decode is one token at a time, so length adds steps to the loop.
2. The model stops with a tool-call boundary. What does your app do next? Run the tool, append the result as tokens, and resume the loop.

Key takeaways

• The model scores the next token; sampling picks one; the loop appends and repeats.
• Products wrap that loop with prompts, tools and stop rules.
• Bound the loop. Stream the tokens. Do not treat fluency as evidence.

Further learning

Sampling is how a logit becomes a token. Next, the mixer inside each step: attention.