Constrained decoding

Constrained decoding makes the model unable to produce output that breaks your schema. Before each token is sampled, every token that would make the output invalid is removed from the choices. The model then picks from what is left. The result is output that is valid by construction, not valid by hope.

The big idea

Ordinary sampling can pick any token in the vocabulary. Constrained decoding changes that. It turns the schema into a grammar and checks, at each step, which next tokens still lead to a valid object.

A spell-checker that only lets you type letters that spell real words is a rough picture. The analogy breaks here: a spell-checker fixes mistakes after you make them. Constrained decoding never lets the mistake onto the page.

Walking through it

The model has written this so far:

{"currency": "

The schema says currency must be USD, EUR, GBP or UNKNOWN. Suppose the model's raw probabilities for the next token are these (illustrative numbers):

The allowed tokens add up to 0.50 + 0.20 + 0.05 = 0.75. The sampler divides each one by 0.75 so the total is 1 again:

U: 0.50 / 0.75 = 0.667
E: 0.20 / 0.75 = 0.267
G: 0.05 / 0.75 = 0.067

The model picks one of these three. It cannot emit x, no matter how likely it was before masking.

The process, once per token:

1. Compile the schema into a grammar.
2. Find every next token that keeps the output valid.
3. Set the other tokens' probability to zero.
4. Sample from what is left, and repeat until the object is complete.

Common mistakes

• Thinking it guarantees correct values. It guarantees the shape. "currency": "USD" is valid even when the invoice is in euros.
• Forcing an answer when evidence is missing. If every field is required, the grammar can push the model to invent a value. Add an explicit null or a status enum as an exit.
• Forgetting the output limit. A cut-off object is still invalid. Size max_tokens for the largest valid object, or cap list lengths in the schema.

Check yourself

• Why can "currency": "USD" still be wrong for a euro invoice? Constrained decoding checks that the value is an allowed enum, not that it matches the document.
• What happens if the output hits max_tokens mid-object? The object is incomplete and invalid. Treat it as a failure and size the budget up.

Key takeaways

• Constrained decoding masks tokens that would break the schema, one token at a time.
• The renormalized probabilities come from the allowed tokens only.
• It guarantees shape, not truth.
• Give the model a legal way to say "unknown" so it does not fill fields with guesses.

Further learning

Next, see how to write the schema itself in schema design.