Validation and repair

A valid structure does not mean a correct answer. The model can return a perfectly shaped object with a negative total or a currency your system does not accept. Validation is the code that checks the business rules the schema cannot express. Repair is the bounded retry that tries once more when validation fails.

The big idea

Treat the model as an untrusted input source. Every output goes through the same path:

1. Parse it into a typed object, such as a Pydantic or Zod model.
2. Check business rules: the amount is not negative, the invoice exists, the date makes sense.
3. Repair on failure: send the error back and ask for a corrected object. Cap this at one or two retries.
4. Route refusals and unrecoverable cases to a human or a fallback. Do not loop.

A customs checkpoint is a fair picture. The paperwork can be complete and still get stopped if the amounts do not match the manifest. The analogy stops at one point: a customs officer does not usually ask the traveler to refill the form with the error message attached. The repair step does.

Walking through it

The rules that a schema cannot express look like this in code:

ALLOWED_CURRENCIES = {"USD", "EUR", "GBP"}


def check_invoice(obj):
    """Return a list of problems. An empty list means the object is usable."""
    problems = []
    if obj.get("currency") not in ALLOWED_CURRENCIES:
        problems.append(f"currency {obj.get('currency')!r} is not allowed")
    total = obj.get("total_cents")
    if not isinstance(total, int) or total < 0:
        problems.append("total_cents must be a non-negative integer")
    return problems


print(check_invoice({"currency": "USD", "total_cents": 120450}))
print(check_invoice({"currency": "JPY", "total_cents": 1204.5}))

Output:

[]
["currency 'JPY' is not allowed", 'total_cents must be a non-negative integer']

The first object passes, so the list is empty. The second fails twice, and both messages are specific enough to send back in a repair turn.

Repair, and when to stop

A repair turn passes the original output and the list of errors back to the model. A typical retry looks like this:

• Attempt 1 fails with total_cents must be a non-negative integer.
• Attempt 2 sends that error back and asks for a corrected object.
• If attempt 2 also fails, stop and route the document to a person.

Two rules matter here:

• Cap the retries at one or two. An unbounded loop turns one bad document into an expensive one.
• Do not repair refusals. If the model declined, retrying usually produces the same refusal. Send it to a fallback path.

Partial output is a failure too. If max_tokens cuts the object in half, the result is incomplete. Count it as an error and size the output budget for the worst case.

Common mistakes

• Trusting strict mode. Strict mode checks the shape. Business rules still need code.
• Unbounded repair loops. One bad input can burn through the budget with no result.
• Forcing repair on refusals. The model has said no. Route to a person instead.
• Ignoring repair rates. A rising repair rate often means a prompt regression or a new document layout.

Check yourself

• Why does a valid object still need check_invoice? The schema checks types and enum values. It cannot check that the total is non-negative in the right currency for this vendor.
• Why limit repair to one or two tries? Each retry costs time and tokens. If the model cannot fix it in a couple of tries, a person should look.

Key takeaways

• Validate every structured output in code, even under strict decoding.
• Send specific error messages back in a bounded repair loop.
• Route refusals and truncated output to a fallback instead of retrying blindly.
• Watch the repair rate as an early warning.

Further learning

Next, measure whether the fields are right, not just whether the JSON parses, in field-level evals.