Field-level evals for extraction

"The JSON is valid" is the easiest metric to report and the least useful one. A model can return valid JSON on every email and still get the invoice ID wrong on 24 of 200 emails. Field-level evals score each field on its own, so you see which fields are weak and which ones are safe.

The big idea

Split the score by field. Overall exact match is a single number that hides the weak field. Per-field accuracy shows where the risk lives.

A bank statement is a good picture. "All transactions are correct" is one claim. "The amounts are right" and "the dates are right" are separate claims, and a reviewer checks each one. The analogy holds, with one caveat: some fields are worth far more than others. A wrong invoice ID that routes a payment is worse than a wrong vendor spelling.

Walking through it

Suppose the eval set has 200 emails. Two setups are compared on the same emails.

Prompt-only JSON:
  valid JSON:            184 / 200 = 92.0%
  all fields correct:    151 / 200 = 75.5%

Strict structured outputs:
  valid JSON:            200 / 200 = 100%
  all fields correct:    173 / 200 = 86.5%

Strict outputs fix the syntax completely. Exact match rises from 75.5% to 86.5%. The gain comes from fewer shape errors, not from better values.

Now look at each field under the strict setup:

vendor:       190 / 200 = 95.0%
invoice_id:   176 / 200 = 88.0%
total_cents:  181 / 200 = 90.5%
currency:     194 / 200 = 97.0%

invoice_id is the weakest field. The cause is often simple. People write Invoice # 4412 and the system expects INV-4412. That is a normalization problem, which code can fix with a lookup. It is not a reason to drop schemas.

Rules for running the evals

• Keep the schema fixed when you compare models. Change only the model ID. Otherwise a schema change gets credited to the model.
• Change one variable at a time when you A/B prompt wording for extraction, on the same eval slice.
• Set service levels on the fields that move money. A wrong ID posted to the ERP costs more than a failed validation that goes to a person.
• Publish a per-field dashboard next to the overall exact-match rate.

Common mistakes

• Celebrating 100% valid JSON. It says nothing about the values.
• Averaging fields together. A 95% average can hide a 88% weak field that matters most.
• Comparing models on different schemas. You cannot tell which change helped.

Check yourself

• Why does strict mode raise exact match but leave invoice_id weak? Strict mode fixes the shape. It does not fix a value the document writes in a different format, so that needs normalization.
• Which metric should you alert on for a payments extractor? The accuracy of the money and ID fields, because a wrong value there moves money.

Key takeaways

• Score each field on its own. Overall JSON validity hides the fields that matter.
• Store money as integer cents, and compare models on a fixed schema.
• Fix weak fields with normalization in code where the problem is formatting.
• Set alerts on the fields that trigger payments or other real actions.

Further learning

Return to the structured outputs hub for the approaches and the cross-cutting mistakes, or see the broader evals guide for how to build the golden set.