Case study: document extraction pipeline

Extracting fields from invoices, contracts, forms and KYC packets is one of the highest-ROI LLM workloads in the enterprise. The output is structured, quality is measurable field-by-field and batching makes cost predictable. The hard parts are not "call the model." They are schemas, confidence routing, human review, idempotent reprocessing and not silently corrupting a ledger when the model misreads a total.

Target for this case study: about 1M documents/month, about 98% field-level accuracy on in-scope types, with humans on the long tail. Hours of latency are fine for most docs. Minutes for a priority queue. Every value links to model version, prompt version and source page. Re-runs must not double-post to the ERP. PII stays out of logs. Regional processing constraints apply.

This workload sits closer to classical data pipelines than to chat. You will lean on structured outputs, multimodal inputs for scans, queues for bursts and batch discounts from latency and cost.

Concept

Field-level accuracy = correct fields / total fields, usually micro-averaged on a labeled set. A document can be "mostly right" and still fail the business if total_due is wrong. Also track critical-field accuracy and document pass rate (all critical fields correct).

Concepts in this section

Read them in order. Each one builds on the one before.

Step	Concept	What it answers
1	Schemas and validators	What does the model fill, and what does code reject?
2	Tiering and batch cost	How do you hit a million docs without a frontier bill?
3	Confidence and review	Which rows auto-commit, and which need a human?
4	Reprocessing	How do you rerun a prompt fix without double-posting?

Extraction is a data product. Assign an owner for schema changes, a pager for sink failures and a weekly accuracy review with ops. When finance says "totals are wrong," you need the doc_id, image, model version and validation errors in one click. That is LLMOps applied to batch systems.

Watch drift: new invoice templates, seasonal vendors, provider model updates. Pin versions. Re-score a frozen golden set nightly. Slice metrics by vendor and by tier. A global 98% can hide one strategic supplier at 80%. When accuracy drops, check OCR and classification first. Teams often burn days retuning extraction prompts when the classifier started sending contracts down the invoice path.

Failure gallery

Swapped currency or decimal (100.00 vs 10000 cents) — currency enum plus range checks per vendor.
Wrong invoice attached to the PO — cross-check PO number and vendor.
Duplicate submissions — hash bytes; idempotent sink.
Prompt change shifts date format — schema regex plus golden set in CI.
Multi-page totals on the last page only — scan all pages or run a dedicated "find total" pass.
Handwritten notes overriding printed amounts — printed wins unless a human confirms.

Common mistakes

Optimizing only micro-average field accuracy. Rare expensive fields get ignored.
Flipping 1M docs to a new schema overnight. Shadow, then canary.
Retuning the extract prompt when the classifier is routing the wrong type.

Key takeaways

Spec volume, field versus critical accuracy, auditability, idempotency and privacy.
Pipeline: ingest → classify → structured extract → validate → auto or human → sink.
JSON schemas plus constrained decoding, then business-rule validators.
Batch APIs and model tiering at million-doc scale.
Route on validation plus calibrated confidence. Version extracts. Reprocess safely.