Extraction reprocessing

Prompt improvements require reprocessing old docs without corrupting downstream systems. Treat every extraction as a pure function of (bytes, schema_v, extractor_v, model_id). If any input changes, you get a new output row. You never overwrite history.

The big idea

• Store raw bytes immutably. Extraction results are derived data keyed by extractor_version.
• Downstream writes use idempotency keys. Prefer "upsert extraction_result" over "insert payment."
• Re-runs write a new result row. Business logic chooses the latest accepted version.
• Never mutate human-approved values silently. Flag for re-review if the new extract disagrees on critical fields.
• Record who accepted a human override and when. That audit trail matters in disputes.

extraction_results(
  doc_id, schema_v, extractor_v, payload_json,
  confidence, status, created_at
)
unique(doc_id, schema_v, extractor_v)

When reprocessing millions of docs after a prompt fix, use batch APIs, checkpoint progress by doc_id and canary on one vendor before the full fleet. A bad reprocess that auto-commits wrong totals is worse than leaving old extracts in place.

Do not flip 1M docs to a new schema in one night. Shadow-extract in parallel, compare field disagreement rates, auto-commit only where old and new agree and validators pass, send disagreements to review. Raise auto-commit share as audit sampling stays clean. Keep the previous extractor version runnable for emergency reprocessing. Schema migrations are much like database migrations, except the "rows" are PDFs you cannot trivially rewrite.

Walking through a date-format shift

A prompt change starts emitting 04/10/2026 instead of 2026-10-04. The schema regex rejects the new strings. CI on the golden set fails before reprocess. If you had overwritten extraction_results in place, you would have lost the last good row. With versioned rows, you keep extractor_v=12 accepted and leave extractor_v=13 in failed_validation until the prompt is fixed.

Duplicate submissions: hash bytes and use an idempotent sink so the same PDF does not post twice.

Common mistakes

• UPDATE in place on the extraction row. History and audit disappear.
• Auto-committing a new version that disagrees with a human-approved critical field.
• Reprocessing the full fleet before a one-vendor canary.

Check yourself

1. Prompt v13 disagrees with a human-approved total_cents. What happens? Flag for re-review. Do not silently replace the approved value.
2. Why unique(doc_id, schema_v, extractor_v)? The same document can have many extracts. Each version is a row, not an overwrite.

Key takeaways

• Version extractions. Reprocess by inserting, not overwriting.
• Idempotent downstream writes. Key on doc_id + schema + extractor.
• Human-approved critical fields are sticky until a person says otherwise.
• Canary one vendor. Keep the previous extractor runnable.

Further learning

The pipeline map sits on document extraction.