Document understanding

Document QA is the multimodal workload that pays the bills. A digital PDF already has a text layer. A scan is a picture of a page. A bar chart's meaning is visual: the text layer never said which quarter grew fastest. The job is to pick a path that matches the file you actually have.

The big idea

Four paths:

1. Native PDF / document APIs — the provider parses text layers and renders pages for vision. Best when PDFs are digital (real text).
2. Page rasterization + vision — render pages to images. Good for scans. Expensive.
3. Classic OCR → text LLM — still valid for high-volume, template-heavy paper. You lose layout and chart semantics unless you keep bounding boxes.
4. Hybrid — OCR for body text, vision for figures and signatures.

Tables and charts are where pure OCR fails. Vision-language models can answer "which quarter grew fastest?" when no sentence in the file contains that claim.

Common failures on real pages:

• Multi-column layouts read in the wrong order
• Footnotes mixed into totals
• Light gray text on screenshots
• Stamps and handwriting over printed totals

Walking through a path choice

A 10-page vendor contract arrives as a digital PDF. The text layer has every clause. Sending all ten pages as images wastes vision tokens and still risks a worse reading order. Use the native document API, or extract the text layer and send vision only for the signature page and the one embedded chart.

A warehouse receipt arrives as a phone photo of a carbon copy. There is no text layer. Rasterize, deskew, crop to the receipt, then run vision. Classic OCR may read the printed header and miss the handwritten total.

Image generation and editing (drafts, inpainting, product-shot consistency) are usually separate endpoints from the chat LLM, even when the vendor UI blends them. Latency, safety filters and pricing differ. Do not assume your text LLM "can draw" unless the API documents image output.

For anything that hits a ledger, pair vision with structured outputs and a deterministic check: total_cents should match the sum of line items plus tax when those lines are present. Schema validity is not a correct total.

Common mistakes

• Rasterizing every digital PDF. The text layer was already there.
• OCR-only on charts. The meaning is in the bars, not the axis labels.
• Trusting a valid JSON object. A well-shaped total can still be a misread.

Check yourself

• A digital 10-page PDF with one chart: what do you send to vision? The chart page, maybe the signature. Not ten rasterized pages.
• Why does OCR fail on "which quarter grew fastest?" That sentence is not in the text layer. The answer is in the picture of the bars.

Key takeaways

• Prefer native text layers. Use vision for scans, figures and signatures.
• Hybrid OCR plus vision matches mixed corpora.
• Charts and tables need vision. Arithmetic checks catch misreads that schemas miss.

Further learning

Next, see how audio and video become tokens in audio and video.