Multimodal evals

Text eval harnesses are not enough. Multimodal fixtures must include the binary assets (images, PDFs, audio) under version control or object storage with stable ids. Demo photos are a different task from a motion-blurred phone snap under warehouse lighting.

The big idea

Score:

• Exact or fuzzy field match for extraction (totals, dates, IDs)
• VQA accuracy on a labeled question set
• Human preference for descriptive tasks (alt text, UI guidance)
• Latency and token cost per successful task

Known weak spots, even on strong models: counting many small objects, precise spatial relations ("left of the icon that is second from the right"), tiny text in compressed JPEGs, rotated or skewed scans, out-of-distribution UIs, and math in diagrams without vector access to the glyphs.

Mitigations: crop, enhance contrast, ask for a two-step "transcribe then reason," and use tools (barcode decoder, PDF text layer) instead of forcing vision to do everything.

Walking through receipts and screenshots

Receipt path. Mobile users photograph receipts. Ops needs vendor, date, total and tax in JSON.

1. Client uploads JPEG (often 3000×4000).
2. Server downscales long edge to 1568–2048 px.
3. Optional: deskew and crop to the receipt box.
4. Multimodal call with a strict schema:
   vendor, date, currency, total_cents, tax_cents, line_items[], confidence.
5. Validate. Low confidence → human queue.
6. Cross-check: total_cents ≈ sum(line_items) + tax when lines exist.

Cost sketch (orders of magnitude):

Image tokens after resize: ~1,500
Text prompt + schema:        ~400
Output tokens:               ~200
At $3 / MTok in, $15 / MTok out:
  input  ≈ 1,900 × $3e-6  ≈ $0.0057
  output ≈ 200  × $15e-6 ≈ $0.0030
  ≈ $0.009 per receipt before retries

At 50k receipts/day, raw model cost is on the order of hundreds of dollars/day. Fine if automation replaces minutes of human entry. Catastrophic if you send uncropped 12 MP images and triple the tokens.

Screenshot path. Accept PNG. Do not JPEG-compress UI text if you control the encoder. Prompt: list visible error text verbatim, then explain the likely cause using only on-screen evidence plus a short product glossary. Eval: exact match on the extracted error string; a rubric on the explanation.

Ship a thin ops console early: the resized image the model actually saw, the fields, the checker errors and the human correction. Measure three rates weekly: schema-valid responses, checker pass rate and human-override rate. If overrides climb while schema validity stays perfect, the grammar is fine and the vision or the preprocessing is not.

Common mistakes

• Eval sets of clean studio photos. Production is blur, glare and odd crops.
• Scoring only schema validity. $50.00 of line items with a $48.07 total is a miss.
• No ops console. You cannot tell a resize artifact from a model limit.

Check yourself

• Line items sum to $50.00 and total_cents is 4807. Did structure fail? No. The checker should fail. The shape was valid.
• Overrides rise, schema validity stays 100%. Where do you look? Preprocessing and vision quality, not the grammar.

Key takeaways

• Fixtures include the real binaries. Ugly inputs belong in the set.
• Score fields, cost and latency, not only parse success.
• Pair structured outputs with arithmetic checks and a human queue.
• Show reviewers the pixels the model saw.

Further learning

Claude docs: VisionClaude's vision capabilities allow it to understand and analyze images, opening up exciting possibilities for multimodal interaction.Claude Platform Docs

Back to the map: multimodal inputs and outputs.