Guide

LLM fundamentals

Batch LLM APIs

Submit a file of requests, get results hours later, typically at about half the sync token price. For overnight work, not chat.

27 of 155 · 2 min read

OpenAI, Anthropic and others offer batch endpoints: submit a file or list of requests, get results within a window (often 24 hours), typically at about 50% lower token prices, with a separate higher rate-limit pool.

Ideal for overnight document extraction, eval suite runs, embedding backfills and offline classification. Not ideal for interactive chat.

Concept

Before you start: interactive traffic wants low TTFT. Offline jobs want cheap tokens. Share prompt templates and schemas between the two paths so offline jobs do not drift.

Walking through 10 million documents

10M documents × 2,000 input + 200 output tokens. At illustrative online prices $3 / $15 per million:

online = 10e6 × (2000 × 3 + 200 × 15) / 1e6
       = 10e6 × (6,000 + 3,000) / 1e6
       = $90,000
batch  ≈ 50%  →  $45,000

Same prompts, next-day latency, half the bill, if your product can wait.

Design for idempotency: every job id maps to one document version so retries do not double-write. See retries for the same idea on the online path.

Map, then merge

Long jobs should not be one giant online call. Prefer:

  • Map-reduce: many short calls over chunks, then a merge call
  • Batch API for the map stage, online API for the interactive merge
  • Agent steps, if a human is waiting, each tool round as a checkpoint

This keeps failure domains small (retry one chunk), improves cache reuse on shared instructions, and avoids a single timeout taking down a 10-minute generation.

Common mistakes

  • Sending user-visible chat through a batch API. You save money and destroy the product.
  • Letting batch prompts drift from online prompts. Share templates and schemas.
  • No job id. A retry double-writes the extraction.

Check yourself

  1. 10M docs, 2,000 input + 200 output, $3 / $15 per million online. About how much is batch at 50% off? $45,000. Online is $90,000.
  2. A user is waiting on a chat reply. Batch or sync? Sync. Batch is for work that can wait hours.

Key takeaways

  • Batch APIs roughly halve token prices for offline work.
  • Use them for extraction, evals, backfills and reports.
  • Share prompts with the online path.
  • Idempotent job ids so retries do not double-write.

Further learning

OpenAI Batch APILearn how to use OpenAIOpenAI Developers

Anthropic has an analogous Message Batches API. Back to LLM APIs. For the interactive cost model, see latency and cost.

Working with LLM APIs

Messages, roles, streaming, stop reasons, errors, rate limits and batch APIs. The plumbing of every LLM app.

Latency and cost optimization

Streaming, routing, cascades, batching and parallelism: making LLM apps fast and affordable without dropping quality.

Evaluating LLM apps

A fixed set of inputs plus a way to score outputs, so you can tell whether a change helped.