Working with LLM APIs

Every LLM product collapses to HTTP: send a JSON body of messages, get text or tool calls back. The differences between vendors matter less than getting the plumbing right — streaming, retries, stop reasons, rate limits and key handling.

If you have built ordinary REST clients, most of this will feel familiar: auth headers, idempotent retries, backoff on 429, structured error bodies. What is new is long-lived streams, token-denominated rate limits, probabilistic outputs, and side-effecting tool rounds interleaved with model calls.

Chat APIs are stateless. Each call must include the full conversation you want the model to see. Memory is your problem: you store turns and resend them. Statelessness scales horizontally and is a footgun for cost: every retry and every follow-up pays to re-send the prefix.

Keep API keys on the server. Browser-exposed keys will be stolen. Use your backend as the only caller, with per-user quotas.

Concepts in this section

For wrapping several vendors, see LLM gateways. For product architecture around chat, see LLM chat product.

What to log on every call

Providers return a request ID. Persist it next to your own id. When a user reports "the AI said something insane at 14:32," you need both IDs. Also store the model ID actually served, not just the alias you requested. Aliases move.

Every serious provider returns a usage block: input tokens, output tokens, and often cache creation/read tokens. Persist them keyed by request, user, feature, model and prompt version.

ts, request_id, tenant_id, feature, model,
input_tokens, output_tokens, cache_read_tokens, cache_write_tokens,
latency_ms_ttft, latency_ms_total, stop_reason, status, error_class

Sample 1% of full prompts into a secure store for eval mining. Do not log all prompts forever in plaintext if they contain PII.

Security basics

• Keys server-side only, rotated, scoped, stored in a secret manager.
• Per-user and per-tenant quotas so one client cannot blow the org TPM budget.
• Allowlists for tools the model may invoke. Never let raw model text become SQL or shell without validation.
• Treat retrieved content as untrusted. See guardrails.

Enterprises often require data to stay in a region. Build region into config next to model ID: regional base URLs, different model availability, batch jobs that must not hop oceans with document contents.

Official SDKs handle SSE parsing, retries and type stubs. Pin SDK versions. Prefer explicit model IDs over "point at whatever is newest." Keep a single module that constructs clients. Feature code should call llm.chat, not scatter keys and retry policy.

A healthy production posture treats the LLM API like any other flaky dependency: budgets, circuit breakers, fallbacks (smaller model or canned response), and SLOs on TTFT and error rate. You can SLO availability and latency even when content quality is measured offline with evals.

Common mistakes

• Keys in the browser. They will be stolen.
• Retrying a tool that already succeeded. Separate "model may retry" from "tool must run once."
• Gateway timeout below p99 generation. Users see failures that are actually truncated successes. Raise the gateway timeout above p99, keep an app-level deadline, and stream so load balancers see active bytes.
• Assuming vendors are drop-in compatible. Tool schemas, system prompt placement and multimodal parts differ. Re-eval on every swap.

Key takeaways

• Chat APIs are stateless. You own history, memory and retries.
• Stream for UX. Always branch on stop reason.
• Retry 429 and 5xx with jitter. Never blindly retry non-idempotent tools.
• Batch APIs roughly halve cost for offline work.
• Keep keys server-side. Quota per user. Log usage and request IDs.

Further learning

Claude API documentationClaude is a highly performant, trustworthy, and intelligent AI platform built by Anthropic. Claude excels at tasks involving language, reasoning, analysis, coding, and more.Claude Platform Docs

OpenAI Batch APILearn how to use OpenAIOpenAI Developers

LLM chat product