Observability and reliability
Structured logs
A structured log is one machine-readable event with named fields. The few fields that always pay off are the request ID, trace IDs, a stable route, and a level.
A structured log is an event written as JSON, with named fields such as status, duration_ms and merchant_id. Humans can still read it in a viewer. Machines can also filter it, count it and join it, which a plain text line does not allow. A single line like "payment declined for acme" can be found only by grep. The same event as a structured record can be grouped by merchant, route or error code in one query.
The big idea
Think of a receipt versus a note scribbled on a napkin. The napkin says the payment failed. The receipt has the amount, the store, the time and the card type, in fixed places. You can sum a stack of receipts. You cannot sum a stack of napkins.
Walking through it
A single structured log line for one failed payment:
{
"ts": "2026-03-14T09:26:53.412Z",
"level": "error",
"service": "checkout",
"msg": "payment declined",
"request_id": "01J8X2K9",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"merchant_id": "acme",
"http.route": "/v1/checkout",
"http.status": 502,
"duration_ms": 1842,
"error.code": "upstream_timeout"
}Four fields pay for themselves every time:
request_id: one ID set at the edge, such as the load balancer, and passed through every service. Without it, you cannot rebuild one user's path across services.trace_idandspan_id: when you also emit traces, the same IDs let you jump from a log line to the trace waterfall and back.- A stable
http.route: prefer/users/:idto/users/48291. Raw paths explode the index and leak IDs. level:debug,info,warnorerror. Production usually runs atinfo. Raise todebugon one canary or for a short window during an incident.
Stable field names matter as much as the fields. If one service writes status and another writes http.status, a query for failed requests misses half of them.
Common mistakes
- Writing free text with the values embedded. "Declined for acme after 1842 ms" cannot be grouped. Put the values in fields.
- Using the raw URL as the route. It makes every ID a separate value.
- Leaving out the request ID. Without it, the logs of one request cannot be connected.
Check yourself
- Why is
http.route=/users/:idbetter than the raw path? It groups all requests to the same endpoint into one value. The raw path makes a new value for each user. - Which fields let you move from a log line to its trace?
trace_idandspan_id, copied onto the log line.
Key takeaways
- A structured log is an event with named fields that a query tool can filter and group.
- Always include a request ID, trace IDs, a stable route, a status, and a level.
- Use one name for each field across every service.
Further learning
The cost and privacy rules for logs are in logs, cost and privacy. The big picture is in observability.
OpenTelemetry documentationOpenTelemetry, also known as OTel, is a vendor-neutral open source Observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, and logs. As an industry-stan…OpenTelemetry
Related
Observability: logs and traces
Structured logs, distributed tracing and OpenTelemetry, so you can ask new questions of a running system instead of guessing which dashboard to open.
Logs, cost and privacy
Logs are often the biggest observability bill, and they can leak secrets. Sample routine traffic, keep every error, never log secrets or PII, and do the volume math before you turn on debug logging.
Distributed tracing
A trace is the full path of one request across services, built from timed spans. Trace context has to be passed on at every hop, or the waterfall breaks into pieces.