Observability: logs and traces

A checkout request fails for one merchant. Latency for everyone else looks fine. Your dashboards show green CPU, healthy error rates and a p99 that barely moved. The failure is real, but monitoring cannot see it, because nobody predicted this question when the graphs were built.

That gap is why observability is a separate idea from monitoring. Monitoring asks the questions you wrote down in advance, such as whether the error rate is above 1%. Observability lets you explore the system's output, which means logs, traces and detailed events, until the cause is clear.

Walking through the topic

The pages below build from the events you emit, to how they link across services, to the tooling that carries them.

Common mistakes

• Putting every dimension into metric labels. High-cardinality values melt the metrics store. Use logs and traces for them.
• Logging secrets or personal data. Treat the log pipeline as a data store with the same controls as the database.
• Missing the request ID. Without it, you cannot follow one request.
• Leaving spans out of database and queue calls. The gap shows up as "the app is slow" with no cause.

Instrumentation checklist for a new service

Before the first production deploy:

1. HTTP middleware emits request spans and structured access logs, with route, status, duration and request ID.
2. Outgoing HTTP and gRPC clients propagate traceparent and record client spans.
3. Database and cache clients are wrapped once in a shared library, so no handler has to remember.
4. Queue producers and consumers copy the trace context into message metadata.
5. The collector endpoint and credentials come from platform config, not from a vendor agent hardcoded in the app.
6. A staging smoke test produces a visible end-to-end waterfall for one synthetic request.

Skipping step 3 is how incidents end with "the app looks fine, but the database is a black box". Shared libraries beat copy-pasted instrumentation.

Key takeaways

• Monitoring answers planned questions. Observability answers new ones from detailed events.
• Structured logs need a request ID, trace IDs, a stable route, a status and a level.
• Traces are trees of spans, linked by traceparent and message metadata.
• OpenTelemetry lets you send everything to a collector, and choose the backend later.
• Use metrics to page, and logs and traces to explain.

Further learning

Start with the first page in the table, then continue down the list. The alerting side of the picture is in metrics and alerting.

Google's Dapper paper is the origin story of production tracing. The OpenTelemetry documentation is the reference for SDKs and the collector.

Dapper, a Large-Scale Distributed Systems Tracing Infrastructureresearch.google OpenTelemetry documentationOpenTelemetry, also known as OTel, is a vendor-neutral open source Observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, and logs. As an industry-stan…OpenTelemetry

Charity Majors, Liz Fong-Jones and George Miranda walk through the observability thesis in this talk. It is worth watching for the mental model, not the tooling pitch.