Trace sampling

At scale, keeping every trace costs too much to store. Sampling means keeping some traces and dropping the rest. The question is when to make the decision. Head sampling decides at the start of the request. Tail sampling waits until the request finishes, and then decides based on what happened.

The big idea

Think of a bouncer at the door versus a reviewer at the end of the night. The bouncer picks guests before they know how the evening goes, so he might turn away the one person who would have caused trouble. The reviewer watches the whole night and keeps the footage of the incidents. The reviewer costs more, because the footage has to be held until the night is over.

Walking through it

Head sampling is the simple option. At the edge, keep one request in N, for example one in 100. The decision is cheap, and nothing needs buffering. The cost is that the rare failure you need may be one of the requests that was dropped.

Tail sampling buffers the whole trace briefly, then keeps the traces that matter:

• every trace with an error,
• every trace slower than the latency target,
• and a baseline of ordinary successes, for comparison.

Take the service from the logs page, at 5,000 requests per second. A 1-in-100 head sample keeps 50 traces per second, and about 1% of the failures. A tail policy that keeps every error, every trace over the latency target, and 1% of the rest, keeps every failure. The cost is the memory needed to hold traces until they finish, and a collector that can handle the buffering.

Common mistakes

• Using only head sampling for errors. The failure you want may be dropped before anyone knows it failed.
• Sampling too little to debug. Check that the sample still contains the slow traces you would open in an incident.
• Building tail sampling without the memory for it. Buffering needs room for every in-flight trace.

Check yourself

1. Why can head sampling miss the trace you need? The decision is made before the request finishes, so the request that later fails may already have been dropped.
2. What does tail sampling cost that head sampling does not? Memory and collector capacity to hold each trace until it finishes.

Key takeaways

• Head sampling decides at the start: cheap, simple, but it can drop the rare failure.
• Tail sampling decides at the end: it keeps errors and slow requests, at the cost of buffering.
• A common start is head sampling plus always keeping errors.

Further learning

The trace model is in distributed tracing. The collector that runs tail sampling is part of OpenTelemetry.

OpenTelemetry documentationOpenTelemetry, also known as OTel, is a vendor-neutral open source Observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, and logs. As an industry-stan…OpenTelemetry