Logs, cost and privacy

Logs are often the largest line on an observability bill, and they are a privacy risk too. A busy service writes a lot of data, and the data can contain passwords, tokens and personal details. Control the cost and the exposure together, with three moves: sample the routine traffic, keep every error, and stop sensitive values at the source.

The big idea

Think of a security camera system. You do not keep every second of footage forever. You keep every incident, sample the quiet hours, and you never record the inside of a bathroom. Logs need the same rules.

Walking through it

Do the volume math before you turn anything on. Say a service handles 5,000 requests per second, and each request writes a 1.5 KB log line.

requests per day  = 5,000 × 86,400       = 432,000,000
log volume per day = 432,000,000 × 1.5 KB = about 648 GB

Now apply a sampling policy. Keep 5% of the routine traffic, and keep every error. Errors are about 0.5% of traffic.

kept sample = 0.05 × 648 GB  = 32.4 GB
kept errors = 0.005 × 648 GB = 3.24 GB
total       = about 36 GB per day, before compression

That is still large, but it is about an eighteenth of the original volume, and it keeps every failure. Debugging almost always needs the failures and the slow requests, not a sample of the boring ones. Keep 100% of slow requests too, as long as your latency target defines "slow".

The privacy rules are short:

• Never log secrets or personal data. That covers passwords, tokens, full card numbers, session cookies and health data. Hash or tokenize a user ID if you need a join key.
• Bound the size of each message. Dumping whole request bodies "just in case" is how credentials leak and retention limits get blown.
• Scrub at the source. Scrubbers in the log agent help, but the safest place to stop a value is where the log line is written.
• Drop debug logs in production unless a flag turns them on for a short window.

Common mistakes

• Estimating cost after enabling debug logs. Do the volume math first.
• Sampling errors along with successes. Keep every error. It is the cheapest part of the data to keep.
• Assuming the agent will scrub everything. Stop sensitive values where they are written.

Check yourself

1. At 5,000 requests per second and 1.5 KB each, roughly how much is logged per day? About 648 GB per day, before compression.
2. Why keep every error when sampling everything else? Errors are the rare, valuable events for debugging, and they make up a small share of the volume.

Key takeaways

• Do the volume math before you enable verbose logs in production.
• Sample routine successes. Keep every error and every slow request.
• Never log secrets or personal data, and bound the size of each message.

Further learning

The fields to log are in structured logs. The tracing side, which has its own sampling, is in distributed tracing.

OpenTelemetry documentationOpenTelemetry, also known as OTel, is a vendor-neutral open source Observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, and logs. As an industry-stan…OpenTelemetry