Testing in production

Pre-prod tests never see every config, data shape, or traffic pattern. That is not an argument against the suite. It is why you complement it with production safety nets.

These do not replace tests. They catch the unknowns the suite cannot invent. Pair them with observability so a canary failure is diagnosable.

The three nets

• Feature flags — ramp exposure; abort without a full rollback.
• Canary deploys — 1% of traffic on the new version; watch error budgets.
• Synthetic monitoring — scripted critical journeys from outside the system, continuously.

"Canary + flag is how we ship Friday without weekend firefighting" is the translation that funds this work with non-engineers.

Walking through checkout

Checkout total with discount-before-tax:

1. Unit-test the pure checkoutTotalCents with a table of (subtotal, discount, tax, flag) → expected cents.
2. Integration-test that posting /checkout persists orders.status = paid against real Postgres.
3. One E2E smoke: card payment in staging with a test Stripe key.
4. In production: flag the new path, canary 1% of traffic, watch the error budget, keep a synthetic that hits checkout from outside.

That mix covers the risky math, the wiring, the full path, and the unknowns — without 40 browser tests.

Build a lightweight test plan into larger tickets before coding: which behaviors are unit vs integration vs E2E, what is explicitly out of scope, and what production signal confirms success. Ten minutes of planning prevents a feature that ships with only happy-path unit tests and a prayer. Revisit the plan in code review. If the risky path has no automated net, that is a blocking comment, not a nit.

When you inherit a pricing change with no integration test, say it in outcome language: "Without an integration test on invoicing, every pricing change has a real chance of a Sev-2; the last one cost Y." Same facts fund characterization tests when you touch legacy.

Common mistakes

• Flags without a watch. A dark launch nobody observes is not a net.
• Canary without observability. You will not know why the 1% failed.
• Synthetics that only hit /health. Script the journey users pay for.
• Treating production nets as a substitute for the suite. They extend it.

Check yourself

1. You want to ship a checkout change on Friday. What is the production net? Flag plus a 1% canary on the error budget, not hope and a full rollback plan.

2. A synthetic only hits /health. What should it do instead? A scripted critical journey — checkout, login, the path that makes money.

Key takeaways

• Flags, canaries, and synthetics extend confidence into production.
• Watch error budgets on the canary; abort without a full rollback.
• Write a ten-minute test plan that names the production signal.
• Translate the risk into incidents and weekend firefighting so the work gets funded.

Further learning

Back to the map on testing strategy.