Kubernetes in production

Getting a Deployment to Ready in a sandbox is the easy part. Production Kubernetes fails in other ways. Pods are oversized or undersized. Probes restart healthy processes. A node drain evicts every replica at once. A rollout ships a bad image faster than anyone notices. This topic is the operational layer on top of core concepts. It covers how to keep services alive under load, during deploys, and during cluster maintenance.

Walking through the topic

The first two pages cover what a pod is allowed to use and when it counts as healthy. The last four cover how it scales, survives maintenance, rolls out, and gets packaged.

Step	Concept	What it answers
1	Resource requests, limits and QoS	How much may a pod use, and what happens when it goes over?
2	Health probes	When does a pod get traffic, and when does it restart?
3	Autoscaling pods and nodes	How do replicas and nodes grow with load?
4	Disruption budgets and spreading replicas	How do drains and node failures avoid taking everything down?
5	Rollouts and graceful termination	How does a deploy swap pods without dropping requests?
6	Packaging, GitOps and operators	How are manifests packaged, applied and pinned?

Common mistakes

Putting a database check on liveness. A dependency blip then restarts every pod at once.
Setting CPU limits far below the real burst. Throttling looks like slowness, and average CPU hides it.
No PodDisruptionBudget on a customer-facing service. A drain can remove every replica.
Ignoring SIGTERM. Every deploy then drops in-flight requests.
Mutable image tags. A tag can be pushed again. Pin digests.

Production readiness checklist

Before calling a service production-ready on Kubernetes:

Requests and limits are set, and the QoS class is intentional. No BestEffort for serving pods.
Readiness and liveness probes are set, plus a startup probe if boot is slow. Liveness stays shallow.
A PodDisruptionBudget with maxUnavailable, and at least three replicas for HA services.
Replicas are spread across zones and nodes.
An HPA or KEDA with floors and ceilings, and cluster capacity to absorb a scale-up.
Graceful shutdown is tested under real load, with a preStop hook if the mesh needs time.
Images are pinned by digest, scanned, run as non-root, with a read-only root filesystem where possible.
A default-deny NetworkPolicy in the namespace, where the platform supports it.
Dashboards for RED, pod restarts, throttling and OOM kills, with burn-rate alerts from SLOs.
A runbook for CrashLoop, Pending, ImagePull, OOM and a failed rollout.

Key takeaways

Requests drive scheduling and autoscaling. Limits drive throttling and OOM kills.
Readiness removes traffic. Liveness restarts processes. Never mix them.
The HPA scales pods, the cluster autoscaler scales nodes, and KEDA scales on events.
PodDisruptionBudgets and topology spread keep maintenance from becoming an outage.
Graceful termination is part of every zero-downtime deploy.
Package with Kustomize or Helm, deliver with GitOps, and pin digests.

Interview angle

Two questions come up often. "What happens if liveness checks the database?" The answer is a restart storm. "Why is my pod Pending?" Check resources, affinity, taints, PVCs and quota, in that order.

Further learning

Carson Anderson's deconstruction helps when production symptoms point at controllers and networking rather than app code. Rewatch the scheduling and endpoints sections when pods stick Pending or Services go dark.