Infrastructure, cloud and SRE

Sections: 6
Pages: 162
Reading time: ~7 h
Take first: Foundations

What you will be able to do

Debug a misbehaving Linux host or container with the standard toolkit and the USE method.
Design a VPC with public and private subnets, load balancers, NAT and private connectivity, and explain each hop.
Package services as containers and run them on Kubernetes with sane resources, probes, autoscaling and rollouts.
Provision infrastructure with Terraform and ship it through a CI/CD pipeline using canary or blue-green deploys.
Instrument a service with logs, metrics and traces, define SLOs and burn-rate alerts, and lead an incident to a blameless postmortem.
Apply least-privilege IAM, manage secrets with KMS/Vault, and reduce a cloud bill with concrete levers.
Explain how GPU clusters are provisioned and scheduled for ML workloads.

Designing a system is half the job; the other half is running it. A service that works on your laptop can still fail in production because of a full disk, a misconfigured security group, a CPU throttle from a cgroup limit, a bad deploy, a missing alert, or a secret that leaked into a container image. This topic is the platform under modern services: from Linux and cloud networking through containers, Kubernetes and Terraform, then the practices that keep services healthy — CI/CD, observability, SLOs, incidents, capacity and disaster recovery — ending with IAM, secrets, cloud cost and GPU infrastructure.

SDE2s are expected to ship and operate their own services. The jump from "works on my machine" to "works at 3 a.m. under load" is mostly infrastructure literacy: knowing which layer is broken, which lever to pull, and which trade-off you're making when you pull it.

Why it matters

Backend work without ops context produces systems that are hard to debug, expensive to run and fragile under change. Interviews for mid-level and senior roles routinely ask about Kubernetes probes, VPC design, canary deploys, SLOs and postmortems — not because everyone is an SRE, but because owning a service means owning its failure modes.

This topic sits next to system design: design picks the boxes; infra keeps the boxes alive. It builds on foundations (processes, networking, DNS, TLS, profiling) and feeds AI and inference work that runs on the same clouds, clusters and GPUs.

Who it's for

You're comfortable writing services and have deployed something (a VM, a PaaS app, or a container), but you haven't designed a VPC, tuned Kubernetes resources, written a Terraform module, defined an SLO, or led an incident. No prior SRE title is required. Concepts from Linux and networking are introduced as they appear; deeper CS prerequisites link back to foundations.

How to study it

Pace. 162 lessons, about 7 hours of reading. Linux and networking first, then containers and Kubernetes, then cloud and delivery, then observability and reliability, then security, cost and GPUs. Don't skip the ops practice units — they're what turn tool knowledge into judgment.
Hands-on. Theory sticks when you've broken something. Good projects: a three-AZ VPC with a private app and public load balancer; a Dockerfile you shrink and harden; a local Kubernetes cluster with probes, HPA and a canary; a Terraform module for an environment; OpenTelemetry traces through two services; an SLO with burn-rate alerts; a chaos drill that kills a pod or AZ and measures RTO.
Use production signals. Prefer real dashboards, journalctl, kubectl describe, flow logs and load-test graphs over slides. When a lesson names a tool (iostat, Reachability Analyzer, Prometheus histograms), run it once on a throwaway box or cluster.
Further learning. Each lesson ends with primary sources — AWS/GCP docs, Brendan Gregg, Google SRE books, CNCF specs. Read at least one per unit; vendor tutorials rot, mechanisms don't.

Roadmap

The numbered list is a shorter path. The sections below are every lesson in the topic.

Linux and networking

The host and the network path every packet takes before your code runs.

Linux for engineers: shell, processes, systemd, permissions, disks, network commands, SSH vs session managers.
Linux performance analysis: USE method, 60-second checklist, CPU/memory/disk/network, profiling, container throttling.
Cloud networking and VPCs: CIDR, subnets, NAT, security groups, peering, PrivateLink, reference three-tier VPC.
Load balancers, DNS and the edge: ALB/NLB, Route 53 policies, ingress, TLS, WAF, CDN, shifting traffic between regions.

Containers and orchestration

Packaging and scheduling the unit of deploy.

Containers from the inside: namespaces, cgroups, images, Dockerfiles.
Kubernetes core concepts: control loop, Pods, Deployments, Services, ConfigMaps.
Kubernetes in production: requests/limits, probes, HPA, PDBs, rollouts, common outages.
Serverless and managed runtimes: Lambda, Cloud Run, cold starts, cost crossover.

Cloud and infrastructure as code

Provider building blocks and declaring them in code.

Cloud primitives on AWS and GCP: regions, zones, compute, storage, databases, messaging.
Infrastructure as code with Terraform: state, plans, modules, environments, safe changes.
Configuration, environments and feature flags: twelve-factor config, parity, secrets injection, flags.

Delivery

Getting changes to production without waking the planet.

Continuous integration and the software supply chain: build, test, artifact, scan, sign.
Deployment strategies and safe releases: rolling, blue-green, canary, rollback, DB migrations.
Platform engineering: internal platforms, golden paths, infra as product.

Observability and reliability

Seeing the system, defining "good enough," and recovering when it isn't.

Observability: logs and traces: structured logs, tracing, OpenTelemetry.
Metrics, dashboards and alerting: golden signals, Prometheus, histograms, pages that matter.
SLIs, SLOs and error budgets: user-facing reliability numbers and speed vs stability.
Incident response: roles, severity, mitigate first.
Blameless postmortems: timelines, factors, action items that finish.
Designing on-call: rotations, runbooks, toil, humane paging.
Capacity planning and load testing: forecasts, limits, headroom.
Disaster recovery and chaos engineering: backups, RPO/RTO, failover drills.

Security, cost and GPUs

Who can do what, what it costs, and how ML capacity is provisioned.

IAM and least privilege: identities, roles, workload identity, zero trust.
Secrets management and encryption: KMS, Vault, data at rest and in transit.
Cloud cost and FinOps: bill anatomy, levers that cut 30–50%.
GPU infrastructure: instance types, interconnects, scheduling, sourcing capacity.

Outcomes

After working through these pages you should be able to debug a slow Linux host or container with the USE method; design a multi-AZ VPC and explain each hop; run Kubernetes with sane resources, probes and rollouts; provision with Terraform and ship through CI/CD with canaries; instrument with logs/metrics/traces and define SLOs; handle incidents and write useful postmortems; apply least-privilege IAM and cut a cloud bill with concrete levers; and explain how GPU clusters are provisioned for ML workloads.

Course resources

Google's SRE books are the canonical reliability texts; start with the chapters on SLOs, monitoring and incident response.

Site Reliability Engineering (Google)Go through the complete table of contents of sre Google book, outlined are the key topics and insights covered in this essential resource for SRE professionals.sre.google The Site Reliability WorkbookThe site reliability workbook table of contents, navigate key SRE concepts of sre and practical strategies for building reliable, scalable systems.sre.google

Brendan Gregg's Linux performance material is the reference for host-level debugging.

Linux Performance Tools, Brendan Gregg, LinuxCon Europe 2014DiaMon Workgroup

Linux Performance (Brendan Gregg)A collection of documents, slides, and videos about Linux performance, mostly created by Brendan Gregg, and with a focus on performance analysis.brendangregg.com

For Kubernetes, the official docs plus a short cluster on your laptop beat any slide deck.

Kubernetes documentationKubernetes is an open source container orchestration engine for automating deployment, scaling, and management of containerized applications. The open source project is hosted by the Cloud Native Computing Foundation.Kubernetes

AWS Well-Architected FrameworkThe AWS Well-Architected Framework provides guidance to help developers build and deploy applications faster, lower risk, and make informed decisions following AWS best practices.Amazon Web Services, Inc.

Linux and networking

Shell pipes and streams

How programs talk through stdin, stdout and stderr, and how pipes chain small tools into one answer to a production question.

Processes and signals

What a process is, how to read its state, and how signals and exit codes tell you why it stopped. Exit 137 and 143 come up constantly.

Permissions and capabilities

How Linux decides who can read, write or enter a file, why "permission denied" often comes from a parent directory, and how to grant one power instead of root.

Disk space and mounts

"No space left on device" has three different causes, only one of which df -h shows. Also how disks attach to folders with mounts.

Linux for engineers

The parts of Linux you touch in production, from shell pipelines and systemd services to permissions, disks, network debugging commands and safe remote access.

Safe bash scripts

Bash fails quietly by default. A few lines at the top of every script turn unset variables and hidden failures into loud errors.

Graceful shutdown

How orchestrators stop a process with SIGTERM, then SIGKILL after a grace period, and how a service drains in-flight work inside that window.

systemd and journald

How systemd runs a service as a unit with restarts and hardening, and how the journal captures its logs so you can query them by service, time and priority.

Debugging connectivity

When A cannot reach B, check each layer from the bottom up and write down what each one said. Timeouts and refusals point to different causes.

The USE method

A checklist for finding a bottleneck: for every resource, check utilization, saturation and errors before you guess at the application.

CPU load and the run queue

What load average really counts, how the run queue shows CPU saturation, and why steal time in the cloud is a problem you cannot fix inside the guest.

Memory, page cache and OOM

Why "free" memory looking low is normal, how to read available memory, and why swap and OOM kills are the two signals that memory is actually short.

Linux performance analysis

The USE method and the 60-second checklist for finding why a box is slow, covering CPU, memory, disk, network and the container limits that look like application bugs.

The 60-second checklist

Nine standard commands that give a fast USE pass over a Linux host in about a minute, and how to read what each one shows.

Disk and network saturation

How to read disk utilization, await and queue depth, and how network retransmits, drops and listen overflows show a path that is overloaded.

Profiling with perf and eBPF

Once a resource is confirmed as the bottleneck, sampling profilers and eBPF tools show which functions burn the CPU or wait on I/O, without rewriting the app.

CIDR and VPC basics

A VPC is an isolated address space. CIDR notation says how big each part of it is. Plan the ranges before you create anything, because overlaps are hard to undo.

Cloud networking and VPCs

VPCs, subnets, routing, NAT, security groups and private connectivity in AWS and GCP, plus a three-AZ reference architecture and how to debug when A cannot reach B.

Routes, internet gateways and NAT

A route table picks the next hop for each packet. Public subnets reach the internet through a gateway. Private subnets get outbound access through NAT, and endpoints skip NAT for cloud APIs.

Security groups and NACLs

Security groups are stateful firewalls on each instance and are the main control. Network ACLs are stateless filters on each subnet. Know which one you are editing, and reference groups instead of ranges.

Connecting VPCs and private endpoints

VPC peering links two networks, a transit hub links many, and PrivateLink publishes one service. Endpoints keep traffic to cloud APIs off NAT and the internet.

Load balancers, DNS and the edge in the cloud

Configuring the traffic layer, from DNS and cloud load balancers through TLS, WAF and CDN to a regional traffic shift. Start here for the map of edge pages.

Cloud load balancers

What a cloud load balancer does, how ALB, NLB and Gateway LB differ, why health checks can take the whole fleet down, and where Ingress fits inside the cluster.

DNS routing and TTL

How DNS routing policies steer traffic between endpoints, and why a TTL sets the real speed of a failover. Covers weighted, latency and failover routing, and the cache trap.

TLS, WAF and CDN at the edge

Where TLS ends, how certificates are renewed, and how a WAF, DDoS protection and a CDN sit at the outermost edge to protect origin servers and cut latency.

Shifting traffic between regions

A step-by-step runbook for moving users from one region to another with weighted DNS, health checks and a fast rollback. Walks through the numbers and a real rollback decision.

Containers and orchestration

Containers vs virtual machines

A virtual machine boots its own kernel. A container shares the host kernel. That one difference explains the speed, the size and the security trade-off.

Namespaces and cgroups

A container is a Linux process with two kernel features applied. Namespaces control what it can see, and cgroups control what it can use.

Containers from the inside

Namespaces, cgroups and layered images - what a container really is, how images are built and shipped, and how to write small, fast, secure Dockerfiles.

Image layers and digests

An image is a stack of read-only layers named by their content hash. Learn why deleting a file does not shrink an image, and why production should pin digests, not tags.

Writing good Dockerfiles

Order Dockerfile steps so the cache works for you, build in stages so compilers stay out of the final image, and run as a non-root user.

Container security

A container is a convenience boundary, not a strong security boundary. Defend it in layers, with minimal images, non-root users, read-only filesystems, dropped capabilities and no secrets in layers.

Shrinking an image

A worked example that takes a 1.2 GB Python image under 150 MB. Measure each layer first, then fix the largest costs one at a time.

The control plane and the reconciliation loop

The parts that run a Kubernetes cluster, and the loop that compares what you asked for with what is running and closes the gap. Includes the path a deployment takes to become running pods.

Kubernetes core concepts

The control loop at the heart of Kubernetes, the objects you use every day, and how a request reaches a pod. Start here for the map of Kubernetes pages.

Pods and workload objects

A pod is the smallest unit you deploy. Workload objects wrap pods with rules for how many run and what happens on failure. Use the one whose rule matches your app.

Services and DNS inside the cluster

How a Kubernetes Service gives a set of pods one stable name and virtual IP, how DNS and readiness route traffic, and what a request passes through on its way to a pod.

Config, storage and labels in Kubernetes

Where pods get their settings, secrets, disks and identity, and how labels and namespaces organize everything. Includes the storage rules that decide where a pod can run.

Container throttling and cgroups

Containers share the host's kernel but live under cgroup limits. CPU throttling and memory limits can make a healthy node look like a slow app, and the fix starts with the numbers.

Debugging CrashLoopBackOff

A step-by-step order for finding out why a container keeps crashing, including exit codes, previous logs and bad probes. Use it before you guess.

Resource requests, limits and QoS

Requests decide where a pod fits. Limits decide what happens when it uses too much. CPU over its limit is throttled, while memory over its limit is killed.

Health probes

Startup, readiness and liveness probes decide when a container gets traffic and when it gets restarted. Liveness should be shallow, because a deep check can turn one dependency blip into a full restart storm.

Kubernetes in production

Requests and limits, probes, autoscaling, disruption budgets, packaging and the mistakes that cause outages.

Autoscaling pods and nodes

The horizontal pod autoscaler changes replica counts, the cluster autoscaler or Karpenter adds nodes, and KEDA scales on events. Choose the metric that matches how the service actually waits.

Disruption budgets and spreading replicas

A PodDisruptionBudget limits how many pods a drain or upgrade can take down at once. Topology spread and anti-affinity keep replicas on different nodes and zones.

Rollouts and graceful termination

How a rolling update swaps pods safely, and the order of events when a pod is deleted. Traffic has to stop before the process does, and requests longer than the grace period get cut off.

Packaging, GitOps and operators

Kustomize, Helm and GitOps decide how Kubernetes manifests are packaged, reviewed and applied. Pin images by digest, and treat operators as code you still need to understand.

Three kinds of serverless

Functions, containers and managed platforms all count as serverless, but they hand you different things to ship and hide different amounts of the server.

Serverless and managed runtimes

Lambda, Cloud Run and friends - what you give up, what you gain and the cost crossover versus always-on containers.

Cold starts

A cold start is the delay before a new serverless environment can serve its first request. Learn its three parts and the mitigations that actually cut it.

GPUs on Kubernetes

How a GPU becomes a schedulable resource in Kubernetes, through the GPU Operator, tainted node pools and nvidia.com/gpu requests, and how MIG and time-slicing pack more work onto each card.

Serverless event sources and retries

Functions run when events arrive from queues, storage, schedules and webhooks. Those events are delivered at least once, so every handler must tolerate duplicates.

Concurrency and database connections in serverless

Serverless scales by adding environments, and each one can open a database connection. Learn how that arithmetic exhausts Postgres, and what to do about it.

Serverless cost crossover

Serverless is cheap when traffic is spiky and expensive when it is steady and high. Compare the request and GB-second math with an always-on container before you choose.

Cloud and infrastructure as code

Regions and availability zones

A region is a set of isolated data centres. An availability zone is one building block inside it. Design every tier to survive losing one zone, and size the spare capacity for it.

Cloud primitives on AWS and GCP

Regions, zones, compute, storage, managed databases and messaging, how the services map between AWS and GCP, and how to decide what to run yourself.

VMs, spot and autoscaling

Pick a VM family from the bottleneck, buy spot capacity for interruptible work, and let an autoscaling group keep a fleet at a target load.

Storage types

Block, object and file storage differ in who can attach them, how they are priced and how fast they respond. Picking the wrong one is a common and expensive mistake.

Managed databases and messaging

The managed database, cache, queue and streaming services on AWS and GCP, what each one is good at, and the limits that decide your design.

Managed vs self-hosted

For each component, weigh the provider's managed service against running open-source software yourself, using operational burden, total cost and lock-in.

Why infrastructure as code

Infrastructure as code keeps cloud resources in reviewed files. Learn the difference between declarative and imperative setup, and what drift is.

Infrastructure as code with Terraform

Declaring infrastructure in code — providers, state, plans, modules, environments and the habits that keep apply safe.

Terraform plan and apply

The Terraform loop. Write resources, read the plan, then apply it. Learn the building blocks and the plan symbols that flag replacements.

Terraform state and locking

The state file maps your code to real resources. Keep it in a remote backend with locking, and treat import, move and remove as production surgery.

Terraform modules and environments

Modules package a standard into a reusable folder. Separate directories, each with its own state, keep dev and prod apart. Workspaces are useful but easy to misuse.

Safe Terraform changes

Run plans in CI, check policy before apply, and guard stateful resources in the config. Force-new changes need extra care because they can replace data.

Twelve-factor config

Keep config in the environment, not the image, so one tested artifact runs in every environment. Separate build, release and run, and promote the same image digest.

Environments and parity

Most teams need only a few environments. Each long-lived one adds cost and drift. Parity, meaning the same shape, is what makes a staging test count as evidence.

Configuration, environments and feature flags

Twelve-factor config, environment parity, secrets injection, feature flags and the blast radius of a bad config push.

Config layering and validation

Config comes from layers, and the most specific one wins. Load it once at startup into a typed structure, and refuse to boot when a required value is missing or wrong.

Feature flags

A feature flag picks a code path at runtime. It lets you deploy code dark, ramp it safely and separate deploy from release. Keep flags short-lived, because stale flags are production hazards.

Dynamic config and blast radius

Remote config and flags can change every process in minutes. That speed is useful for kill switches, and it is why config pushes cause as many outages as bad deploys. Roll them out in stages, validate them and keep a rollback.

Delivery

Pipeline stages and caching

Order CI checks from cheap to expensive, run independent jobs in parallel, and cache what has not changed. These three levers decide whether a pipeline takes 5 minutes or 40.

Continuous integration and the software supply chain

Fast, trustworthy pipelines - build, test, artifact, scan and sign - plus DORA metrics and supply-chain controls that keep trunk green.

Monorepos and affected builds

In a monorepo, run only the packages a change touches and the packages that depend on them. A dependency graph and a remote cache make that cheap.

Flaky tests

A flaky test passes and fails with no code change. Detect it, quarantine it with an owner, and fix the root cause. Never silently retry it into green.

Supply-chain security

The pipeline is part of the software supply chain. Pin dependencies, generate SBOMs, sign artifacts, and use OIDC instead of stored cloud keys, so each hop can be checked.

DORA metrics

Four numbers that describe how well a delivery system works - deploy frequency, lead time, change failure rate and time to restore. They diagnose the system, not the people in it.

Deploy vs release

A deploy puts new bits on servers. A release changes what users experience. Keeping them separate lets you ship often and roll back the right thing.

Deployment strategies and safe releases

Rolling, blue-green and canary deploys, automated rollback, and zero-downtime schema changes - ship often without betting the fleet on every commit.

Rollout strategies

Rolling, blue-green, canary and shadow deploys replace a running version in different ways. Each trades speed, cost and blast radius differently.

Automated canary analysis

A canary needs gates, not a person watching a graph. Compare the canary with the baseline on agreed metrics over a bake period, then promote it or roll it back.

Rollback and roll-forward

Rollback restores the previous artifact or traffic shape. Roll-forward ships a fix on top of the broken release. Both are valid, and the choice depends on how fast you can restore service and whether the data has changed.

Zero-downtime schema changes

Use expand and contract to change a database while old and new app versions run together. Add compatible changes first, switch the code, backfill, and remove the old shape last.

Cognitive load and team types

Why product teams slow down when they must master cloud, CI and security themselves, and the Team Topologies vocabulary for splitting that work across team types.

Golden paths

A golden path is an opinionated, maintained way to create, ship and observe a service. It should be the easy default, not a prison, and the test is whether a new hire can reach production in a day.

Platform engineering

Internal developer platforms, golden paths and treating infrastructure as a product. Cut cognitive load without building a second Kubernetes.

Catalogs and portals

A developer portal is the interface on top of the platform, and a service catalog answers who owns a service, where its code lives, and how to reach them at 3 a.m. Start with ownership, not with the portal's chrome.

Platform as a product

Treat internal engineers as customers, measure adoption and time to a safe production deploy, and choose a thin platform over a thick one until a clear need forces a thicker abstraction.

Build vs buy for platforms

Most platform parts should be bought or taken from open source. The homegrown PaaS trap is a custom YAML dialect that only three people understand, and that nobody funds to upgrade.

Guardrails and scorecards

Scorecards make drift visible. Guardrails make the unsafe path hard for the baselines that prevent company-scale incidents. Mandates should be rare and come with migration help.

Observability and reliability

Structured logs

A structured log is one machine-readable event with named fields. The few fields that always pay off are the request ID, trace IDs, a stable route, and a level.

Monitoring versus observability

Monitoring checks the questions you wrote down in advance. Observability lets you ask new questions of a running system, by slicing high-cardinality events after the fact.

Wide events

One rich event per request, with dozens of fields, lets you find the cohort that is broken when every average looks healthy. You can start with a single structured log line.

Observability: logs and traces

Structured logs, distributed tracing and OpenTelemetry, so you can ask new questions of a running system instead of guessing which dashboard to open.

Logs, cost and privacy

Logs are often the biggest observability bill, and they can leak secrets. Sample routine traffic, keep every error, never log secrets or PII, and do the volume math before you turn on debug logging.

Distributed tracing

A trace is the full path of one request across services, built from timed spans. Trace context has to be passed on at every hop, or the waterfall breaks into pieces.

Trace sampling

You rarely keep every trace at scale. Head sampling decides at the start and is simple. Tail sampling decides at the end, so it can keep the errors and the slow requests.

OpenTelemetry

A vendor-neutral standard for producing traces, metrics and logs. Instrument once in the app, send to a collector, and choose the backend later without rewriting code.

Metric types and rate

The four metric shapes (counter, gauge, histogram, summary) and why counters are always turned into a per-second rate before anyone reads or alerts on them.

Labels and cardinality

Every unique label combination in Prometheus is a separate time series. Unbounded labels such as user IDs can crash the metrics store, so keep those values in logs and traces.

Metrics, dashboards and alerting

Golden signals, RED and USE, Prometheus histograms and alerts that page you only when users are hurting.

Histograms and fleet percentiles

Why you cannot average p99 latency across pods, and how to aggregate histogram buckets first and then take the quantile. Summaries cannot be merged, so they are the wrong tool for fleet SLOs.

RED, USE and dashboards

Three frameworks for choosing what to graph, and a dashboard layout that answers "is this service okay?" in one screen. RED on top, USE underneath.

Alerting on symptoms

Page a human only for user pain, such as an error ratio or a burn rate, and send everything else to a ticket. Every page needs a runbook and a first step.

Alert fatigue and cleanup

When pages are frequent or useless, responders stop trusting them. Measure the alerting system itself, and run a regular cleanup loop that fixes, widens or deletes noisy rules.

SLIs, SLOs and SLAs

Measure reliability as a ratio of good events to total events. Learn the three terms, and how to pick measures from what users notice.

SLIs, SLOs and error budgets

Define reliability in numbers users care about, then use error budgets and burn-rate alerts to balance shipping speed against stability.

Error budgets

The error budget is the failure your SLO allows. Turn it into numbers, and agree on a policy for what happens when it runs low.

Burn-rate alerts

A burn rate compares your error rate with the pace that would use the budget exactly over the window. Page on fast burns over two windows, to catch problems early without noise.

Latency SLOs and histograms

A latency SLO counts requests that finish under a threshold. Build it from histogram buckets with an edge at the threshold, and never from averaged percentiles.

Rolling out SLOs on a team with none

Start with one measured journey, let the data set the first target, then add alerts and a policy. A strict target that the team ignores is worse than none.

Incident response

Run an incident with clear roles, severity and communication. Mitigate first, debug second, then hand off to the postmortem. Start here for the map of incident pages.

Incident severity and roles

How to rate an incident's severity so everyone shares the same urgency, and how to split the work into commander, operations, communications and scribe roles so one person is not doing everything.

The first fifteen minutes

The order of actions in the first fifteen minutes of an incident, with a worked timeline for a database connection pool failure that ends in a rollback.

Mitigate before root cause

During an incident, restore service first using reversible, practiced levers, and find the root cause once users are no longer failing. Covers the mitigation menu, hypothesis timeboxing and risky-change sign-off.

Incident communication and handoffs

How often to update during an incident, what to say to each audience, the status words to use, and how to hand off or close an incident cleanly.

Blameless culture

Blameless postmortems ask which conditions made a failure possible, not who to blame. Accountability for fixes stays in place.

Blameless postmortems

Learning from failure with timelines, contributing factors and action items that actually get done — without turning people into the root cause.

Writing the summary and timeline

Start with user impact, then build the timeline from machine data. Record the silent window between failure and detection, because it is often the most important line.

Contributing factors, not a single root cause

Complex failures come from several causes that combine. List factors by category, and do not stop at the first person, because that fixes nothing.

Action items that get finished

Good action items are specific, owned, prioritized and dated. Tracking them is what turns a postmortem into an actual fix.

Sharing postmortem learnings

A postmortem that stays in one team's folder teaches one team. Build an incident library, run short reviews, and read public write-ups for structure.

Rotation design

How to size an on-call rotation so people can sustain it, with a primary, a secondary, written handoffs, and a clear definition of what coverage means.

Paging policy

A page is an interruption that costs sleep, so reserve it for user pain, set an acknowledgment target, and route everything else to a ticket. Google's target is at most two incidents per 12-hour shift.

Designing on-call

Sustainable rotations, executable runbooks, toil budgets and a paging policy that keeps humans healthy while production stays covered.

Executable runbooks

A runbook is a checklist a tired person can follow at 3 a.m. Each page-level alert needs one, with the user impact, quick checks, mitigations in order, and when to escalate.

Toil and its budget

Toil is manual, repetitive work that scales with the service and leaves nothing behind. Google's guidance is to keep it under about half of an SRE's time, and to automate the top offenders first.

On-call health and fatigue

On-call costs sleep and focus. Healthy programs pay for it, cap consecutive nights, allow handoffs when sick, and review the load every month. Overload is a reliability defect.

Ownership models for on-call

Developer-owned on-call, dedicated SRE and hybrids each suit different systems. The failed middle is when SRE takes the pages for product bugs that product teams could fix.

Demand forecasting

Two things drive demand, steady growth and discrete events such as launches. Plan against the peak you will actually see, and turn every marketing event into a load estimate.

Utilization and the knee

Queues form when work arrives faster than it is served. Latency stays flat for a long time, then climbs steeply as utilization nears 100%. That bend is the knee, and it is why planning targets sit near 70 to 80%.

Capacity planning and load testing

Forecasting demand, finding the latency knee with load tests and keeping headroom for peaks, deploys and AZ loss.

Little's law and pool sizing

Little's law says in-flight work equals arrival rate times time in system. Use it to size thread and connection pools, and to see why a database, not the app, is often the real ceiling.

Load testing for the knee

A load test finds the offered load where latency turns up sharply or throughput stops growing. Use open arrival rates, realistic traffic and soak runs, and read the generator's own metrics.

Headroom for failure

Capacity must cover a zone failing during peak, rolling deploys, slow autoscaling and retry storms. Size for the survivors, and keep them under the knee.

Worked example - a 5x launch

Plan a checkout API for a 5x traffic launch in five steps - app compute, failure headroom, the database, load-test proof and launch controls. Each step is arithmetic you can check.

Disaster recovery and chaos engineering

RPO and RTO, backups that restore, game days and region drills, and deliberate failure injection, so confidence comes from measured recovery. Start here for the map of disaster recovery pages.

RPO and RTO

The two numbers that shape every recovery plan. RPO is how much data you can lose, RTO is how long you can be down, and each one maps to a recovery tier with a price.

Backups that restore

What makes a backup real, why restores must be timed, how point-in-time recovery works, and the GitLab 2017 outage that shows what untested backups cost.

Game days and region drills

How to rehearse a failover or region evacuation with the real team and runbook, write a testable hypothesis, measure the actual RTO and RPO, and include degraded dependencies in the plan.

Chaos engineering

How to inject controlled failures to test steady state, following the hypothesis, inject, measure, fix and automate loop, with blast-radius limits and abort criteria.