The canonical scaling path

Most successful web products follow the same path. Each step is triggered by a specific symptom and adds complexity. Take a step only when its symptom shows up.

The big idea

The path is a ladder of bottlenecks. Each rung solves the bottleneck that the previous rung exposed. Teams get into trouble by jumping to the top rung, with microservices, sharding and a streaming platform, while they still have the scale of the second rung.

Walking through it

1. One server. The app and database share one box. This is fine for a prototype and often for the first thousands of users. It ends when the two compete for CPU and memory, or when a deploy takes everything down.
2. Separate the database. Move it to its own machine or a managed service. Each can now be sized on its own. It ends when one app server cannot handle the traffic, or when you cannot deploy without downtime.
3. Load balancer and several stateless app servers. This needs the work in stateless services. You get zero-downtime deploys and survive a node failure. It ends when the database becomes the bottleneck, usually on reads.
4. Cache, read replicas and a CDN. A cache absorbs hot reads. Read replicas take the rest. A CDN serves static files and media from the edge. Many read-heavy systems grow a long way here. The new problems are cache invalidation and replica lag, where users do not see their own writes.
5. Queues, sharding and regions. Queues move slow work off the request path. Sharding splits data once one primary cannot take the writes. Multi-region deployments cut latency and survive a region outage. These are the expensive steps, and each one adds failure modes.

The stage 5 architecture looks like this:

Common mistakes

• Jumping to stage 5 early. Sharding and microservices at stage 2 scale add cost with no benefit.
• Skipping stateless work. Without it, stage 3 cannot happen.
• Adding a step without a symptom. Write down the metric that justifies each step.

Check yourself

1. What symptom usually ends stage 3?

   Answer: the database becomes the bottleneck, usually on reads.

2. Which step should come before sharding, and why?

   Answer: a cache and read replicas. They are cheaper and reversible, and many read-heavy systems grow far with them.

Key takeaways

• Each stage of the path is triggered by a symptom, so measure before you add a part.
• The order is: separate the database, make app servers stateless, add cache and replicas, then queues, shards and regions.
• The first three stages cover most products for a long time.

Further learning

This AWS re:Invent talk walks through almost exactly this evolution, from one instance to a multi-tier architecture serving 10 million users, with the reasoning for each step:

Instagram's engineers describe how they scaled out, scaled up and scaled the team. It is a useful real-world companion to the stages above:

Next, see the three independent ways to split a system, in the scale cube.