Scaling fundamentals

A system scales if you can add capacity by adding resources, roughly in proportion. Double the servers, and you roughly double the requests you can serve. Systems that do not scale hit a wall. Adding machines does nothing, or makes things worse, because a shared component, usually the database, is already saturated.

Most real systems do not start distributed. They start as one process on one machine. They add parts one at a time, as specific bottlenecks appear: a separate database, a load balancer, a cache, replicas, a CDN, queues, shards, more regions. Knowing that order, and why each step comes when it does, is most of what scaling means in practice.

Three words that get mixed up

• Performance is how fast one request is served at a given load.
• Scalability is how capacity grows as you add resources.
• Elasticity is how quickly and cheaply capacity follows demand, both up and down.

A system can be fast but not scalable. A single big in-memory server is one example. It can also be scalable but slow, such as a batch pipeline that takes an hour but handles ten times more data with ten times the workers. A useful test is: "If traffic tripled tomorrow, what would we do, and how long would it take?" The answer reveals all three properties at once.

Concepts in this topic

Read them in order. Each one builds on the last.

Common mistakes

• Scaling before measuring. Adding a queue or a shard before finding the saturated resource moves the problem.
• Choosing infrastructure for big-company reasons. Each component needs to be justified by a number.
• Underestimating the cost of distribution. Sharding, microservices and multi-region each bring operations work. A sharded database needs routing, cross-shard queries and resharding tools, which can take months.
• Forgetting that the database is the usual limit. The app tier is stateless and cheap to copy. The database is shared and stateful.

Key takeaways

• Scalability, performance and elasticity are different properties, so measure the one you need.
• Scale up first. Modern machines are large, and a single big database goes a long way.
• Horizontal scaling requires statelessness, with sessions, files and queues moved out of the process.
• Take each step only when a measured bottleneck demands it, and write down the next step.
• Keep peak utilization around 50 to 70%, because latency rises sharply as utilization nears 100%.

Further learning

Dan McKinley's essay "Choose Boring Technology" argues that each team has a small budget of unfamiliar technology. Spend it where it differentiates your product, and use well-understood tools everywhere else:

Choose Boring Technology (Dan McKinley)Choose Boring TechnologyDan McKinley :: Math, Programming, and Minority Reports

Martin Kleppmann's book covers the replication, partitioning and consistency trade-offs that appear once a system outgrows one machine:

Designing Data-Intensive Applications (Kleppmann)Designing Data-Intensive Applications (DDIA) — an O’Reilly book by Martin Kleppmann (The Wild Boar Book)dataintensive.net

Start with vertical versus horizontal scaling to see the first choice in this topic.