Load balancing and proxies

Once you run more than one copy of a service, something has to decide which copy handles each request. That something is a load balancer: a server that spreads requests across a pool of backends, the servers that do the real work. It looks like plumbing, but it keeps the system available. It hides failed machines, lets you deploy without dropping requests, and decides whether a slow server gets more traffic or less.

A wrong choice here causes outages. A balancer that keeps sending traffic to one bad instance, or that trusts a broken health check, can turn one failure into many.

This section goes in the order you need the ideas. First, what the balancer can see at each network layer. Then how it picks a backend, how it decides a backend is sick, how it finds backends in the first place, and how users reach the right region.

Concepts in this section

Common mistakes

• Retrying at every layer. Three layers that each retry three times turn one request into 3 × 3 × 3 = 27 attempts against a service that is already failing. Retry at one layer only, and only for requests that are safe to repeat. See resilience.
• Balancing long-lived connections as if they were requests. gRPC channels and WebSockets are balanced once, when they open. Scale a gRPC service from 4 to 8 instances and the new four get almost nothing, because clients keep their old connections. Use an HTTP/2-aware proxy, or set a maximum connection age so clients reconnect.
• Running one balancer with no redundancy. Use an active-passive pair, many instances behind anycast or DNS, or a managed service. Managed balancers scale up over minutes, so a sudden 10× spike can overrun one sized for normal traffic.
• Trusting the fleet average. A fleet can look healthy while a few instances run hot. Check per-instance CPU and request rate.

Key takeaways

• A load balancer distributes work, detects unhealthy backends, and gives clients one stable address.
• Layer 4 forwards connections cheaply. Layer 7 reads each request and can route, retry and authenticate it.
• Least outstanding requests, often done with power of two choices, is a strong default that needs no tuning.
• Use active health checks and passive outlier detection together, and keep health checks away from shared dependencies.
• Retries multiply across layers, and long-lived connections defeat rebalancing. Design both on purpose.

Further learning

Matt Klein, the creator of Envoy, wrote a clear overview of modern layer 4 and layer 7 balancing, service meshes and global load balancing.

Introduction to modern network load balancing and proxyingIt was brought to my attention recently that there is a dearth of introductory educational material available about modern network load balancing and proxying. I thought to myself: How can this be? Load balancing is o…Linux.com

Start with layer 4 vs layer 7 for the first concept.