Service discovery and service meshes

A load balancer is only useful if it knows which instances exist right now. Instances come and go with autoscaling, deploys and crashes. Service discovery is how a balancer or client learns the current list. A service mesh goes further: it puts a small proxy beside every instance, so balancing, retries and encryption happen there instead of in each app.

The big idea

Discovery comes in three common styles.

• DNS-based. The service name resolves to a list of instance IPs. It is simple, but many HTTP clients resolve once at startup and never again, so they keep old addresses.
• Registry-based. Instances register in Consul, etcd or ZooKeeper and send a heartbeat with a time-to-live (TTL). If the heartbeats stop, the entry expires. Clients watch the registry for changes.
• Platform-based. Kubernetes tracks pods in EndpointSlices. A ClusterIP Service is a virtual IP that kube-proxy turns into rules on every node. That balances connections, not requests.

With client-side load balancing, there is no middle box. The client library gets the endpoint list from discovery and picks a backend itself. gRPC supports this natively. You save a hop and a single point of failure, and you can balance per request even on HTTP/2. The cost is that balancing logic lives in every client, in every language, and a fix means redeploying every caller.

Walking through it

Here is a call between two services in a mesh.

App A never picks a backend. Sidecar A asks discovery for endpoints, balances the call, encrypts it and sends it. The control plane pushes config to every sidecar. The price is an extra proxy hop on each side, often 1 to 3 ms at p99 (the latency that 99% of calls beat). You also pay memory for each sidecar, and the control plane becomes critical infrastructure. Istio's newer ambient mode replaces per-pod sidecars with one proxy per node, which cuts that cost.

A mesh earns its keep at dozens of services in several languages. For five services in one language, a good HTTP client library is usually enough.

Common mistakes

• Trusting DNS alone. Clients cache resolved IPs and miss new or removed instances.
• Assuming a ClusterIP Service balances requests. It balances connections, so many HTTP/2 requests on one connection land on one pod.
• Adopting a mesh too early. A mesh adds a hop, memory and a control plane. Add it when the number of services and languages makes per-client logic painful.

Check yourself

1. Why can an app that resolved DNS once keep sending traffic to dead instances?

   Its HTTP client cached the IP and never resolves again, so it can't see removed or new instances.

2. Why is a ClusterIP Service a weak tool for balancing HTTP/2 traffic?

   It balances connections. All the requests on one persistent connection go to the same pod.

Key takeaways

• Discovery answers "which instances exist now?" Each style trades simplicity for freshness.
• Client-side balancing removes a hop but puts the logic in every client.
• A service mesh moves balancing, retries, timeouts and mTLS into a sidecar beside each instance.
• A mesh adds a hop, memory and a critical control plane, so adopt it when the size of the system justifies it.

Further learning

Next: global load balancing, which covers how a user's request reaches the right region before the regional balancers take over.