Terminating TLS and mTLS

Encrypted traffic has to be decrypted somewhere, because the application needs plaintext to work. The point where that happens is called termination. Choosing it decides which hops are encrypted, who holds the certificates, and what a load balancer can see. Inside a data center, the same question returns for every service-to-service call.

The big idea: where the plaintext lives

There are three common placements.

Edge termination is the usual default. Re-encryption is the common answer when a compliance rule, such as PCI DSS or HIPAA, requires encryption in transit everywhere.

Walking through a request

Here is a request through a load balancer that re-encrypts.

client  ──TLS (cert: www.example.com)──►  load balancer
                                          decrypts, reads the path, picks a backend
load balancer ──TLS (cert: backend)──►   app server
                                          decrypts, runs the handler

Each hop has its own certificate and its own expiry date. Every extra hop adds one more thing to renew. This is why automation matters even more with re-encryption.

Zero trust and mutual TLS

Inside a data center, the older assumption was that the internal network was safe. The newer one, called zero trust, says not to trust the network. Each service encrypts and authenticates every call.

In mutual TLS (mTLS), the client also presents a certificate, so each side proves its identity to the other. A call is then allowed because of who the caller is, not which IP address it came from.

Managing certificates for thousands of short-lived containers by hand is not practical. Two approaches automate it.

• SPIFFE defines a standard workload identity, written as a URI such as spiffe://prod.example.com/ns/payments/sa/api. The identity is placed in the SAN of a short-lived certificate, often valid for an hour. SPIRE is a reference implementation that issues these certificates after checking what a workload is.
• Service meshes, such as Istio and Linkerd, add a proxy beside each workload. The proxy does mTLS and rotates certificates automatically. The application speaks plain HTTP to localhost. Authorization rules can then say "only payments/api may call ledger" in terms of identity, not IP addresses.

Walking through an authorization rule

A mesh policy is written in terms of identity, so it survives pods being rescheduled to new IP addresses. This sketch shows the meaning of such a rule, not a real mesh configuration file.

allow:  spiffe://prod.example.com/ns/payments/sa/api
   to:  spiffe://prod.example.com/ns/ledger/sa/writer
   for: POST /entries
deny:   everything else

When the payments/api workload moves to a different node, its identity is unchanged, so the rule still applies. An IP-based firewall rule would break.

Common mistakes

• Assuming edge termination encrypts everything. Traffic after the load balancer may be plaintext.
• Adding hops without adding renewal. Each certificate expires on its own schedule.
• Using shared API keys between services. A key that never rotates is a long-lived secret. Short-lived identities are safer.
• Writing firewall rules by IP address for services. Pods change addresses. Identity does not.

Check yourself

1. A load balancer must route by URL path. Which placement can it use? Edge termination or re-encryption, because the load balancer must read the HTTP request. Passthrough hides it.
2. Why is an identity-based rule better than an IP-based rule for services on Kubernetes? Pods are rescheduled with new IPs, while the identity stays with the workload.

Key takeaways

• Termination is where plaintext begins. Choose edge, re-encryption or passthrough based on what must be encrypted and what the load balancer must read.
• Re-encryption encrypts every hop, at the cost of more certificates to renew.
• mTLS proves each service's identity to the other, in both directions.
• Use short-lived workload identities and a mesh or SPIFFE so certificates rotate automatically.

Further learning

Next, return to the hub, TLS and certificates, for the performance side of the handshake and a hands-on check with openssl.

The Illustrated TLS 1.3 ConnectionEvery byte of a TLS connection explained and reproducedtls13.xargs.org