Certificates and chains of trust

The handshake gives the client a secure channel, but not yet a name for the other end. The certificate closes that gap. It is a signed document that binds a public key to the hostnames it is valid for. The client trusts it because it can follow a chain of signatures back to a root it already has.

The big idea: a signed binding

An X.509 certificate says, in effect, "this public key belongs to api.example.com, and I, the issuer, vouch for it." The issuer signs the whole document with its private key. Anyone with the issuer's public key can check that signature.

Walking through a chain

Browsers and operating systems ship a root store of trusted root certificates. Roots are kept offline and rarely sign anything directly. A root signs an intermediate certificate, and the intermediate signs the leaf, the certificate for your site.

The client checks the chain from the leaf upward. At each step it verifies the signature, the validity period and the names. It stops at a root it already trusts. The server sends the leaf and the intermediates, but not the root, because the client already has the root.

The missing intermediate

A common mistake is for the server to send only the leaf. Desktop Chrome often still works, because it caches intermediates and fetches missing ones. The site looks fine in your browser.

curl, Java, Python, Go and Android apps often do not fetch missing intermediates. They fail with an error such as "unable to get local issuer certificate". The site works in one tool and breaks in another.

Reading a chain with openssl

openssl s_client opens a TLS connection and prints the chain it received. Always pass -servername, so SNI is sent and the server returns the right certificate. The output below is illustrative, so check it against a live host.

openssl s_client -connect api.example.com:443 -servername api.example.com -showcerts </dev/null

Certificate chain
 0 s:CN = api.example.com
   i:C = US, O = Let's Encrypt, CN = R11
 1 s:C = US, O = Let's Encrypt, CN = R11
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
Verify return code: 0 (ok)

Read the chain from the top. Certificate 0 is the leaf. Each certificate's i: (issuer) should match the next certificate's s: (subject), ending at a root you trust. Verify return code: 0 (ok) means the chain checked out. Anything else, such as 21 (unable to verify the first certificate), usually means a missing intermediate.

Getting a certificate: ACME and Let's Encrypt

Let's Encrypt, launched in 2015, issues free certificates through the ACME protocol. An ACME client, such as certbot, cert-manager in Kubernetes or Caddy, proves that you control the domain with one of three challenges:

• HTTP-01: serve a token at a fixed path on port 80.
• DNS-01: publish a token in a _acme-challenge TXT record. It is required for wildcard certificates, and it works for servers that are not reachable from the internet.
• TLS-ALPN-01: answer a special TLS handshake on port 443.

The client requests the certificate, installs it, and renews it before it expires.

Certificate Transparency

How would you know if a certificate was issued for your domain by mistake, or by an attacker? Certificate Transparency (CT) requires publicly trusted certificates to be written to public, append-only logs. Browsers reject certificates that lack proof of logging. You can search the logs at crt.sh and set up alerts for new certificates on your domains.

CT also reveals names. A public certificate for jenkins-staging.example.com makes that internal-sounding hostname visible to everyone.

Common mistakes

• Testing only in a browser. Browsers can hide a missing intermediate.
• Using the Common Name instead of the SAN. Browsers check the SAN.
• Expecting a wildcard to cover the bare domain. Add example.com to the SAN separately.
• Requesting public certificates for internal names without thinking. CT publishes them.

Check yourself

1. Why can a site work in Chrome but fail in a Python script? The server is missing an intermediate. Chrome fetches or caches it, while Python does not.
2. Does *.example.com cover example.com? No. Add the bare domain to the SAN as well.

Key takeaways

• A certificate binds a public key to hostnames through a signature from an issuer.
• The chain runs from the leaf, through intermediates, to a root the client already trusts.
• Servers must send their intermediates. Test with openssl s_client, not only a browser.
• ACME automates issuance. Certificate Transparency makes issuance public.

Further learning

Next, read certificate expiry and renewal to learn why expired certificates cause outages, and how to prevent them.