Guide

Traffic and messaging

CDN routing and cache hierarchy

A CDN sends each user to a nearby point of presence, then uses tiers, an origin shield and request collapsing so that a new object reaches your origin about once, not once per PoP.

88 of 285 · 3 min read

A CDN runs many servers in data centers around the world. Each cluster is a point of presence (PoP). Large CDNs run hundreds of PoPs, many inside the networks that their users connect through. Two questions decide how well it works: how does a user reach a nearby PoP, and what happens when that PoP doesn't have the content?

Concept

The origin is your own server or storage bucket, the place a cache fetches from on a miss. A miss at a PoP means the PoP has to ask a bigger cache or the origin for the object.

The big idea

Without any hierarchy, every PoP fetches each object from the origin itself. If 300 PoPs ask for the same new video segment, the origin serves it 300 times.

The fix is to add layers between the PoPs and the origin. Picture a chain of stores fed by one central warehouse. Each store restocks from a regional depot, and each depot restocks from one warehouse that talks to the factory. The factory sees one order, not hundreds.

Walking through it

Routing. There are two common ways to route a user to a nearby PoP.

  • DNS-based routing. Your hostname is a CNAME to the CDN's name. The CDN's DNS servers look at where the query came from and return the IP of a nearby, healthy PoP.
  • Anycast. Every PoP announces the same IP addresses through BGP. Internet routing delivers each packet to the closest PoP.

Hierarchy. Three mechanisms cut the origin's load.

  • Tiered caching. Edge PoPs fetch from a smaller set of larger regional caches, instead of from the origin.
  • Origin shield. One cache sits directly in front of the origin. Every miss from every tier passes through it, so the origin sees at most one fetch per object per TTL.
  • Request collapsing. If 5,000 users request the same uncached object in one second, the PoP sends one upstream request and holds the other 4,999 until it returns.

The effect on the origin is large. Suppose you publish 10,000 new images a day, and each is requested in 250 PoPs.

Without a shield: 10,000 × 250 = 2,500,000 origin fetches
With a shield:    10,000 origin fetches

The same content, 250 times less origin traffic.

Common mistakes

  • Skipping the shield. Each PoP then pulls every new object from the origin on its own.
  • Mixing up the two routing methods. DNS returns one nearby IP. Anycast announces one IP from many places. Both send users to a close PoP, through different mechanisms.
  • Expecting new content everywhere at once. The first request for an object still travels up the tiers. Only later requests are fast.

Check yourself

  1. With 10,000 new images, each requested in 250 PoPs, how many origin fetches happen with and without a shield? Up to 2,500,000 without a shield, and 10,000 with one.
  2. What does request collapsing do? It holds concurrent requests for the same uncached object, so the PoP makes one upstream request for all of them.

Key takeaways

  • A CDN routes each user to a nearby PoP, using DNS or anycast.
  • Tiers and an origin shield turn one fetch per PoP into roughly one fetch per object.
  • Request collapsing stops a crowd of concurrent misses from becoming a crowd of origin requests.

Further learning

Cloudflare's explainer: What is a CDN? covers PoPs and caching in plain terms.

Next, read about cache control and cache keys, which decide what the CDN stores and for how long.

CDNs and the edge

Serve content from points of presence near users. A CDN cuts latency, absorbs attacks and cuts bandwidth costs, and its cache keys, headers and purge rules decide how well it works.

Speed of light and round trips

Distance sets a floor on latency that no server tuning can remove. A fresh HTTPS request pays several round trips before the first byte, which is why content has to move closer to users.

Caching

Where to put a cache, how to read and write through it, how to size it from hit-rate math, and how to keep it from serving stale data or melting down.