The URL frontier

A crawler starts from a few seed URLs. Every page it fetches adds new links, so the list of pages still to fetch grows quickly. That list is the URL frontier. A plain first-in, first-out queue is too simple for it. It ignores importance, lets one large site flood the queue, and fills memory.

The big idea

Think of a delivery driver's to-do list. The driver does the high-priority stops first. The driver also never makes three drops at the same house in a row, because the house would not be happy about it.

The first half of that is priority. The second half is politeness, which gets its own page. This one covers priority.

The picture breaks when priority is the only rule. High-priority pages from one large site can still crowd out everything else. That is why politeness exists.

Walking through it

The frontier has front queues, one per priority level. The fetcher takes from the front queue with a bias toward high priority. A simplified trace uses strict order:

Front queues:
  high: [wiki/a, wiki/b]
  mid:  [news/1]
  low:  [spam/9]

Fetch 1: take from high → wiki/a
Fetch 2: take from high → wiki/b
Fetch 3: high is empty  → take from mid → news/1
Fetch 4: mid is empty   → take from low → spam/9

Real frontiers pick randomly, weighted toward high priority, rather than in strict order. Low-priority pages still get fetched, just less often.

Priority comes from a score. The score might combine a site's importance, how often the page changes, or whether it is a fresh discovery. The scoring function is where product policy becomes code. A crawl that values unique pages scores differently from one that values fresh news.

Common mistakes

• One FIFO queue for everything. Big sites flood the queue, and important pages wait behind spam.
• Keeping the whole frontier in memory. Use disk-backed queues, and expect the frontier to grow large.
• Losing the frontier on deploy. Checkpoint it to durable storage, or months of discovered URLs disappear.
• Writing a priority score with no goal. Decide whether the crawl values coverage or freshness first.

Check yourself

1. Why not use one first-in, first-out queue for every discovered URL?

Answer: It ignores importance, and one big site can flood the queue, hit that host hard and use up memory.

2. What does a front queue represent?

Answer: A priority level. High-value pages go to a high-priority front queue, so they are fetched more often.

Key takeaways

• The frontier is the crawler's list of URLs still to fetch.
• Front queues express priority. Back queues, covered next, express politeness.
• Keep the frontier on disk and checkpoint it, so restarts lose nothing.
• The priority function is where the crawl's goal is written down.

Further learning

Next, see how per-host back queues keep the crawl polite in politeness and back queues.