Case study: web crawler

A crawler looks like a loop over URLs until you reach scale. At a billion pages a month, the hard problems are politeness (not overloading a small site), deduplication (the same content under a thousand URLs) and spider traps (endless links that waste the crawl). Important pages also need revisits often enough to stay useful.

The classic design is Mercator, described by Heydon and Najork, plus the frontier design in Manning, Raghavan and Schütze's Introduction to Information Retrieval. Crawls for LLM training add opt-out signals and quality filters on the same skeleton.

Requirements

• Fetch on the order of 10⁹ pages per month, about 400 per second on average.
• Be polite: cap requests per host, space them by seconds, and honor robots.txt.
• Recrawl important or changing pages more often than the long tail.
• Survive timeouts, traps, soft error pages and broken encodings.

Ranking is out of scope here. See search systems for the index side.

Concepts in this topic

Architecture

The pipeline runs in order: seed the frontier, schedule each URL when politeness allows, fetch with timeouts and size caps, parse out text and links, deduplicate, then store the page for later indexing or training.

DNS and robots.txt

Slow DNS lookups once blocked whole crawlers. Keep a large DNS cache and resolve names asynchronously. Fetch robots.txt once per host and cache it for hours. Check every URL against the cached rules before fetching. Decide in advance whether a failed robots.txt fetch fails open or closed, and apply that choice everywhere.

Distributed crawling

Assign hosts to workers with hash(hostname) % N. Each worker owns the frontier for its hosts, so politeness stays local. Links that point to another worker's host are forwarded to that worker. Checkpoint frontiers to durable storage, so a deploy does not lose months of discovered URLs.

Storage handoff

Store fetched bytes in object storage, keyed by content hash. Keep URL metadata alongside: last fetch time, last status, ETag, content fingerprint, priority and the next fetch due time. The recrawl scheduler reads that table. Publish a "document ready" event to a queue, so indexers can fall behind without slowing fetches.

Failure modes

Politeness is product policy

Legal and product teams set the policy, and engineering enforces it. Examples include one request per second per host by default, faster rates for partners, a contact address in the User-Agent, and no crawling of login or cart pages.

Common mistakes

• Sharing one queue across hosts. One slow site then blocks threads that could fetch elsewhere.
• Counting raw fetches as success. Measure useful pages per second, after deduplication and quality filters.
• Seeding from noisy directories. Spam becomes the first thing you crawl, and it uses up budget.
• Normalizing URLs too aggressively or too little. Both waste budget in different ways.

Key takeaways

• A scalable crawler is a frontier, polite fetchers, a parser, deduplication and storage, sharded by host.
• Front queues set priority, and per-host back queues enforce politeness.
• Cache DNS and robots.txt, and rate-limit each host and the whole crawl.
• Defend against spider traps, and schedule recrawls by importance and change rate.
• LLM crawls add quality filters, PII checks and opt-out handling to the same backbone.

Further learning

A complete "design a web crawler" interview with an ex-Meta staff engineer. It is good for the dataflow-first presentation.

Introduction to Information Retrieval: Web crawling and indexesWeb crawling and indexesnlp.stanford.edu Mercator: A Scalable, Extensible Web Crawler (Najork)wwwmerc.pdfmarc.najork.org ByteByteGo blog (Alex Xu)Explain complex systems with simple terms, from the authors of the best-selling system design book series. Join over 1,000,000 friendly readers. Click to read ByteByteGo Newsletter, a Substack publication.blog.bytebytego.com

Probabilistic data structures