Deduplicating URLs and content

The web repeats itself. One page appears under many URLs, and many pages are near-copies of each other. A crawler needs two checks. First, has it seen this URL before? Second, has it already stored this content? Fetching a duplicate wastes budget, and storing one wastes space and index work.

The big idea

Think of a guest list at a door. The list can wrongly say a stranger is a guest, so that stranger is turned away. It never turns away a real guest who is on the list.

A Bloom filter behaves the same way. It can say a URL was seen when it was not, so the crawler skips a rare new page. It never says "not seen" for a URL it has actually recorded. The picture breaks because a skipped page is lost, and the crawler has no record of it.

Walking through it

URL normalization comes first. Two strings can name the same page. Clean them up before the seen-check:

https://Example.com:443/a/../b?utm_source=x&id=1
  → https://example.com/b?id=1

The scheme and host are lowercased, the default port is dropped, .. is resolved, and the tracking parameter is removed. Each choice is a policy. Removing too much can merge two different articles. Removing too little means refetching the same page forever.

URL-seen uses a Bloom filter. It stores a fixed-size bit array, and a URL is recorded by setting a few bits chosen by its hashes. A false positive skips a URL the crawler has never fetched. That loss is small and accepted.

Content-seen uses fingerprints:

• An exact hash of the page body catches identical copies.
• SimHash gives similar texts similar fingerprints, so near-duplicates such as mirrors and print views can be detected.

Store the fingerprints, so a duplicate page can skip indexing. Its links may still be worth extracting once.

Common mistakes

• Normalizing too aggressively. Stripping parameters that change the content merges distinct pages.
• Normalizing too little. Tracking parameters create new URLs for the same page, so the crawler refetches it.
• Exact hashes only. Mirrors and print pages then fill the index.
• Setting a SimHash threshold too high. Legitimately different pages get collapsed.

Check yourself

1. A Bloom filter says "seen" for a URL the crawler never fetched. What is the cost?

Answer: That page is skipped. It is a small, accepted loss, not a wasted duplicate fetch.

2. Why strip the utm_source parameter from a URL?

Answer: Tracking parameters make the same page look like a new URL, so without stripping them the crawler fetches it again.

Key takeaways

• Bloom filters answer "definitely not seen" or "maybe seen" in little memory.
• Exact hashes catch identical bodies. SimHash catches near-duplicates.
• Normalize URLs before the seen-check, and strip only parameters that do not change the content.
• Every deduplication choice trades storage against the risk of merging distinct pages.

Further learning

Next, see how the crawler deals with endless URLs and keeps pages fresh in spider traps and recrawl.