Spider traps and recrawl scheduling

A spider trap is a site that generates URLs forever. A calendar that links to next month, and then the month after that, is one example. Session IDs that make every visit a new URL are another. A crawler that follows every link never finishes. Traps must be capped, and the recrawl schedule decides which known pages deserve another visit.

The big idea

Picture a hallway with doors that open onto more hallways, with no exit. A visitor who keeps walking never leaves. The fix is a rule: each visit to a building gets a fixed number of steps, whatever the building looks like.

The picture breaks for large, real sites. A large catalog can look like a trap. The caps must be set from what the host actually contains.

Walking through it

A calendar trap looks like this:

/calendar?date=2026-10  → links to 2026-11
/calendar?date=2026-11  → links to 2026-12
/calendar?date=2026-12  → links to 2027-01
... every page links forward, forever

Without a limit, this host takes every fetch in the budget. Three defenses help:

• Per-host daily cap. Set it from the host's real size, and stop fetching once it is reached.
• Pattern detection. Watch for URLs that differ only in a date or session parameter, and demote or block those patterns.
• Length and parameter budgets. Reject URLs that are too long or carry too many query parameters.

Prefer sitemaps and canonical tags over following every link. A sitemap is the site's own list of pages, so it is a far better guide than the link structure.

Recrawl scheduling

Once a page is known, the crawler must decide when to visit it again. A simple rule uses change history. If a page changed on each of its last three visits, schedule the next visit sooner. If it has not changed, back off and let it sleep for longer.

Visit 1: changed     → revisit soon
Visit 2: changed     → revisit soon
Visit 3: changed     → keep the interval short
Visit 4: unchanged   → lengthen the interval

Importance also matters. A news front page deserves frequent visits, while an archived page can wait for weeks.

Watch out for soft 404s, pages that return status 200 but say "not found" in the body. Content checks catch them, and site-level anomaly checks catch whole sections that are soft errors.

Common mistakes

• Following every link on a host. One trap can consume the day's budget.
• Treating all URLs as equal. Without importance scores, the crawl spends its budget on low-value pages.
• Recrawling static pages on a short fixed schedule. It wastes budget that changing pages need.
• Trusting status codes alone. A 200 response can still be an error page.

Check yourself

1. Why does a cap per host protect the crawl budget?

Answer: A trap can generate an unlimited number of URLs, and the cap stops one host from using the whole day's fetches.

2. A news page changed on each of its last three visits. What should the scheduler do?

Answer: Shorten its recrawl interval, so new changes are picked up sooner.

Key takeaways

• Spider traps create endless URLs. Cap pages per host, and watch for repeating patterns.
• Crawl budget is finite, so every fetch is a fetch not spent somewhere else.
• Schedule recrawls from change history and importance.
• Decide the goal first: unique coverage or fresh important pages.

Further learning

Return to the web crawler case study for the full design, including storage, distributed scheduling and the LLM crawl twist.