Crawl Delay

TLDR: Crawl delay is the waiting period a crawler observes between consecutive requests to the same host. It is declared as an unofficial robots.txt directive and enforced client-side by the crawler.

Crawl delay controls request pacing. A crawl delay of 5 means the crawler waits five seconds after completing one request before issuing the next to that host. The directive is the most direct expression of polite crawling, and because no server can force a client to wait, honoring it is voluntary.

The robots.txt Directive:

User-agent: *
Crawl-delay: 10
Disallow: /admin/

User-agent: MyCrawler
Crawl-delay: 2
Disallow:

Delay is declared per user-agent group. A crawler identifying as MyCrawler waits two seconds; anything else waits ten. The directive was never part of the original robots.txt specification and is not covered by RFC 9309, which standardized the protocol in 2022. It remains in wide use regardless.

Crawler Support:

Google’s omission is deliberate rather than an oversight. Googlebot adjusts its own rate based on observed server response times, slowing when latency rises. The reasoning is that a static declared value cannot reflect current server conditions.

Choosing a Delay Value:

Delay and Crawl Duration:

Delay has a direct and often underestimated effect on completion time. Crawling 100,000 pages at a one-second delay takes roughly 28 hours sequentially. At five seconds it takes nearly six days. At ten seconds, close to twelve.

This is why delay cannot be set in isolation from crawl architecture. The way to maintain both politeness and throughput is parallelism across hosts rather than speed against any single host. Ten thousand domains crawled concurrently at a one-second per-host delay yields high aggregate throughput while every individual server sees a single request per second.

Delay vs. Rate Limiting:

The two mechanisms are often confused. Crawl delay is client-side and voluntary: the crawler chooses to wait. Rate limiting is server-side and enforced: the server rejects requests that exceed its threshold, typically returning 429 Too Many Requests.

A crawler can ignore crawl delay but cannot ignore rate limiting, since the requests simply fail. Respecting the declared delay is largely how a crawler avoids encountering the enforced limit at all.

Implementing Per-Host Delay:

from collections import defaultdict
import time

next_allowed = defaultdict(float)

def fetch_with_delay(url, host, delay):
    wait = next_allowed[host] - time.monotonic()
    if wait > 0:
        time.sleep(wait)

    response = requests.get(url, timeout=10)
    next_allowed[host] = time.monotonic() + delay
    return response

Tracking the next permitted request time per host, rather than sleeping after every request globally, allows the crawler to work on other hosts while one waits. This is the same principle behind the back-queue layer in a URL frontier.

Bright Data’s Crawl API reads and applies declared crawl delays automatically, and adapts pacing when target servers show signs of strain.