How to Scrape a Static Website Without Getting Blocked

Walking a paginated site the way Handling Pagination and Infinite Scroll describes means issuing hundreds of requests to one host, which is exactly the pattern a rate limiter is built to notice.

Throttle, check the status, and back off on 429/503 to avoid getting blocked.

On a static site — plain HTML, no client-side rendering, usually no commercial bot-management product — three changes remove almost every block: send a complete, coherent set of browser headers; keep one Session so connections and cookies persist; and space requests with jitter while honouring Retry-After and robots.txt. Proxies are a scaling tool, not a fix, and reaching for them before the headers are right just spreads the same detectable pattern across more addresses.

The Signals a Static Site Can Actually See

A server without a JavaScript challenge has a narrow view of you: the TCP and TLS handshake, the request line, the headers, the cookies you send back, and the timing between requests. That is the whole surface, and it is worth ranking because the effort is very unevenly distributed.

Cheap header signals sit at the top of the list, which is why fixing them removes most blocks on a static site. Rate and referer matter, but only after the obvious tells are gone.

The single biggest tell is the default User-Agent. requests sends python-requests/2.32.5, httpx sends python-httpx/0.28.1, and a one-line filter on either blocks the overwhelming majority of casual scraping. Fixing it costs nothing.

The second is header coherence rather than any single header. Real browsers send Accept, Accept-Language, Accept-Encoding, and — for Chromium-based clients — Sec-Fetch-Site, Sec-Fetch-Mode and Sec-Ch-Ua. A request that claims to be Chrome 136 but sends only User-Agent and Accept: */* is inconsistent in a way that is trivial to test for. Header order is also observable over HTTP/2, though few static sites check it.

Below that sit IP reputation and cadence. A datacenter ASN is not itself suspicious — plenty of legitimate traffic originates there — but combined with no cookies and a perfectly regular interval it is conclusive. The difference between address types, and when paying for the expensive kind is justified, is covered in Residential vs Datacenter Proxies.

TLS fingerprinting deserves a mention even here. Python's ssl module produces a cipher and extension ordering that does not match any shipped browser, so a JA3 or JA4 hash gives away a requests client claiming to be Chrome regardless of headers. Static sites rarely check, but if you see a 403 that no header change fixes, that is the likely cause and Using curl_cffi to Impersonate Browsers is the remedy.

Cadence: What Actually Gets You Rate Limited

Two properties of your request timing matter, and they pull in different directions.

The first is throughput. Most rate limiters are a token bucket keyed on IP: a bucket of N tokens refilling at R per second, one token per request, 429 when the bucket empties. This means a burst is tolerated and a sustained rate above R is not, which is why a scraper can run fine for a minute and then fail continuously.

The second is regularity. A time.sleep(1) loop produces an inter-arrival time with near-zero variance, which no human session ever exhibits. Simple heuristics flag it, so add jitter — and add it multiplicatively around a base delay rather than as a small fixed window, so the distribution has a realistic tail.

Illustrative shape of the same crawl run three ways. An unthrottled run dies in under a minute, a perfectly regular one survives longer but is still a recognisable pattern, and a jittered run that honours Retry-After keeps returning 200.

The numbers in that figure are illustrative of the shape rather than measurements of any particular site; the sequence is what generalises. Start from robots.txt: many sites publish a Crawl-delay directive, and where they do, that value is a stated contract you should honour rather than a number to negotiate. Where they do not, one request every two to five seconds per host is a defensible default for a static site, and it is rarely the constraint on total throughput anyway — crawling ten hosts at that rate gives you the same volume as hammering one.

The other half of cadence is not sending the request at all. A crawl that re-fetches unchanged pages every run burns budget for nothing; conditional requests with ETag and If-Modified-Since turn most of those into a 12-byte 304, which almost never counts against a rate limit. That is the substance of Caching and Incremental Crawling, and on a site you poll repeatedly it reduces block risk more than any header change.

A Polite Crawler You Can Run

This puts the pieces together: robots.txt compliance, coherent headers, a persistent session with connection reuse, retry with backoff on the right status codes, and Retry-After honoured when the server states it.

"""A crawler that respects robots.txt, backs off, and honours Retry-After."""
import random
import time
import urllib.robotparser
from urllib.parse import urljoin, urlparse

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

USER_AGENT = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"
)
BROWSER_HEADERS = {
    "User-Agent": USER_AGENT,
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "same-origin",
    "Upgrade-Insecure-Requests": "1",
    "Connection": "keep-alive",
}


def build_session() -> requests.Session:
    retry = Retry(
        total=4,
        backoff_factor=1.5,          # 1.5 s, 3 s, 6 s, 12 s
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=frozenset({"GET", "HEAD"}),
        respect_retry_after_header=True,
        raise_on_status=False,
    )
    session = requests.Session()
    session.headers.update(BROWSER_HEADERS)
    adapter = HTTPAdapter(max_retries=retry, pool_connections=4, pool_maxsize=4)
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    return session


def robots_for(session: requests.Session, url: str) -> urllib.robotparser.RobotFileParser:
    root = f"{urlparse(url).scheme}://{urlparse(url).netloc}"
    parser = urllib.robotparser.RobotFileParser()
    response = session.get(urljoin(root, "/robots.txt"), timeout=15)
    parser.parse(response.text.splitlines() if response.ok else [])
    return parser


def polite_get(session: requests.Session, url: str, base_delay: float = 3.0) -> requests.Response | None:
    time.sleep(base_delay * random.uniform(0.7, 1.8))
    response = session.get(url, timeout=(5, 30))
    if response.status_code == 429:
        wait = float(response.headers.get("Retry-After", base_delay * 10))
        print(f"429 on {url}; sleeping {wait:.0f}s")
        time.sleep(wait)
        response = session.get(url, timeout=(5, 30))
    if response.status_code >= 400:
        print(f"{response.status_code} on {url}")
        return None
    return response


def crawl(start: str, pages: int = 5) -> list[str]:
    session = build_session()
    rules = robots_for(session, start)
    crawl_delay = rules.crawl_delay(USER_AGENT)
    delay = float(crawl_delay) if crawl_delay else 3.0

    collected: list[str] = []
    url: str | None = start
    for _ in range(pages):
        if url is None or not rules.can_fetch(USER_AGENT, url):
            break
        response = polite_get(session, url, base_delay=delay)
        if response is None:
            break
        collected.append(response.text)
        url = next_page_url(response.text, url)
    return collected


def next_page_url(html: str, current: str) -> str | None:
    from bs4 import BeautifulSoup

    link = BeautifulSoup(html, "lxml").select_one("li.next > a")
    return urljoin(current, link["href"]) if link else None


if __name__ == "__main__":
    pages = crawl("https://books.toscrape.com/catalogue/page-1.html", pages=3)
    print(f"fetched {len(pages)} page(s)")

Four details are load-bearing. timeout=(5, 30) sets connect and read timeouts separately, so a dead host fails in five seconds while a slow-but-alive one still gets half a minute. respect_retry_after_header=True makes urllib3 use the server's stated wait instead of its own backoff curve. allowed_methods restricts automatic retries to idempotent verbs — retrying a POST can double-submit. And pool_maxsize=4 caps concurrent connections to one host, which is both polite and enough: HTTP keep-alive means those four sockets are reused rather than renegotiated, and TLS handshakes are a visible cost on a long crawl.

Rotating the User-Agent between requests is deliberately absent. Within a single session it is counterproductive — a client whose browser identity changes mid-session while the cookies stay the same is more anomalous, not less. Rotation belongs at the session boundary, as described in How to Rotate User Agents in Python.

Reading the Response You Get Back

Blocks do not always announce themselves with a status code. Add cheap assertions so a soft block fails loudly rather than filling your database with empty rows.

403 immediately on the first request usually means headers or IP reputation, not rate. Nothing about waiting will help; change the client.
429 means you exceeded a documented limit. Read Retry-After, and if it is present, honour it exactly — ignoring it is the fastest route from a temporary throttle to a permanent ban.
503 with a short body and a cf-mitigated or server: cloudflare header is a challenge, not an outage.
200 with a body 90% smaller than usual, or containing captcha, Access Denied, or unusual traffic, is a soft block. Compare len(response.content) against a rolling median and alert when it drops.

Retry logic that distinguishes these cases is worth factoring out; Retrying Failed Requests with Tenacity covers the declarative version, and Detecting Silent Scraper Failures covers noticing the 200-with-no-data case before a week of runs is wasted.

Edge Cases and Caveats

Accept-Encoding: br without the decoder. requests advertises Brotli only when brotli or brotlicffi is installed. Hard-coding it in your headers without the package yields unreadable bytes. Install requests[socks]-style extras or drop br from the list.
Referer on a first request. Sending a Referer for a URL you navigated to directly is itself inconsistent. Set it only for pages you reached from a link you actually fetched.
Session cookies expire mid-crawl. A long run can outlive the session cookie and silently drop to anonymous responses. Persist and refresh the jar as in Managing Cookies and Sessions.
robots.txt returning 404 or 500. RobotFileParser treats an unparsed file as fully permissive. Decide explicitly what a missing file means for you rather than inheriting that default.
Rate limits keyed on something other than IP. Some limiters key on a session cookie or an account. Rotating IPs then does nothing while rotating sessions does.
Concurrency undoing your delay. Ten workers each sleeping three seconds produce 3.3 requests per second to one host. Enforce the limit globally, not per worker — see Limiting Concurrency with Semaphores.
HEAD requests are not free. Some servers rate-limit HEAD on the same bucket as GET, so probing with HEAD before every fetch doubles your consumption.
Terms of service and law. robots.txt is a technical signal, not a licence. Personal data, paywalled content and explicit contractual prohibitions carry obligations that no amount of header tuning addresses.

Frequently Asked Questions

Why do I get a 403 on the very first request? A 403 before you have sent enough traffic to trip any rate limit points at the request itself, not the rate — almost always the default python-requests User-Agent, an incomplete header set, or an IP range the site already distrusts. Copy the full header block from your browser's network panel, replay it, and if the 403 persists with byte-identical headers, the discriminator is below HTTP, at the TLS layer.

How long should I wait between requests? Take the Crawl-delay from robots.txt when the site publishes one, since that is the operator telling you their limit. Otherwise a base of two to five seconds per host with multiplicative jitter is a reasonable default for static content, and you scale total throughput by crawling more hosts concurrently rather than one host faster.

Do I need proxies to scrape a static site? Usually not for a few thousand pages spread over hours from one address, provided your headers are coherent and you are not hammering. Proxies become necessary when you need parallel throughput against a single host, when your address is already reputationally poor, or when the site geo-restricts content — the trade-offs are in Rotating Proxies and Managing IP Blocks.

How do I tell a soft block from a real page? Record a baseline for each URL pattern — response size, status, and the presence of a selector you know should match — then compare every response against it. A page that returns 200 with a tenth of the usual bytes, or where your primary selector matches zero elements, is a block dressed as a success, and catching it at fetch time is far cheaper than discovering it in the data.

Handling Pagination and Infinite Scroll — the parent topic, and the traversal this crawler performs.
Caching and Incremental Crawling — the requests you avoid sending at all.
Residential vs Datacenter Proxies — what changes when one address is no longer enough.
Understanding HTTP Requests and Responses — the header and status-code mechanics behind all of the above.