Advanced Scraping Techniques and Anti-Bot Evasion

There is a point at which a scraper stops failing for reasons you control. The selectors are right, the pagination terminates, the headers look sane — and the target still returns a 403, or serves a page that is visibly missing the data, or shows an interstitial that never resolves. At that point the problem is no longer extraction; it is that a detection system has classified your client as automated and is treating it differently from a browser. This path explains what those systems measure, in what order, and what each measurement costs you to satisfy.

It assumes you are comfortable with the material in The Complete Guide to Python Web Scraping — requests, sessions, selectors, pagination — and that the target you are working on has genuinely resisted a well-behaved HTTP client. Nothing here is a substitute for asking whether you are allowed to collect the data in the first place; the techniques are described so you can diagnose and fix a legitimate collection job, and every one of them works better when paired with a slower, politer crawl.

How Detection Actually Works

A modern defence is not a single check but a stack of them, evaluated cheapest-first, producing a score. Each layer looks at a different part of the connection, and each one costs you a different amount of engineering to satisfy. Understanding the ordering is what stops you from buying residential proxies to fix a problem that was a missing Accept-Language header.

The first layer is the request itself: the User-Agent string, whether the accompanying headers are the ones a real browser would send alongside it, and whether their order matches. Python's requests sends a small, alphabetically tidy header set that no browser has ever produced, which is trivially detectable and equally trivially fixed.

The second layer is the IP address — its ASN, whether it belongs to a known cloud provider, and how much traffic has recently come from it. The third is the TLS handshake, which happens before a single byte of HTTP is exchanged and which fingerprints your client's cipher suites, extensions, and elliptic curves. The fourth is JavaScript: a challenge script that runs in the page, probes the environment, and posts back a token. The fifth and most expensive is behavioural — dwell time, mouse movement, scroll cadence, the ratio of page views to asset requests.

import requests

BROWSER_HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,"
              "image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Upgrade-Insecure-Requests": "1",
}

def show_what_the_server_sees(headers: dict[str, str]) -> dict[str, str]:
    response = requests.get("https://httpbin.org/headers", headers=headers, timeout=10)
    response.raise_for_status()
    return response.json()["headers"]

print(show_what_the_server_sees({}))                 # bare requests: obviously automated
print(show_what_the_server_sees(BROWSER_HEADERS))    # plausible browser navigation

Run both and compare. The bare call reports User-Agent: python-requests/2.32.3 and no Sec-Fetch-* headers at all — a combination that no browser produces and that a rule engine can reject in constant time.

Reading the Block Before Reacting to It

The instinct when a request is refused is to change everything at once: new headers, a proxy, a browser. That makes the problem unfalsifiable. The shape of the refusal tells you which layer rejected you, and each layer has a different fix.

A 403 on the very first request, before any pattern of behaviour exists, is a static check — headers or the TLS handshake. A 200 with a body that is missing the data means you were not blocked at all; the page renders client-side. A 429 after a run of successful requests is a rate limit keyed on your IP or session. A challenge page after some successful browsing is a behavioural score that dropped below a threshold. The diagnostic below distinguishes them mechanically instead of by intuition.

import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,*/*;q=0.8",
}
CHALLENGE_MARKERS = ("cf-challenge", "_cf_chl", "Just a moment", "Checking your browser")

def classify_block(url: str, expected: str) -> str:
    response = requests.get(url, headers=HEADERS, timeout=15, allow_redirects=True)
    body = response.text
    if response.status_code == 429:
        return "rate limit: slow down or change egress IP"
    if response.status_code == 403:
        return "static rejection: headers or TLS fingerprint"
    if any(marker in body for marker in CHALLENGE_MARKERS):
        return "interactive challenge: needs a real browser or a token"
    if response.ok and expected not in body:
        return "not blocked: the page renders client-side"
    return "ok"

print(classify_block("https://books.toscrape.com/", "A Light in the Attic"))

Against a friendly target that prints ok. Against a protected one it prints the layer you need to work on, which is the only thing worth knowing before you change any code.

Matching the TLS Handshake

Before HTTP exists, your client and the server negotiate TLS, and the ClientHello message in that negotiation is a fingerprint. The list of cipher suites, the extensions, their order, the supported groups and signature algorithms — hashed together, these form a JA3 or JA4 signature. Python's requests uses OpenSSL through urllib3, which produces a signature no Chrome build has ever emitted. A defence can therefore reject you before reading your carefully crafted User-Agent, which is why a 403 sometimes survives every header change you try.

The fix is to use a client that reproduces a real browser's handshake. curl_cffi binds to a build of curl patched to impersonate specific browser versions, so the ClientHello matches the User-Agent you claim. The full mechanism is in TLS and JA3 Fingerprint Evasion, and the library-specific detail is in Using curl_cffi to Impersonate Browsers.

# pip install "curl_cffi==0.7.1"
from curl_cffi import requests as curl_requests

def compare_fingerprints() -> None:
    """Same endpoint, two TLS stacks, two different JA3 hashes."""
    for profile in ("chrome124", "safari17_0"):
        response = curl_requests.get(
            "https://tls.browserleaks.com/json",
            impersonate=profile,
            timeout=20,
        )
        payload = response.json()
        print(profile, "->", payload.get("ja3_hash"), payload.get("user_agent", "")[:40])

compare_fingerprints()

Each profile prints a different JA3 hash, and both differ from what plain requests produces against the same endpoint. The important discipline is consistency: an impersonation profile of chrome124 paired with a hand-written User-Agent claiming Firefox is a mismatch that is easier to detect than either signal alone.

Driving a Real Browser with Playwright

When the data only exists after JavaScript has executed, or when the defence requires a script to run and post a token back, a real browser engine is the answer. Playwright is the better default for new work: it auto-waits on selectors instead of making you sprinkle sleeps, it exposes network interception directly, and its async API composes with the rest of an asyncio scraper.

The cost is real. A Chromium instance uses hundreds of megabytes of RAM and takes a second or more to start, so a browser-per-URL design will not scale. Use one browser with many contexts — each context is an isolated cookie jar and cache, which is cheap — and route the pages that genuinely need rendering through it while everything else stays on an HTTP client. Details are in Using Playwright for Modern Web Automation, and the scroll-driven case in Handling Infinite Scroll with Playwright.

# pip install "playwright==1.47.0" && playwright install chromium
import asyncio
from playwright.async_api import async_playwright

async def titles_from(url: str) -> list[str]:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            viewport={"width": 1440, "height": 900},
            locale="en-GB",
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                       "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
        )
        # Skip images and fonts: roughly halves the bytes and the wall-clock time.
        await context.route(
            "**/*",
            lambda route: route.abort()
            if route.request.resource_type in {"image", "font", "media"}
            else route.continue_(),
        )
        page = await context.new_page()
        await page.goto(url, wait_until="domcontentloaded", timeout=30_000)
        await page.wait_for_selector("article.product_pod h3 a", timeout=15_000)
        titles = await page.eval_on_selector_all(
            "article.product_pod h3 a", "nodes => nodes.map(n => n.getAttribute('title'))"
        )
        await browser.close()
        return titles

print(asyncio.run(titles_from("https://books.toscrape.com/"))[:5])

The context.route call is the single highest-value optimisation in browser scraping. Blocking images, fonts, and media typically removes 60–80% of the transferred bytes on a content-heavy page, and it removes them before they are ever requested.

Selenium Where the Target Demands It

Selenium remains the right tool in two situations: an environment where a WebDriver-based stack is already standardised, and a target that behaves differently under Chrome DevTools Protocol automation than under WebDriver. Its drawback is the waiting model — implicit waits apply globally and interact badly with explicit ones, and mixing the two produces timeouts that appear random.

The rule is to pick one and stay with it. Set no implicit wait and use WebDriverWait with an expected condition everywhere, so each wait states what it is waiting for and fails with a message that names it. Full coverage is in Mastering Selenium for Dynamic Websites, and the waiting model specifically in Explicit vs Implicit Waits in Selenium.

# pip install "selenium==4.24.0"
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

def quotes_on_first_page() -> list[str]:
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument("--window-size=1440,900")
    options.add_argument(
        "--user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    )
    driver = webdriver.Chrome(options=options)
    try:
        driver.get("https://quotes.toscrape.com/js/")
        WebDriverWait(driver, 15).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, "div.quote span.text"))
        )
        return [e.text for e in driver.find_elements(By.CSS_SELECTOR, "div.quote span.text")]
    finally:
        driver.quit()

print(quotes_on_first_page()[:3])

That target renders its quotes with JavaScript, so a plain HTTP fetch of the same URL returns an empty container — it is a useful, permanently available way to verify that your browser stack is genuinely executing scripts.

Erasing the Automation Tells

Launching a browser is not the same as looking like one. A default Playwright or Selenium session exposes navigator.webdriver === true, an empty navigator.plugins array, a languages list that does not match the Accept-Language header, a headless-specific WebGL vendor string, and a canvas rendering that hashes to a known automation value. A challenge script reads all of these in a few milliseconds.

The counter-measures fall into two groups. Configuration fixes — a real viewport, a matching locale and timezone, a non-headless-suffixed User-Agent — cost nothing and remove the crudest tells. Patching fixes, injected before page scripts run, rewrite the properties themselves. The full surface is mapped in Browser Fingerprint and Stealth Configuration, with the graphics-specific case in Spoofing Canvas and WebGL Fingerprints and a tooling comparison in undetected-chromedriver vs playwright-stealth.

import asyncio
from playwright.async_api import async_playwright

PATCH = """
Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
Object.defineProperty(navigator, 'languages', { get: () => ['en-GB', 'en'] });
Object.defineProperty(navigator, 'plugins', { get: () => [1, 2, 3, 4, 5] });
window.chrome = window.chrome || { runtime: {} };
"""

async def report_tells() -> dict[str, object]:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(locale="en-GB", timezone_id="Europe/London")
        await context.add_init_script(PATCH)     # runs before any page script
        page = await context.new_page()
        await page.goto("https://httpbin.org/html", wait_until="domcontentloaded")
        result = await page.evaluate(
            "() => ({ webdriver: navigator.webdriver,"
            " languages: navigator.languages,"
            " plugins: navigator.plugins.length,"
            " hasChrome: !!window.chrome })"
        )
        await browser.close()
        return result

print(asyncio.run(report_tells()))

add_init_script is the load-bearing call: it registers the patch on the context so it executes on every page and every iframe before the site's own scripts, which is the only ordering in which the override is observed. Injecting the same code after navigation is too late.

Spreading Load Across IP Addresses

Every other technique on this page is undone by sending ten thousand requests from one address. IP reputation is cheap for a defence to evaluate and expensive for you to work around, and the two variables that matter are the pool's provenance and how you rotate through it.

Datacenter addresses are fast and cheap and belong to ASNs that any defence can enumerate; they are fine for permissive targets and useless against strict ones. Residential and mobile addresses carry the reputation of real subscribers, cost one to two orders of magnitude more per gigabyte, and are noticeably slower. Choose per target rather than by default — the comparison is in Residential vs Datacenter Proxies, and pool management, health checks, and block-aware rotation in Rotating Proxies and Managing IP Blocks.

Rotation strategy matters as much as pool quality. Rotating on every request breaks any session-based flow, because your login cookie now arrives from a new country. Rotate per session instead: bind one proxy to one Session for the life of a logical task, and retire it when it starts collecting refusals.

import itertools
import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
}

class ProxyPool:
    """One proxy per session; a proxy is retired after repeated refusals."""

    def __init__(self, proxies: list[str]) -> None:
        self._cycle = itertools.cycle(proxies)
        self._strikes: dict[str, int] = {p: 0 for p in proxies}

    def session(self) -> tuple[requests.Session, str]:
        proxy = next(self._cycle)
        session = requests.Session()
        session.headers.update(HEADERS)
        session.proxies = {"http": proxy, "https": proxy}
        return session, proxy

    def report(self, proxy: str, status: int) -> None:
        if status in (403, 407, 429):
            self._strikes[proxy] += 1
        else:
            self._strikes[proxy] = 0

    def is_burned(self, proxy: str) -> bool:
        return self._strikes.get(proxy, 0) >= 3

pool = ProxyPool(["http://user:pass@proxy-a.example.com:8000"])
session, proxy = pool.session()
response = session.get("https://httpbin.org/ip", timeout=20)
pool.report(proxy, response.status_code)
print(response.status_code, "burned:", pool.is_burned(proxy))

Replace the placeholder with a real endpoint and https://httpbin.org/ip will echo the exit address, which is the quickest way to confirm that traffic is actually leaving through the proxy rather than through your own connection.

Getting Through a Managed Challenge

Cloudflare, Akamai, DataDome, and their peers do not simply block; they interpose. A managed challenge serves a small page whose script measures the environment, solves a proof-of-work, and posts the result back for a clearance cookie. Everything afterwards depends on that cookie and on continuing to look like the client that earned it.

Three consequences follow. First, you cannot fetch the clearance cookie in a browser and then use it from a plain HTTP client with a different TLS fingerprint — the cookie is bound to the fingerprint that obtained it. Second, the cookie expires, so a long crawl needs to detect a re-challenge and repeat the handshake rather than failing. Third, an explicit human-verification widget is a different thing from a managed challenge and needs a different response. The mechanics are in Bypassing Cloudflare and Akamai Protections, and the widget case in Solving CAPTCHAs with Python.

import time
import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,*/*;q=0.8",
}
CHALLENGED = ("Just a moment", "cf_chl_opt", "Checking your browser")

def fetch_with_challenge_detection(url: str, session: requests.Session) -> str | None:
    """Return the body, or None when a re-challenge is required."""
    for attempt in range(3):
        response = session.get(url, headers=HEADERS, timeout=25)
        if response.status_code in (403, 503) or any(m in response.text for m in CHALLENGED):
            time.sleep(2 ** attempt)      # 1s, 2s, 4s before deciding it is real
            continue
        response.raise_for_status()
        return response.text
    return None

with requests.Session() as s:
    body = fetch_with_challenge_detection("https://books.toscrape.com/", s)
    print("challenged" if body is None else f"{len(body)} bytes")

Detecting the challenge explicitly, rather than letting raise_for_status() throw, is what lets a long-running crawl re-establish clearance and continue instead of dying at hour three.

Reading the Mobile App's API Instead

The most effective move is often to stop attacking the web front end. A site's mobile application usually talks to a JSON API that carries the same data, and that API is frequently defended far less aggressively — it has to serve clients on unpredictable mobile IPs, with older app versions, over flaky connections, so aggressive blocking would break real users. There is often no JavaScript challenge at all, because there is no browser to run it in.

The workflow is to proxy the device's traffic through an intercepting proxy, observe the requests the app makes, then replay them from Python with the same headers and authentication. Certificate pinning is the usual obstacle and has known handling. The whole approach is covered in Scraping Mobile App APIs, with the capture step in Intercepting App Traffic with mitmproxy, the pinning problem in Handling Certificate Pinning in API Analysis, and the replay step in Replaying Mobile API Requests in Python.

import requests

APP_HEADERS = {
    "User-Agent": "ExampleApp/8.4.1 (Android 14; Pixel 7)",
    "Accept": "application/json",
    "X-App-Version": "8.4.1",
    "X-Platform": "android",
    "Accept-Language": "en-GB",
}

def call_app_api(endpoint: str, cursor: str | None = None) -> dict:
    params: dict[str, str | int] = {"limit": 50}
    if cursor:
        params["cursor"] = cursor
    response = requests.get(endpoint, params=params, headers=APP_HEADERS, timeout=20)
    response.raise_for_status()
    return response.json()

payload = call_app_api("https://httpbin.org/get")
print(payload["headers"]["User-Agent"], payload["args"])

Against httpbin.org this simply echoes the headers back, which is the point: it confirms that the app-shaped request you assembled is being transmitted exactly as intended before you aim it at a real endpoint.

Pacing, the Layer You Cannot Buy Around

Every technique above can be purchased or coded. Behaviour cannot. Once the static checks pass, what remains is a scoring model watching the shape of your traffic, and the tells it looks for are the ones a script produces naturally: a perfectly constant interval between requests, a URL sequence that walks the site in exact index order, a session that requests only documents and never a stylesheet, and a client that never once follows an internal link it did not compute.

Fixing this is unglamorous and cheap. Add jitter so the interval distribution is not a spike. Back off multiplicatively when a refusal appears and, crucially, recover slowly rather than jumping straight back to full rate. Warm a session by fetching the listing page before the detail pages it links to, so the Referer chain is real. None of that requires a browser.

import random
import time

class Pacer:
    """Jittered delay with multiplicative back-off and gradual recovery."""

    def __init__(self, base: float = 1.0, ceiling: float = 60.0) -> None:
        self.base = base
        self.ceiling = ceiling
        self.current = base

    def wait(self) -> None:
        time.sleep(self.current * random.uniform(0.7, 1.4))

    def observe(self, status: int) -> None:
        if status in (403, 429, 503):
            self.current = min(self.current * 2.0, self.ceiling)
        else:
            self.current = max(self.base, self.current * 0.9)

pacer = Pacer(base=1.0)
for status in (200, 200, 429, 200, 200, 200):
    pacer.wait()
    pacer.observe(status)
    print(status, f"next delay ~{pacer.current:.2f}s")

The asymmetry is deliberate: double on failure, shrink by ten percent on success. A crawl that halves its delay the moment one request succeeds will oscillate between blocked and unblocked all day and never finish.

Common Pitfalls

• Changing five things at once. A defence gives you exactly one bit of feedback per request. Change one variable, re-test, and record the result, or you will end up paying for residential proxies to solve a header problem.
• Inconsistent identity across layers. A Chrome User-Agent with a Firefox TLS fingerprint, an Accept-Language of en-US with a Berlin exit IP, and a timezone of UTC in a browser claiming to be in London are each more detectable than any single wrong value.
• Rotating the IP mid-session. Session-based flows break when the cookie that was issued to a Frankfurt address arrives from São Paulo. Bind a proxy to a session, not to a request.
• Reaching for a browser first. Rendering is the slowest and most expensive option and often unnecessary. Check for a JSON endpoint before you install Chromium — Data Extraction Patterns and APIs covers finding it.
• Treating a clearance cookie as portable. It is bound to the fingerprint that earned it. Carrying it to a client with a different TLS handshake produces an immediate re-challenge.
• Ignoring the request rate once evasion works. Passing the fingerprint checks only moves you to the behavioural layer, where an inhuman request cadence is exactly what is being measured.
• Using free proxy lists. They are enumerated and flagged, frequently intercept traffic, and their success rate against a defended target rounds to zero. See Best Free and Paid Proxy Providers for Scraping.

Frequently Asked Questions

Why do I get a 403 with requests but the page loads fine in my browser? Most often the TLS fingerprint. The rejection happens during the handshake, before your headers are read, which is why editing the User-Agent changes nothing. Test with a client that impersonates a real browser handshake, such as curl_cffi; if that succeeds, the handshake was the cause and no amount of header work would have fixed it.

Do I need a headless browser to scrape a JavaScript-heavy site? Usually not. Most single-page applications hydrate from an endpoint that returns clean JSON, and calling it directly is faster, cheaper, and less fragile than rendering. Reserve a browser for pages where the values are computed in the client, or where a challenge script has to execute to earn a clearance cookie.

Which is better for stealth, Playwright or Selenium? Neither is stealthy by default; both expose navigator.webdriver and a headless-specific rendering profile until you patch them. Playwright is the better starting point because add_init_script gives you a clean way to install patches before page scripts run, and because its context model makes it cheap to isolate identities. The tooling comparison is in undetected-chromedriver vs playwright-stealth.

How many requests can I send before a rate limit triggers? There is no portable number; it depends on the target's capacity and its own thresholds, and it is often keyed on a combination of IP, session, and endpoint. Start at roughly one request per second per IP, watch for the first 429, and treat that as the ceiling rather than as a starting point to push against.

Is rotating user agents enough to avoid detection? On its own, no — it is the cheapest layer and the least informative. Rotating the User-Agent while leaving the TLS fingerprint, header order, and IP unchanged produces a client that claims to be five different browsers from one address with one handshake, which is more suspicious than a single consistent identity. See How to Rotate User Agents in Python.

Is it legal to use these techniques? That question has no general answer and this site does not give legal advice. Legality depends on your jurisdiction, the target's terms of service, whether you are circumventing an access control, and what kind of data is involved. Resolve it before you build, prefer a published API where one exists, and never collect personal data you have no basis to hold.

Related

• Mastering Selenium for Dynamic Websites — WebDriver, waiting models, and legacy targets
• Using Playwright for Modern Web Automation — contexts, auto-waiting, and network interception
• Rotating Proxies and Managing IP Blocks — pool health, rotation strategy, and block detection
• Bypassing Cloudflare and Akamai Protections — managed challenges and clearance cookies
• TLS and JA3 Fingerprint Evasion — what the handshake reveals and how to change it
• Browser Fingerprint and Stealth Configuration — the JavaScript-visible surface of an automated browser
• Scraping Mobile App APIs — the less defended interface to the same data
• The Complete Guide to Python Web Scraping — the fundamentals this path builds on
• Scaling Python Web Scrapers — running an evasion-aware crawl at volume
• Data Extraction Patterns and APIs — finding the JSON before deciding to render