Bypassing Cloudflare and Akamai Protections

Edge bot management sits in front of a large share of the commercial web, and it is the layer most often responsible when a Python client that works everywhere else returns a wall of 403s. This guide belongs to Advanced Scraping Techniques and Anti-Bot Evasion and explains the mechanism: what these systems measure, how a challenge is issued and cleared, and how to work out which specific signal is failing so you can fix a misconfigured client rather than guess. It is written for people collecting data they are entitled to collect — from their own properties, from partners, from sites whose terms permit it. A protection you are not authorised to pass is a refusal, and the correct response to a refusal is to ask for access or use the published API, not to escalate.

When Edge Protection Is Actually Your Problem

Before spending a day on fingerprints, confirm the edge is what is refusing you. The failure signatures are distinctive.

The distinction that matters most: a challenge is an invitation to prove something, while a block is a decision. A challenge page carries a script and a submission endpoint; a block carries an error code and nothing to submit. If you consistently receive blocks rather than challenges, no amount of client tuning is the answer — the site has classified you and is declining. Sites with a data-access programme, an official API, or a bulk-download endpoint almost always want you to use it, and that route is faster to implement and far more stable than anything described below.

Prerequisites

Python 3.10 or newer. Diagnosis needs a client that can present a genuine browser handshake and, for the challenge stage, a real browser.

pip install "curl_cffi>=0.7" "playwright>=1.44" "httpx>=0.27"
playwright install chromium

You also want a way to see what the server sees. A TLS inspection endpoint reports the fingerprint your client actually presented, which is the single most useful diagnostic in this whole area:

python -c "from curl_cffi import requests; print(requests.get('https://tls.peet.ws/api/all').json()['tls']['ja3_hash'])"

Run the same request through stock httpx and compare the two hashes. If they differ — and they will — you have just measured the gap the edge is scoring.

Step-by-Step: Diagnosing an Edge Refusal

1. Read the response, not just the status code

Both major vendors put their identity in the response headers, and the body distinguishes a challenge from a block. Capture all three before changing anything.

import httpx

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}

VENDOR_HEADERS = ("cf-ray", "cf-mitigated", "server", "x-akamai-request-id", "akamai-grn")

def diagnose(url: str) -> dict[str, str]:
    """Return the status, vendor headers and body signature for one request."""
    with httpx.Client(follow_redirects=True, timeout=20.0) as client:
        response = client.get(url, headers=HEADERS)
    body = response.text[:6000].lower()
    return {
        "status": str(response.status_code),
        "length": str(len(response.content)),
        "vendor": ", ".join(
            f"{k}={response.headers[k]}" for k in VENDOR_HEADERS if k in response.headers
        ),
        "challenge": str(any(
            marker in body
            for marker in ("just a moment", "checking your browser", "cdn-cgi/challenge")
        )),
    }

if __name__ == "__main__":
    print(diagnose("https://books.toscrape.com/"))

A cf-ray header identifies Cloudflare; x-akamai-request-id or a server: AkamaiGHost value identifies Akamai. cf-mitigated: challenge states outright that a challenge was served. Record the cf-ray value: if you ever need to contact the site owner about being wrongly blocked, that identifier lets them find your request in their logs in seconds, and asking is very often the fastest resolution available.

2. Understand the ladder you are on

These systems do not make a binary allow/deny decision. They compute a score from many weak signals and pick the cheapest response that resolves the uncertainty.

The first rung costs nothing and you never notice it. The second rung — a managed challenge — serves a small page whose script performs some work in the browser and posts the result back; passing it yields a clearance cookie. The third rung requires a human. The fourth is a refusal.

Each rung down the ladder is a message about the signals you are sending. Being pushed from rung one to rung two by a change in your client is diagnostic information: whatever you altered moved your score. Being pinned at rung four regardless of what you send means the decision is not about your client at all — it is about your IP range, your ASN, or a rule the site has deliberately configured. That is the point to stop tuning and start asking.

3. Present an honest, consistent client

Most scrapers fail these checks not because they are hiding, but because they are internally contradictory: a Chrome User-Agent on an OpenSSL handshake, or Chrome client hints with no Sec-Fetch-* headers. The signal that scores badly is the mismatch.

from curl_cffi import requests

BROWSER_HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept": (
        "text/html,application/xhtml+xml,application/xml;q=0.9,"
        "image/avif,image/webp,image/apng,*/*;q=0.8"
    ),
    "Accept-Language": "en-US,en;q=0.9",
    "Sec-Ch-Ua": '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": '"Windows"',
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
}

def fetch_aligned(url: str) -> tuple[int, int]:
    """Fetch with a Chrome TLS profile and a matching Chrome header set."""
    with requests.Session(impersonate="chrome124") as session:
        response = session.get(url, headers=BROWSER_HEADERS, timeout=25)
        return response.status_code, len(response.content)

if __name__ == "__main__":
    print(fetch_aligned("https://books.toscrape.com/"))

The rule is one identity, declared consistently at every layer. If you claim Chrome 124, the TLS profile, the client hints and the Sec-Fetch-* set must all be Chrome 124's. Claiming Chrome 124 while impersonating Chrome 110's handshake reintroduces exactly the drift you removed. The mechanics of why the handshake is decisive — and why it is read before a single header exists — are worked through in the TLS guide linked above.

4. Diff your client against a real browser

Guessing which header is missing wastes days. Capture what a real browser sends, capture what your client sends, and compare the two mechanically. An echo endpoint that reflects the request back gives you both halves without a proxy.

import json
from curl_cffi import requests as cffi_requests

ECHO_URL = "https://httpbin.org/headers"

def echo_headers(**session_kwargs) -> dict[str, str]:
    """Return the headers an echo service observed for one request."""
    with cffi_requests.Session(**session_kwargs) as session:
        response = session.get(ECHO_URL, timeout=20)
        response.raise_for_status()
        return {k.lower(): v for k, v in response.json()["headers"].items()}

def diff_headers(reference: dict[str, str], candidate: dict[str, str]) -> dict[str, list[str]]:
    """Report headers the reference sends that the candidate does not, and vice versa."""
    return {
        "missing": sorted(set(reference) - set(candidate)),
        "extra": sorted(set(candidate) - set(reference)),
        "different": sorted(
            k for k in set(reference) & set(candidate) if reference[k] != candidate[k]
        ),
    }

if __name__ == "__main__":
    chrome_like = echo_headers(impersonate="chrome124")
    plain = echo_headers()
    print(json.dumps(diff_headers(chrome_like, plain), indent=2))

Run the same URL through a headful browser with DevTools open, copy the request as cURL, and use that as the reference set. The missing list is your immediate work queue. Pay particular attention to header order as well as presence: browsers emit a stable order and many stock clients sort alphabetically, which is itself a distinguishing signal that no individual header value reveals.

Three families account for most gaps in practice. The Sec-Fetch-* set describes the navigation context and is absent from every naive client. The Sec-Ch-Ua* client hints describe the browser brand and platform, and their version must agree with the User-Agent. And Accept-Encoding should advertise the compressions you can actually decode — advertising br and then failing on a Brotli body is a self-inflicted error that looks like a block.

5. Let a real browser earn the clearance cookie

A managed challenge is a piece of JavaScript. No HTTP client can evaluate it, which is why the honest solution is to run a real browser for the challenge and then reuse what it earned.

import asyncio
import json
from pathlib import Path
from playwright.async_api import async_playwright

STATE = Path("edge_state.json")

async def earn_clearance(url: str, settle_ms: int = 8000) -> dict[str, str]:
    """Load a challenged page in a real browser and export the resulting cookies."""
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        context = await browser.new_context(
            locale="en-US",
            timezone_id="America/New_York",
            viewport={"width": 1440, "height": 900},
        )
        page = await context.new_page()
        await page.goto(url, wait_until="domcontentloaded", timeout=60000)
        await page.wait_for_timeout(settle_ms)
        cookies = {c["name"]: c["value"] for c in await context.cookies()}
        STATE.write_text(json.dumps(cookies, indent=2), encoding="utf-8")
        await browser.close()
        return cookies

if __name__ == "__main__":
    print(sorted(asyncio.run(earn_clearance("https://books.toscrape.com/"))))

Two constraints govern what you can do with the result. The clearance cookie is bound to the client that earned it — the IP address and, in current implementations, the TLS fingerprint — so moving it to a different machine or a different proxy exit invalidates it immediately. And it expires, typically within an hour, which means a long crawl needs a refresh strategy rather than a one-off capture. Running the browser headful (headless=False) removes a whole family of environment differences at the cost of needing a display or a virtual framebuffer.

6. Reuse the session, and pace it

Once you hold a valid session, the cheapest thing you can do is keep it. Re-triggering a challenge on every request is both slow and the clearest possible signal that you are not a browser.

import random
import time
from curl_cffi import requests

def make_session(cookies: dict[str, str]) -> requests.Session:
    """Build an impersonating session seeded with browser-earned cookies."""
    session = requests.Session(impersonate="chrome124")
    session.headers.update(BROWSER_HEADERS)
    for name, value in cookies.items():
        session.cookies.set(name, value)
    return session

def paced_get(session: requests.Session, url: str, min_gap: float = 1.5) -> requests.Response:
    """Fetch with a randomised gap so request timing is not machine-regular."""
    time.sleep(min_gap + random.expovariate(1.0))
    response = session.get(url, timeout=25)
    if response.status_code in (403, 429, 503):
        raise RuntimeError(
            f"session no longer accepted ({response.status_code}); re-earn clearance"
        )
    return response

random.expovariate produces exponentially distributed gaps, which is what independent human arrivals actually look like; a fixed time.sleep(2) produces a metronome, and a metronome is trivially detectable. Raising the exception rather than retrying is deliberate — when a session stops being accepted, hammering it is exactly the wrong response. Re-earn the session, or stop.

7. Know when to stop

If you have aligned TLS, headers and browser environment, paced your requests conservatively, and the answer is still a hard block from every address you try, the site has decided. Escalating past that point — rotating through consumer IPs to evade a deliberate rule, or paying a service to defeat an interactive challenge on a site that has refused you — moves from technical work to unauthorised access, with contractual and in many jurisdictions legal consequences. The productive alternatives are genuinely better: check for an official API, look for a bulk export or a data partnership, ask the site owner (with your cf-ray in hand), or find the same data in a public source. For mobile-first products, the sanctioned client is often a different surface entirely, which is the subject of Scraping Mobile App APIs.

Performance and Scaling Considerations

Challenge handling is expensive, so the metric to optimise is challenges per thousand requests, not requests per second. A managed challenge costs one to eight seconds of browser time and roughly 200 MB of resident memory for the duration; a plain fetch with a valid cookie costs 50-300 ms and no browser at all. A pipeline that earns a session in a browser and then serves several hundred requests from a lightweight impersonating client therefore outperforms a browser-only pipeline by one to two orders of magnitude.

That leads to a simple architecture: a small pool of browser workers whose only job is to mint sessions, and a much larger pool of HTTP workers that consume them. Store each session as a bundle of cookies, the exit IP it was earned on, and its issue time, and treat those three as inseparable. Refresh a session before it expires rather than after, so a refresh never blocks a live request. Expect to need one browser worker per twenty to fifty HTTP workers, tuned by measuring how long your sessions actually survive.

Watch the tail rather than the mean. A healthy pipeline shows a stable, low challenge rate; a rising rate over an hour means your identities are being learned, and continuing at the same volume will convert challenges into blocks. Treat challenge rate as a load signal and back off when it climbs — the same feedback loop described in Detecting Silent Scraper Failures, which also covers the case where you keep getting 200s that contain nothing.

Cache aggressively. Every request you do not repeat is a challenge you do not risk, and conditional requests with ETag and If-Modified-Since turn most re-crawls into cheap 304s.

Finally, keep the browser tier small on purpose. Browser workers are the part of the system that is expensive, fragile and slow to start, so treat them as a scarce resource with a queue in front of them rather than as something every task may launch. A single session-minting service with a bounded worker pool, a health check and an explicit failure mode is far easier to reason about than session logic scattered across every spider.

Common Errors and Fixes

403 with cf-mitigated: challenge on every request, including the first. Your handshake is being scored before any header is read. Confirm by comparing JA3 hashes between your client and a real browser; if they differ, move to an impersonating client rather than adding more headers.

curl_cffi.requests.errors.RequestsError: Failed to perform, ErrCode: 35. The impersonation profile is not available in your installed version. Run pip install -U curl_cffi and select a profile the version documents. Profiles are removed as browser versions age, so this breaks on upgrade if you pin an old target.

Challenge page loads in Playwright and never resolves. The page is waiting for a script that a blocked resource type would have provided. If you registered a route that aborts stylesheets or scripts, disable it for the challenge navigation — challenge pages need their own assets.

Clearance cookie works once, then returns 403. Either it expired, or you used it from a different exit IP than the one that earned it. Bind the cookie bundle to its proxy endpoint and discard both together; never share a session across your pool.

httpx.ConnectError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] against an Akamai-fronted host. The server rejected your cipher list outright at the TLS layer. This is the handshake being refused rather than an application decision, and only a client with a browser-grade TLS stack will complete it.

Everything works locally and fails from your server. Datacenter ASNs carry materially worse reputation than consumer ISPs at the edge, and many rules key on ASN directly. Verify by running the identical script from a residential connection; if it passes, the difference is network reputation, not code.

Interactive challenge appears where a managed challenge used to. Your score has degraded, usually from volume or from an identity the site has now learned. Reduce request rate substantially and re-earn sessions less frequently; if the interactive challenge persists, treat it as a refusal rather than an obstacle.

200 responses that contain a few hundred bytes and no data. An edge can serve a stub rather than an error, which passes every status-code check you wrote. Assert on a field you always expect — a product title, a row count, a JSON key — and treat its absence as a failed request rather than an empty result set.

Redirect loop between the page and a challenge path. Your client is not storing the cookie the challenge sets, usually because redirects are being followed by a client with cookie persistence disabled. Use a session object rather than one-shot requests, and confirm the cookie jar is non-empty after the first response.

Different results from the same code on two machines. Compare the two environments at the TLS layer first: a different curl_cffi version, a corporate TLS-inspecting middlebox, or a VPN can each rewrite the handshake without touching a line of your code. The inspection endpoint settles the question in one request.

Frequently Asked Questions

Can plain requests get past a managed challenge? No. requests delegates TLS to the system OpenSSL, so its handshake is recognisable before any header is sent, and it cannot execute the JavaScript a managed challenge serves. An impersonating client fixes the first problem; only a real browser fixes the second.

Why am I still challenged after switching to residential proxies? Because the IP is one input among many. A residential address paired with a stock Python handshake, a header set no browser sends, or a headless environment with an obviously synthetic WebGL renderer still scores poorly. Fix the client signals first; the IP change only helps once everything else is consistent.

How long does a clearance cookie last? Typically between thirty minutes and a few hours, and the site owner configures it. Treat the lifetime as unknown, refresh proactively on a timer well inside the shortest value you have observed, and always handle the case where a request comes back challenged mid-batch.

Is it legal to work around these protections? It depends entirely on your relationship with the site. Accessing your own property, a partner's system under contract, or a site whose terms permit automated collection is ordinary engineering. Circumventing a technical measure on a system you have no authorisation to access is not, and being able to do it technically does not make it permissible.

What should I do when a site blocks me but I have a legitimate need for the data? Ask. Provide the cf-ray or Akamai request identifier from a blocked response, explain your use case and your rate, and request an allowlist entry or API credentials. Site owners routinely grant access to identifiable, well-behaved clients, and a sanctioned integration is more stable than any evasion technique.

Related

• Advanced Scraping Techniques and Anti-Bot Evasion — the parent section for this topic
• Solving CAPTCHAs with Python — what the interactive rung of the ladder involves
• Using curl_cffi to Impersonate Browsers — the impersonating client in detail
• Mastering Selenium for Dynamic Websites — the WebDriver route to running challenge scripts
• Managing Cookies and Sessions — storing and replaying the session a browser earned