Python Web Scraping
Reading layout

Handling CSRF Tokens When Scraping

A cross-site request forgery token is the single most common reason a correctly built form submission gets refused, and this article — part of Handling Forms and Authentication — covers where the value lives, where it has to be sent, and how to read a 403 that arrives with a perfectly good session cookie attached.

Three locations a CSRF token can be published in A server-issued token reaches the client either as a hidden form input, as a meta tag in the document head, or as a cookie that must be mirrored into a request header. Server issues a tokenbound to your sessionHidden inputinside the form markupname="csrfmiddlewaretoken"goes in the POST bodyMeta tagin the document headname="csrf-token"read the content attributeCookieset on the first GETcsrftoken=abc123echoed in X-CSRFToken
The same token value reaches your client through one of three channels. Check all three before deciding a site has no CSRF protection.

A CSRF token is a value the server generates, ties to your session, and requires back on any state-changing request. It is not a secret in the way a password is: it is published to your own client in plain sight. What makes it work is that a page on another origin can cause your browser to send your cookie but cannot read the page or the cookie, so it cannot supply the matching value. For a scraper the practical consequence is simple — the token must be read from the response you just received, in the same session, and sent back in the exact place the server looks for it.

How the Token Is Bound to Your Session

The server keeps, or can derive, the correct token for the session identified by your cookie. It compares what you sent against what it expects for that session. Three consequences follow directly, and each one is a bug people hit.

A token from a different session is worthless. Fetching the login page with requests.get and then posting with a separate call means the two requests belong to different sessions, so the token cannot match. Use one session object for both.

A token from a previous run is worthless. Tokens are commonly regenerated on login, on logout, and on a schedule. Caching one in a config file works for a day and then fails permanently.

A token can be single-use. Some frameworks rotate the value on every response, so the token you extracted from page A is invalid the moment page B is served. When that happens, re-read the token from the most recent response before each submission rather than holding one for the session.

Stateless variants exist too. A signed double-submit token encodes the session binding into the value itself using an HMAC, so the server stores nothing. From the client's point of view the handling is identical: read it fresh, send it back unchanged.

The Three Places the Value Hides

A hidden input inside the form. The classic server-rendered case. Django uses csrfmiddlewaretoken, Rails uses authenticity_token, Laravel uses _token, Spring uses _csrf. It travels in the request body alongside the other fields, which is why building the payload from the parsed form — rather than from a hard-coded dict — handles it without any special code.

A meta tag in the document head. Single-page applications need the token available to JavaScript, so it is published as <meta name="csrf-token" content="...">. The framework's own fetch wrapper copies it into a request header. Your client has to do the same: reading the meta tag and sending the body without the header will still fail.

A cookie mirrored into a request header. The double-submit pattern. The server sets a readable cookie such as csrftoken or XSRF-TOKEN on the first GET, and expects the same value echoed in X-CSRFToken or X-XSRF-TOKEN. Angular's HTTP client does this by default, which is why XSRF-TOKEN shows up so often on APIs behind an Angular front end.

The double-submit cookie verification sequence A cookie arrives, the client echoes the same value in a request header, and the server compares the two: matching values return 200 while a mismatch returns 403. Double-submit cookie check1. Cookie arrivesSet-Cookie: csrftokenon the first GET2. You echo itX-CSRFToken: samecookie still sent too3. Server comparesheader against cookiebyte for byteWhy it worksa cross-site page cannot read your cookieMatch200 and a bodyMismatch403, session intact
Double-submit works because a cross-origin page can send your cookie but cannot read it, so it cannot put the matching value in a header.

Note that in the double-submit case the cookie is deliberately not HttpOnly — front-end code must be able to read it. That is not a misconfiguration, it is the mechanism. It also means the value is trivially available to your client through the session's cookie jar, with no HTML parsing at all.

Reading and Sending the Token

Both variants in one runnable script. The first function handles a token in the markup; the second handles the cookie-to-header form. httpbin.org echoes the request back, so you can confirm what actually went over the wire.

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

FIELD_NAMES = (
    "csrfmiddlewaretoken",  # Django
    "authenticity_token",   # Rails
    "_token",               # Laravel
    "_csrf",                # Spring, Express
    "csrf_token",           # generic
)
META_NAMES = ("csrf-token", "csrf_token", "_csrf", "X-CSRF-TOKEN")


def token_from_html(html: str) -> tuple[str, str] | None:
    """Return (where, value) for a token found in a hidden input or a meta tag."""
    soup = BeautifulSoup(html, "lxml")

    for name in FIELD_NAMES:
        field = soup.find("input", attrs={"name": name})
        if field is not None and field.get("value"):
            return ("input", field["value"])

    for name in META_NAMES:
        meta = soup.find("meta", attrs={"name": name})
        if meta is not None and meta.get("content"):
            return ("meta", meta["content"])

    return None


def token_from_cookies(session: requests.Session) -> tuple[str, str] | None:
    """Return (header_name, value) for a token published as a readable cookie."""
    pairs = {
        "csrftoken": "X-CSRFToken",
        "XSRF-TOKEN": "X-XSRF-TOKEN",
        "_csrf": "X-CSRF-Token",
        "csrf_token": "X-CSRF-Token",
    }
    jar = session.cookies.get_dict()
    for cookie_name, header_name in pairs.items():
        if jar.get(cookie_name):
            return (header_name, jar[cookie_name])
    return None


def submit_with_token(session: requests.Session, page_url: str, post_url: str,
                      payload: dict[str, str]) -> requests.Response:
    """Fetch the page, harvest whatever token it publishes, then submit."""
    page = session.get(page_url, timeout=20)
    page.raise_for_status()

    body = dict(payload)
    headers = {"Referer": page.url}

    found = token_from_html(page.text)
    if found is not None:
        where, value = found
        if where == "input":
            body["csrf_token"] = value
        else:
            headers["X-CSRF-Token"] = value

    from_cookie = token_from_cookies(session)
    if from_cookie is not None:
        header_name, value = from_cookie
        headers[header_name] = value

    return session.post(post_url, data=body, headers=headers, timeout=20)


session = requests.Session()
session.headers.update(HEADERS)
session.cookies.set("csrftoken", "demo-token-123", domain="httpbin.org", path="/")

response = submit_with_token(
    session,
    page_url="https://httpbin.org/html",
    post_url="https://httpbin.org/post",
    payload={"identity": "example", "comment": "hello"},
)
echoed = response.json()
print(echoed["form"])
print({k: v for k, v in echoed["headers"].items() if "CSRF" in k.upper()})

The function checks the markup first and the cookie jar second, and sends both when both are present. That is not wasteful — a server that only reads one of them ignores the other, and sending both removes the need to know in advance which variant a given endpoint uses. The Referer header is included for the same reason: several frameworks pair a token check with an origin check, and a missing Referer fails the second one even when the token is correct.

For a JSON API the shape changes slightly. Send the body with json= instead of data=, and put the token only in the header — a JSON endpoint has nowhere to read a form field from.

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": (
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    )
})
session.cookies.set("XSRF-TOKEN", "abc987", domain="httpbin.org", path="/")

response = session.post(
    "https://httpbin.org/post",
    json={"query": "example"},
    headers={
        "X-XSRF-TOKEN": session.cookies.get("XSRF-TOKEN"),
        "Origin": "https://httpbin.org",
        "Referer": "https://httpbin.org/",
    },
    timeout=20,
)
print(response.json()["json"], response.status_code)

Reading a 403 That Arrives with a Valid Session

A 403 on a request whose session cookie is fine is almost always one of four things, and the four have different fixes.

Causes of a 403 response on an authenticated request Four rows pairing a cause of a 403 response — missing token, stale token, token in the wrong place, and missing Origin or Referer headers — with the change that fixes each one. 403 while the session is still goodWhat fixes itThe token was never senthidden input skipped when building the payloadre-read every inputThe token is stalereused after the server rotated itre-fetch the page firstRight value, wrong placesent in the body when a header is requiredmove it to X-CSRFTokenOrigin or Referer missingthe framework checks those headers toosend both headers
A 403 on a request that carries a good session cookie is almost always a token problem, and each cause has a different fix.
  • The token was never sent. Usually because the payload was hard-coded rather than built from the form. Print the keys you are about to submit and compare them against the input names on the page.
  • The token is stale. The server rotated it and you are replaying an old value. Re-fetch the page immediately before the submission and take the token from that response.
  • The value is right but in the wrong place. A body field when the server wants a header, or the reverse. Send both when you are unsure.
  • Origin or Referer is missing. Framework CSRF middleware frequently checks these in addition to the token, and a plain requests call sends neither. Set them to the site's own scheme and host.

Work through them in that order, because each check is cheap and the first three are settled by printing three values: the payload keys, the request headers, and the cookie jar. A minute spent printing those beats an hour spent adjusting header casing on a request that was missing the token entirely.

There is a fifth possibility that looks identical from the outside: the session cookie is not being sent at all. A 403 on a request that carries no cookie is not a token failure, it is a session failure, and no amount of token handling will fix it. Confirm by printing session.cookies.get_dict() immediately before the call — an empty dict, or a jar whose entries carry a domain that does not match the host you are posting to, is the whole explanation.

A useful discriminator: if the response body is HTML containing a phrase like "CSRF verification failed" or "invalid authenticity token", it is definitely the token. If it is a generic block page or a challenge, the request never reached the application, and the problem is a protection layer rather than a token — that territory is covered in Bypassing Cloudflare and Akamai Protections.

Edge Cases and Caveats

  • Per-request rotation. Some deployments return a fresh token on every response and invalidate the previous one immediately. Read the token from the newest response you hold, and never cache it beyond the next submission.
  • SameSite=Strict cookies. A strict cookie is not sent on cross-site navigations. This rarely affects a scraper, which always originates its own requests, but it explains why a browser-exported cookie sometimes behaves differently from a freshly fetched one.
  • Two tokens on one page. A page can carry a token for the form and a separate one for the site's API layer. If a submission fails while another succeeds, check that you are not mixing the two values.
  • Tokens tied to a form instance. Drupal's form_build_id and similar identifiers bind the token to one rendered instance of the form. Reusing the token with a different build identifier fails even though both values are current.
  • The token is not in the HTML at all. If neither a hidden input, a meta tag nor a cookie carries a plausible value, the front end is fetching it from an endpoint such as /api/csrf. Find it in the network panel — the method is described in Finding Hidden API Endpoints in Network Traffic.
  • Concurrency and rotation do not mix. If the server rotates tokens per response, parallel submissions from one session will invalidate each other. Serialise state-changing requests and keep concurrency for reads.
  • Header name casing. HTTP header names are case-insensitive per the specification, but a handful of application-level checks compare strings directly. If a correct token is still refused, try the exact casing the site's own front end uses.

Frequently Asked Questions

Is a CSRF token secret? No. It is published to your own client in the page or in a readable cookie, and it is meant to be. Its security value comes from the same-origin policy — another site can make your browser send your cookies but cannot read the token to go with them. That is also why reading it in a scraper is trivial and entirely legitimate for your own session.

Why does the same token work once and then fail? The server rotates it. Single-use and per-response rotation are both common, particularly on login and logout endpoints. Fetch the page again immediately before each submission and take the token from that response rather than holding one for the whole run.

Do I need to send the token in both the body and a header? Only one is required, but sending both is harmless and saves you from having to identify the variant in advance. A server reading the form field ignores an unexpected header, and one reading the header ignores an extra body key.

I send the token and still get 403 — what else is checked?Origin and Referer. Django, Rails and several others verify that a state-changing request appears to come from the site itself, and a bare HTTP client sends neither header. Set both to the site's scheme and host, and confirm the session cookie is actually being attached by printing the cookie jar before the request.

Back to home