Python Web Scraping
Reading layout

Scraping Behind Bearer Tokens and OAuth

When a sign-in returns JSON instead of setting a cookie, the site is using token authentication, and this article — part of Handling Forms and Authentication — covers obtaining that token for your own account, sending it correctly, and handling the expiry that will otherwise stop a long run halfway through.

Three origins of a bearer token A JSON login endpoint, a value read out of browser storage, and an OAuth client-credentials exchange all converge on the same Authorization header sent with every request. Login endpointPOST /api/login returns JSONBrowser storagelocalStorage value you copy outOAuth exchangeclient_credentials grantAuthorization: Bearerone header on every requestwith an expiry you must plan for
Whatever the origin, the token ends up as one header. The difference is how you obtain it and how long it stays valid.

A bearer token is a string the server accepts as proof of identity in an Authorization header. There is no cookie jar and no per-request state; the header alone decides whether the request is authenticated. That makes token auth pleasantly simple to script and slightly more dangerous to hold, because anyone with the string has your access until it expires. As with every other flow in this section, the assumption is that the token belongs to an account you own on a service whose terms permit automated access.

Where the Token Comes From

Three origins account for nearly everything you will meet.

A login endpoint that returns JSON. You POST credentials to something like /api/auth/login and receive {"access_token": "...", "expires_in": 3600}. This is the most common shape behind a single-page application, and it is usually the same endpoint the site's own front end calls.

A value already in browser storage. Sign in by hand, open the developer tools, and read localStorage or sessionStorage. This is the fastest way to get moving and the worst way to stay running: the value expires, often within the hour, and there is no code path to renew it. Use it to prove an endpoint works, then automate the acquisition properly.

An OAuth token exchange. If the service issues client credentials, you POST grant_type=client_credentials with a client id and secret to a token endpoint and receive an access token in return. This is the only one of the three designed for machines, and it is the one to prefer — it usually comes with published rate limits and a stable contract, which no HTML page ever offers.

There is a fourth case worth naming so you can rule it out: the authorization-code grant, the flow with a browser redirect and a consent screen. It is designed for an application acting on behalf of a human, and it cannot be completed head-lessly without a user present. If a service only offers that flow, complete it once by hand and keep the refresh token it produces.

Sending the Header

The header format is defined by RFC 6750 and there is exactly one correct shape: the scheme, one space, the token.

import os

import httpx


def api_headers(token: str) -> dict[str, str]:
    return {
        "Authorization": f"Bearer {token}",
        "Accept": "application/json",
        "User-Agent": (
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
            "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
        ),
    }


token = os.environ.get("API_TOKEN", "demo-token")
response = httpx.get(
    "https://httpbin.org/bearer", headers=api_headers(token), timeout=20.0
)
print(response.status_code, response.json())

https://httpbin.org/bearer returns 401 without a bearer header and echoes the token back with it, which makes it a useful target for confirming your header construction before pointing the code at a real service.

Three mistakes account for most 401 responses that are not expiries. Writing Authorization: {token} without the Bearer prefix. Including the surrounding quotes when copying the value out of a JSON response or a browser inspector. And attaching the header to a client that then follows a redirect to a different host — httpx and requests both strip Authorization on a cross-host redirect, deliberately, so that your credential is not leaked to an unrelated origin. If a request works with follow_redirects=False and fails without it, that is the cause.

Expiry, and Why the 401 Is the Signal

Access tokens are short-lived by design. An hour is typical; fifteen minutes is common on financial and identity services. The refresh token that renews them lives for days or weeks and is the value that actually needs protecting.

Access token windows inside one refresh token window Three consecutive one-hour access token windows separated by refresh points, all sitting inside a single refresh token window that lasts for days. access token — expires_in 3600valid, 1 hourvalid, 1 hourvalid, 1 hourrefreshrefreshrefresh token — valid for days or weeksone long-lived secret, kept in an environment variableRefresh when the server returns 401, not when your clock says so
Access tokens are short and disposable; the refresh token is the long-lived secret. Treat only the second one as a credential worth protecting.

You can compute an expiry from expires_in and refresh proactively, and it is worth doing as an optimisation. It is not sufficient on its own. Server clocks drift, a token can be revoked before its nominal expiry, and a deploy can invalidate an entire issuance. The 401 is the authoritative signal; the timer is a hint that saves a round trip. Build for the 401 and add the timer afterwards.

Refresh-on-401 with a single retry A request returns 401, the client exchanges its refresh token for a new access token, and replays the original request once. A second 401 raises instead of looping. Refresh on 401, retry exactly once1. Send requestAuthorization: Bearer2. Server: 401access token expired3. RefreshPOST /oauth/token4. Retry oncenew header, replaya second 401 raises — it does not loop
The 401 is the signal to refresh. Retrying exactly once turns an expiry into a hiccup; retrying forever turns a revoked token into an infinite loop.

Retry exactly once. If a request fails with 401, refresh, replay it, and if the replay also fails with 401, raise. Looping on 401 produces an infinite loop that hammers the token endpoint with a credential that is not going to start working — the fastest way to have an API key revoked.

A Client That Refreshes on 401

This wraps httpx.Client so that every call goes through one place. The token is fetched lazily on first use, refreshed on a 401, and the original request replayed once.

import os
import threading

import httpx


class TokenAuthError(RuntimeError):
    """Raised when a token cannot be obtained or is refused twice."""


class TokenClient:
    """An httpx client that fetches a bearer token and refreshes it on 401."""

    def __init__(self, token_url: str, client_id: str, client_secret: str,
                 base_url: str = "", timeout: float = 20.0) -> None:
        self._token_url = token_url
        self._client_id = client_id
        self._client_secret = client_secret
        self._token: str | None = None
        self._lock = threading.Lock()
        self._client = httpx.Client(
            base_url=base_url,
            timeout=timeout,
            headers={
                "Accept": "application/json",
                "User-Agent": (
                    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                    "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
                ),
            },
        )

    def _fetch_token(self) -> str:
        response = self._client.post(
            self._token_url,
            data={
                "grant_type": "client_credentials",
                "client_id": self._client_id,
                "client_secret": self._client_secret,
            },
            headers={"Content-Type": "application/x-www-form-urlencoded"},
        )
        if response.status_code >= 400:
            raise TokenAuthError(
                f"token endpoint returned {response.status_code}: {response.text[:200]}"
            )
        payload = response.json()
        token = payload.get("access_token") or payload.get("token")
        if not token:
            raise TokenAuthError(f"no access_token in response: {sorted(payload)}")
        return token

    def _ensure_token(self, force: bool = False) -> str:
        with self._lock:
            if force or self._token is None:
                self._token = self._fetch_token()
            return self._token

    def request(self, method: str, url: str, **kwargs) -> httpx.Response:
        token = self._ensure_token()
        headers = dict(kwargs.pop("headers", {}))
        headers["Authorization"] = f"Bearer {token}"

        response = self._client.request(method, url, headers=headers, **kwargs)
        if response.status_code != 401:
            return response

        refreshed = self._ensure_token(force=True)
        headers["Authorization"] = f"Bearer {refreshed}"
        retried = self._client.request(method, url, headers=headers, **kwargs)
        if retried.status_code == 401:
            raise TokenAuthError(f"still 401 after refresh: {method} {url}")
        return retried

    def get(self, url: str, **kwargs) -> httpx.Response:
        return self.request("GET", url, **kwargs)

    def close(self) -> None:
        self._client.close()

    def __enter__(self) -> "TokenClient":
        return self

    def __exit__(self, *exc_info: object) -> None:
        self.close()


if __name__ == "__main__":
    with TokenClient(
        token_url="https://httpbin.org/post",
        client_id=os.environ.get("API_CLIENT_ID", "demo-id"),
        client_secret=os.environ.get("API_CLIENT_SECRET", "demo-secret"),
    ) as client:
        try:
            page = client.get("https://httpbin.org/bearer")
            print(page.status_code, page.json())
        except TokenAuthError as error:
            print(f"authentication failed: {error}")

Running it as written exercises the failure branch, because httpbin.org/post echoes your request rather than issuing a token and therefore has no access_token key — the TokenAuthError you see is the code working. Point token_url at a real token endpoint and the same class serves the whole run.

A few design decisions are load-bearing. The lock means that under threads only one refresh happens rather than one per worker; without it, a burst of expiries produces a burst of identical token requests, which is exactly the pattern that trips a rate limit on an auth endpoint. kwargs is copied rather than mutated so the replay is byte-identical to the original. And the retry is a single explicit replay, not a loop or a recursive call, so a revoked credential surfaces as an exception in under two round trips.

For a body that is a consumable stream — a file upload — the replay cannot re-read it. Materialise such bodies in memory before the first attempt, or refresh proactively based on expires_in so the replay path is never reached.

Adding the proactive timer on top is a small change: record time.monotonic() + expires_in - 60 when a token is issued, and have _ensure_token refresh when that deadline has passed rather than waiting for the server to reject a request. Use a monotonic clock, not wall time, so a system clock adjustment mid-run cannot make a live token look expired or an expired one look live. The 401 branch stays exactly where it is — the timer only reduces how often that branch is reached, it does not replace it.

If the service uses the refresh-token grant rather than client credentials, only _fetch_token changes: post grant_type=refresh_token with the stored refresh value, then read both access_token and, when present, a rotated refresh_token out of the response and persist the new one before the next request goes out. Everything else in the class is unchanged, which is the reason for putting the token logic behind a single method in the first place.

Edge Cases and Caveats

  • Refresh-token rotation. Many providers issue a new refresh token with every refresh and invalidate the old one. If you persist tokens, write the new refresh value back immediately, or the next process start will fail with a token that was valid a minute ago.
  • expires_in is seconds, not a timestamp. Add it to the time of receipt, and subtract a safety margin of 30–60 seconds so a request is not in flight when the token dies.
  • Scopes. A token issued for a narrow scope returns 403, not 401, on an endpoint outside it. Refreshing will not help; the scope has to be requested at issue time.
  • 401 versus 403. 401 means the credential is missing, malformed or expired and refreshing is the right response. 403 means the credential is understood but not permitted, and retrying is pointless.
  • Rate limits are per token. Concurrency does not multiply your quota when every worker shares one token. Cap parallelism to the documented limit — the technique is in Limiting Concurrency with Semaphores.
  • Tokens in logs. An Authorization header printed once into a log file is a live credential sitting in that file until expiry. Redact it in any request-logging middleware before the first deploy, not after.
  • Async clients need the same care. Replace threading.Lock with asyncio.Lock and httpx.Client with httpx.AsyncClient; the refresh-once-retry-once logic is unchanged. The surrounding patterns are in Asynchronous Scraping with Asyncio and HTTPX.
  • Storage. Client secrets and refresh tokens belong in environment variables or a secret manager, never in source and never in a committed configuration file. A cached access token on disk needs restrictive permissions and a path outside the repository.

Frequently Asked Questions

Should I copy a token out of the browser's localStorage? Only to confirm quickly that an endpoint returns what you expect. It expires, usually within an hour, and nothing in your code can renew it — so any scraper built on a hand-copied token is a scraper that stops overnight. Once the endpoint is proven, automate the token acquisition through the login or token endpoint the front end itself uses.

What is the difference between an access token and a refresh token? The access token is sent with every request and is deliberately short-lived, so a leaked copy stops working quickly. The refresh token is sent only to the token endpoint, lives for days or weeks, and exists purely to mint new access tokens. Protect the refresh token the way you would protect a password.

Why does my request lose the Authorization header on a redirect? Both httpx and requests strip Authorization when a redirect crosses to a different host, so that your credential is not handed to an origin you did not intend to trust. Request the final URL directly, or inspect response.history to find where the hop goes and re-attach the header deliberately for that host.

Can I just refresh on a timer instead of on 401? A timer based on expires_in is a good optimisation but not a sufficient strategy on its own. Tokens get revoked early, server clocks drift, and deploys can invalidate an issuance. Handle the 401 as the authoritative signal and treat the timer as a way to avoid the occasional wasted round trip.

Back to home