Managing Cookies and Sessions

Part of The Complete Guide to Python Web Scraping, this guide covers state: how a scraper stays recognised across hundreds of requests, why a cookie you can see in the jar sometimes is not sent, and how to keep a session alive across restarts. The scope is the client side of state — the login flow that establishes it in the first place belongs to Handling Forms and Authentication.

HTTP has no memory, so every appearance of continuity is something the client re-sends. That has two consequences for scraping. First, a Session object is not a convenience wrapper — it is the component that holds your identity, and using bare requests.get() calls throws that identity away on every request. Second, cookies are not a dictionary: each one carries domain, path, secure and expiry attributes that decide independently whether it goes out, and a mismatch fails silently.

When to Use a Session

The short answer is "almost always", but the reasons differ and they determine how much machinery you need around it.

The last two rows are the ones people get wrong. A single global session shared across five domains and eight threads looks efficient and is the source of both cross-site cookie leakage and intermittent, unreproducible cookie loss.

Prerequisites

Python 3.10 or newer, in the environment from Setting Up Your Python Scraping Environment. Everything here uses requests plus the standard library's http.cookiejar for persistence.

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install "requests>=2.31" "beautifulsoup4>=4.12" "lxml>=5.1"

http.cookiejar, pickle and json are standard library, so no extra install is needed for saving state. Familiarity with status codes and header semantics is assumed — see Understanding HTTP Requests and Responses.

Step-by-Step: Holding State Correctly

1. Create one session and set its defaults once

Session.headers is merged into every request, so the browser header block belongs there rather than on each call. The session also owns the cookie jar, the connection pool, proxy settings and TLS verification, which makes it the single place to configure identity.

import requests
from requests.adapters import HTTPAdapter

BROWSER_HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
    "Accept-Encoding": "gzip, deflate",
    "Connection": "keep-alive",
}


def build_session(pool: int = 16) -> requests.Session:
    """One configured session: headers, connection pool and timeouts in one place."""
    session = requests.Session()
    session.headers.update(BROWSER_HEADERS)
    adapter = HTTPAdapter(pool_connections=pool, pool_maxsize=pool)
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    return session


with build_session() as s:
    for path in ("", "catalogue/page-2.html", "catalogue/page-3.html"):
        r = s.get(f"https://books.toscrape.com/{path}", timeout=(5, 20))
        print(r.status_code, r.url, f"{r.elapsed.total_seconds() * 1000:.0f} ms")

Watch the elapsed values: the first request pays for DNS, TCP and TLS, and the following ones do not. On a crawl of a thousand pages against one host that saved handshake is several minutes of pure latency.

2. Understand why a stored cookie is not sent

session.cookies.get_dict() shows everything in the jar with no regard for scope, which is why people conclude a cookie "is there" while the server keeps saying they are logged out. The jar applies four checks per request, and failing any one withholds the cookie without a warning.

import requests

session = requests.Session()
session.cookies.set("app_session", "abc123", domain="example.com", path="/app")
session.cookies.set("api_key", "k-999", domain="api.example.com", path="/")
session.cookies.set("tracker", "t-1", domain=".example.com", path="/")

for cookie in session.cookies:
    print(f"{cookie.name:>12}  domain={cookie.domain:<16} path={cookie.path:<6} secure={cookie.secure}")


def would_send(session: requests.Session, url: str) -> str:
    """Ask requests itself which cookies the jar attaches to a given URL."""
    prepared = session.prepare_request(requests.Request("GET", url))
    return prepared.headers.get("Cookie", "<none>")


for url in (
    "https://example.com/app/dashboard",
    "https://example.com/public/index.html",
    "https://api.example.com/v2/items",
    "https://shop.example.com/basket",
):
    print(f"{url:<44} -> {would_send(session, url)}")

prepare_request runs the real jar logic without sending anything, which makes it the fastest way to answer "why is this cookie missing". The output shows app_session only on the /app path, api_key only on the API host, and tracker on every example.com subdomain because its domain begins with a dot.

The rules in order: the request host must match the cookie's Domain (a leading dot allows subdomains, no dot means host-only); the request path must be at or below the cookie's Path; a cookie marked Secure is never sent over plain HTTP; and an expired cookie is dropped from the jar entirely. Setting a cookie with domain="example.com" and then requesting www.example.com is the classic near-miss.

3. Prefer letting the server set cookies

Hand-setting cookies is a last resort. When a site issues a consent, region or anti-bot cookie, the value is usually derived from the response you have not made yet, so the correct sequence is to request the entry page first and let Set-Cookie populate the jar naturally.

import requests

with requests.Session() as s:
    s.headers.update({
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/136.0.0.0 Safari/537.36",
        "Accept-Language": "en-GB,en;q=0.9",
    })
    warmup = s.get("https://httpbin.org/cookies/set?consent=granted&region=uk", timeout=(5, 20))
    print("jar after warm-up:", s.cookies.get_dict())

    check = s.get("https://httpbin.org/cookies", timeout=(5, 20))
    print("server sees      :", check.json()["cookies"])

A warm-up request costs one round trip and eliminates a whole class of 403 and redirect-loop failures, because you are now carrying the same state a browser would have after landing on the site.

4. Detect expiry rather than waiting for the crash

Sessions end: an idle timeout, a server-side rotation, an IP change, or a deliberate invalidation. The symptom is rarely an exception. It is a 200 containing the logged-out version of the page, or a 302 to a sign-in URL. Write a single predicate that says whether a response still looks authenticated, and check it after every fetch.

import requests


class SessionExpired(RuntimeError):
    """Raised when a response no longer looks authenticated."""


def assert_authenticated(response: requests.Response) -> requests.Response:
    """Treat log-out redirects and login markup as errors, not as data."""
    if response.status_code in (401, 403):
        raise SessionExpired(f"{response.status_code} on {response.url}")
    if "/login" in response.url or "/signin" in response.url:
        raise SessionExpired(f"redirected to a login page: {response.url}")
    body = response.text[:4000].lower()
    if 'name="password"' in body or "sign in to continue" in body:
        raise SessionExpired(f"login form present at {response.url}")
    return response

Pair that with a re-authenticate-and-retry wrapper so a single expiry costs one extra request rather than the whole run. The authentication step itself, including CSRF token handling, is covered in Handling CSRF Tokens When Scraping and Logging In with Python Requests.

5. Save the jar between runs

Re-authenticating on every execution is slow and, on sites that count logins, conspicuous. http.cookiejar.MozillaCookieJar serialises to the standard cookies.txt format, which requests accepts directly and which is readable by curl and browser extensions.

from http.cookiejar import MozillaCookieJar
from pathlib import Path

import requests

JAR_PATH = Path("data/cookies.txt")


def session_with_jar() -> requests.Session:
    """Load a saved jar if one exists; new cookies are appended to it."""
    jar = MozillaCookieJar(JAR_PATH)
    if JAR_PATH.exists():
        jar.load(ignore_discard=True, ignore_expires=True)
    session = requests.Session()
    session.cookies = jar
    session.headers.update({"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) Chrome/136.0.0.0 Safari/537.36"})
    return session


s = session_with_jar()
s.get("https://httpbin.org/cookies/set?visit=1", timeout=(5, 20))
JAR_PATH.parent.mkdir(parents=True, exist_ok=True)
s.cookies.save(ignore_discard=True, ignore_expires=True)
print("saved", len(s.cookies), "cookie(s)")

ignore_discard=True is required to persist session cookies, which by definition are meant to vanish when the browser closes — and which are exactly the ones carrying your login. Treat the file as a credential: restrict its permissions and never commit it. The full pattern, including expiry-aware reloading, is in Persisting a Session Between Runs.

6. Isolate identities that must not mix

A cookie jar is a single namespace. If you scrape two accounts, two regions, or two unrelated sites, use separate sessions — otherwise a cookie set by one context is sent in the other, which at best pollutes your data and at worst logs you in as the wrong user.

import requests


class SessionPool:
    """One configured session per named identity, created on first use."""

    def __init__(self, user_agent: str) -> None:
        self._ua = user_agent
        self._sessions: dict[str, requests.Session] = {}

    def get(self, identity: str) -> requests.Session:
        if identity not in self._sessions:
            session = requests.Session()
            session.headers.update({"User-Agent": self._ua, "Accept-Language": "en-GB,en;q=0.9"})
            self._sessions[identity] = session
        return self._sessions[identity]

    def close(self) -> None:
        for session in self._sessions.values():
            session.close()
        self._sessions.clear()


pool = SessionPool("Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/136.0.0.0 Safari/537.36")
print(pool.get("account-a") is pool.get("account-b"))   # False
pool.close()

When each identity also needs its own exit IP, bind a proxy to the session as well — the pairing rules are in Rotating Proxies and Managing IP Blocks.

7. Send per-request cookies without touching the jar

Passing cookies= to a single call merges that value in for that request only and leaves the jar unchanged. This is the right way to test a hypothesis about which cookie matters, because it avoids permanently contaminating the session you are debugging.

import requests

with requests.Session() as s:
    s.headers.update({"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) Chrome/136.0.0.0 Safari/537.36"})
    one_off = s.get("https://httpbin.org/cookies", cookies={"experiment": "b"}, timeout=(5, 20))
    print("sent this time :", one_off.json()["cookies"])
    print("jar unchanged  :", s.cookies.get_dict())

8. Re-authenticate once, then retry the original request

Expiry detection is only half of the answer; the other half is recovering from it without losing the page you were fetching. Wrap the fetch so a single SessionExpired triggers one login and one replay, and a second failure in a row aborts rather than looping.

from collections.abc import Callable

import requests


def fetch_authenticated(
    session: requests.Session,
    url: str,
    login: Callable[[requests.Session], None],
) -> requests.Response:
    """Fetch a protected URL, refreshing the session at most once."""
    for attempt in (1, 2):
        response = session.get(url, timeout=(5, 20))
        try:
            return assert_authenticated(response)
        except SessionExpired:
            if attempt == 2:
                raise
            session.cookies.clear()
            login(session)
    raise SessionExpired(f"unreachable: {url}")

Clearing the jar before re-logging in matters. A stale session cookie left in place is frequently sent alongside the new one, and servers that see two conflicting identifiers usually keep the older, invalid one — producing a login that appears to succeed and then immediately fails again.

Performance and Scaling Considerations

Connection reuse is the largest single win. A cold HTTPS request spends 80–250 ms on DNS, TCP and TLS before any application data moves. A pooled session pays that once per host. Across 1,000 sequential requests to one domain that is roughly four minutes of latency removed for a one-line change.

Size the pool to your concurrency. The default adapter allows ten connections per host. Running twenty threads against one domain produces Connection pool is full, discarding connection warnings and, worse, silently closed sockets that then have to be re-established — turning your concurrency gain back into handshake cost. Mount an adapter with pool_maxsize at least equal to your worker count.

A session is not thread-safe. The cookie jar and the connection pool are mutated on every request, and concurrent mutation produces lost cookies and occasional RuntimeError from dictionary resizing. Give each thread its own session, or use threading.local() so one is created lazily per thread.

import threading
import requests

_local = threading.local()


def session_for_thread() -> requests.Session:
    """One session per thread — never share a jar across workers."""
    if not hasattr(_local, "session"):
        s = requests.Session()
        s.headers.update({"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) Chrome/136.0.0.0 Safari/537.36"})
        _local.session = s
    return _local.session

Jars grow. Sites set analytics and experiment cookies on almost every response, so a long crawl can accumulate hundreds of them, and every one is serialised into the Cookie header on every subsequent matching request. A 4 KB cookie header on 10,000 requests is 40 MB of pointless upload, and some servers reject headers over 8 KB with a 400. Clear the ones you do not need periodically with session.cookies.clear(domain, path, name).

Async clients need the same discipline. requests blocks the event loop, so async crawls use httpx.AsyncClient or aiohttp.ClientSession, both of which carry an equivalent cookie jar and connection pool. Create one per identity and reuse it for the client's whole lifetime rather than per request — see Asynchronous Scraping with Asyncio and HTTPX.

Close what you open. A Session holds sockets until it is closed or garbage collected. In a long-running process that creates sessions dynamically, use it as a context manager or call .close() explicitly, or you will eventually hit the process file-descriptor limit.

Common Errors and Fixes

The login succeeds but the next request is logged out. Almost always two separate requests.get() calls instead of one session, so the auth cookie was discarded. Less commonly, the login response set the cookie for example.com and you then requested www.example.com. Use one session, and print response.cookies immediately after the login POST to see the domain that was actually set.

import requests

with requests.Session() as s:
    s.headers.update({"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/136.0.0.0 Safari/537.36"})
    login = s.post("https://httpbin.org/cookies/set?auth=token123", timeout=(5, 20))
    print("set by server:", [(c.name, c.domain, c.path) for c in login.cookies])
    print("jar now      :", s.cookies.get_dict())

requests.exceptions.TooManyRedirects: Exceeded 30 redirects. The server is bouncing you between a page and a consent or login endpoint because the cookie it expects never arrived. Do a warm-up request to the site root with allow_redirects=False and read the Set-Cookie and Location headers to see what it wants.

urllib3.exceptions.MaxRetryError after a burst of concurrency. The connection pool was exhausted and re-establishing connections started to time out. Raise pool_maxsize to at least the number of workers and lower the concurrency until the error stops.

from requests.adapters import HTTPAdapter
import requests

session = requests.Session()
session.mount("https://", HTTPAdapter(pool_connections=32, pool_maxsize=32, max_retries=0))

Cookies vanish between runs even though the file was saved.ignore_discard defaulted to False, so session cookies — the ones that matter — were skipped on save, on load, or both. Pass ignore_discard=True and ignore_expires=True to both save() and load().

http.cookiejar.LoadError: 'cookies.txt' does not look like a Netscape format cookies file. The file was written by pickle or by a browser extension using a different dialect. Delete it and regenerate with MozillaCookieJar.save(), or convert explicitly rather than hand-editing the format.

A cookie appears in get_dict() but the server still rejects the request.get_dict() ignores scope entirely. Iterate the jar and inspect cookie.domain, cookie.path, cookie.secure and cookie.expires against the URL you are requesting; one of the four is mismatched. This is also why two cookies with the same name but different domains silently collapse into one entry in the dict view.

Occasional missing cookies under threads with no traceback. Two threads mutated one jar concurrently. Move to one session per thread; there is no locking configuration that makes a shared jar correct.

Frequently Asked Questions

What is the difference between a cookie and a session? A cookie is a small key-value pair with scoping attributes that the client stores and re-sends. A session is server-side state that a cookie merely points at, plus — in requests — the client object that holds the jar and the connection pool. Managing "sessions" in a scraper means managing the cookies that identify you, since the server-side half is not yours to touch.

Do I need a session if I am not logging in? Usually yes, for performance rather than authentication. Connection pooling alone justifies it, and many sites set a consent, region or anti-bot cookie on first contact that later requests are expected to carry. A bare requests.get() fails both of those.

How do I reuse cookies exported from my browser? Export them as a Netscape-format cookies.txt file and load it with MozillaCookieJar, then assign the jar to session.cookies. Send the same User-Agent the browser used, because some sites bind the session to a fingerprint and will invalidate it when a different client presents the cookie.

Can I share one session across threads or async tasks? No. The cookie jar and connection pool are mutated per request and neither is safe under concurrency. Create one session per thread with threading.local(), or one httpx.AsyncClient per logical identity for async code.

How do I know when the session has expired? Do not rely on an exception, because expiry usually returns a 200. Check three things after each fetch: the status is not 401 or 403, the final response.url is not a login path, and the body does not contain a password field. Any one of those failing should trigger re-authentication and a single retry.

Related

• The Complete Guide to Python Web Scraping — the parent guide this page belongs to.
• Persisting a Session Between Runs — saving, reloading and validating a stored jar.
• Handling Forms and Authentication — establishing the session in the first place.
• Understanding HTTP Requests and Responses — the headers and status codes this state rides on.
• Handling Pagination and Infinite Scroll — where one long-lived session pays for itself.