Persisting a Session Between Runs

A scraper that logs in on every start wastes a round-trip, hands the target a repeating authentication pattern, and risks tripping login rate limits; this page extends Managing Cookies and Sessions to cover keeping that state across process boundaries.

The approach in short: serialise the cookie jar, not the Session. Convert the jar to a plain dictionary or list of records, write it as JSON with file mode 0600, and rebuild a fresh Session around it on the next run. Do not pickle the Session object — it captures a live object graph including adapters and connection pools, and the file becomes unreadable the moment a library version changes underneath it. Record each cookie's expiry so a stale jar is discarded rather than silently sending dead credentials, and treat the file exactly as you would a password, because it authenticates as you until it expires.

Why Pickling a Session Object Is Fragile

pickle.dump(session, f) looks like the obvious answer and is a maintenance liability for three separate reasons.

Pickle stores a reference to the class by module path and reconstructs the instance by importing it and restoring its __dict__. That makes the file a snapshot of one library's internal layout at one version. When requests or urllib3 renames an attribute, changes a default, or moves a class between modules, the restored object is either wrong or fails to unpickle at all — and the failure surfaces as an obscure AttributeError deep inside the HTTP stack rather than as a clear "your saved file is stale".

A Session also owns things that should never be revived from disk. It holds HTTPAdapter instances wrapping urllib3 pool managers, which reference sockets, TLS contexts and thread state. Those objects are meaningful only in the process that created them. Restoring a pickled session gives you an object whose configuration is not the configuration the current process would have chosen — the retry policy, pool sizes and TLS settings all come from whenever the file was written.

The third reason is decisive on its own: unpickling executes code. pickle.load on a file an attacker can modify is arbitrary code execution in your process, with no signature check and no sandbox. A cookie file living in a shared cache directory, a CI artefact store, or a mounted volume is exactly the kind of file that can be modified.

None of that buys anything. The only durable state in a session is the cookie jar plus the headers you configured, and headers belong in code, not in a serialised blob.

What to Serialise Instead

requests ships helpers for the simple case. requests.utils.dict_from_cookiejar(session.cookies) returns a plain {name: value} dictionary, and requests.utils.cookiejar_from_dict(data) builds a jar back from one. Two lines each way, and the resulting JSON is readable and diffable.

The catch is that a flat name-to-value mapping discards everything except the name and the value. Gone are domain, path, expires, secure and httponly. For a single-host scraper that is usually acceptable, because you will send those cookies to exactly one origin anyway. It stops being acceptable in three situations: when the site splits state across subdomains, since the restored cookies default to the host you attach them to and a cookie meant for api.example.com sent to www.example.com is either ignored or leaked; when different cookies have different paths, since collapsing them means sending an admin-scoped cookie to public endpoints; and when you want to check expiry, since without expires you cannot tell a fresh jar from a dead one before making a request.

The durable format is therefore a list of records, one per cookie, keeping the fields that determine scope and lifetime. Iterating a RequestsCookieJar yields Cookie objects with .name, .value, .domain, .path, .expires, .secure and a _rest dictionary that carries HttpOnly. Writing those to JSON and rebuilding with session.cookies.set(...) round-trips everything that matters, stays human-readable, and cannot execute anything on load.

For browser-driven work the equivalent is Playwright's context.storage_state(path="state.json"), which writes a JSON document containing both cookies and per-origin localStorage. That second part matters more than it sounds: many single-page applications keep the access token in localStorage and only a session identifier in a cookie, so a cookies-only export restores half the state and the app behaves as though logged out. Reload with browser.new_context(storage_state="state.json"). The file is plain JSON in both directions, which is one more reason not to reach for pickle.

Handling Expiry

A restored jar is a claim about the past, and the first thing to do with it is check whether the claim still holds.

Cookie expires values are Unix timestamps in seconds, or None for a session cookie that a real browser would discard when the window closed. Session cookies are the majority of authentication cookies, and they are exactly the ones you are trying to persist — so "no expiry recorded" does not mean "valid forever", it means "the server made no promise". Handle that by recording your own saved_at timestamp alongside the jar and applying a local maximum age, typically a few hours to a day depending on how aggressively the site rotates.

For cookies that do carry an expiry, filter on load: drop anything already past, and if the cookie your authentication actually depends on is among them, skip the restore entirely and log in fresh. Restoring a jar that is missing its key cookie is worse than restoring nothing, because the request then arrives with partial state and many applications respond by redirecting to a login page that your parser cheerfully treats as the target document.

Server-side invalidation is invisible from the jar. A cookie can be well within its expiry and already revoked because the user logged out elsewhere, the server rotated its signing key, or the address changed. So the jar is an optimisation, never a guarantee — always keep a re-authentication path and trigger it on the first response that indicates a logged-out state. Note that this is not always a 401; a great many sites return 200 with a login form, which is why a cheap sentinel check on the response body is worth more than a status-code check.

A Working Save-and-Restore Cycle

from __future__ import annotations

import json
import os
import time
from pathlib import Path
from typing import Any

import requests

JAR_PATH = Path.home() / ".cache" / "scraper" / "cookies.json"
MAX_AGE_SECONDS = 6 * 60 * 60
HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
}


def save_jar(session: requests.Session, path: Path = JAR_PATH) -> None:
    """Write the cookie jar as JSON with owner-only permissions."""
    payload: dict[str, Any] = {
        "saved_at": int(time.time()),
        "cookies": [
            {
                "name": c.name,
                "value": c.value,
                "domain": c.domain,
                "path": c.path,
                "expires": c.expires,
                "secure": bool(c.secure),
            }
            for c in session.cookies
        ],
    }
    path.parent.mkdir(parents=True, exist_ok=True)
    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
    with os.fdopen(fd, "w", encoding="utf-8") as handle:
        json.dump(payload, handle, indent=2)


def load_jar(session: requests.Session, path: Path = JAR_PATH) -> bool:
    """Restore unexpired cookies. Returns False if the jar is missing or stale."""
    if not path.exists():
        return False
    try:
        payload = json.loads(path.read_text(encoding="utf-8"))
    except (json.JSONDecodeError, UnicodeDecodeError):
        path.unlink(missing_ok=True)
        return False

    now = int(time.time())
    if now - int(payload.get("saved_at", 0)) > MAX_AGE_SECONDS:
        return False

    restored = 0
    for cookie in payload.get("cookies", []):
        expires = cookie.get("expires")
        if expires is not None and expires < now:
            continue
        session.cookies.set(
            cookie["name"],
            cookie["value"],
            domain=cookie.get("domain", ""),
            path=cookie.get("path", "/"),
            secure=cookie.get("secure", False),
        )
        restored += 1
    return restored > 0


def is_authenticated(session: requests.Session, probe_url: str) -> bool:
    """Cheap check that the restored jar is still accepted by the server."""
    response = session.get(probe_url, timeout=15, allow_redirects=False)
    if response.status_code in (301, 302, 303, 307, 308):
        return "login" not in response.headers.get("Location", "").lower()
    return response.status_code == 200 and "sign in" not in response.text.lower()


def build_session(login_url: str, probe_url: str) -> requests.Session:
    """Reuse a saved jar when it still works, otherwise authenticate once."""
    session = requests.Session()
    session.headers.update(HEADERS)

    if load_jar(session) and is_authenticated(session, probe_url):
        return session

    session.cookies.clear()
    session.post(
        login_url,
        data={
            "username": os.environ["SCRAPER_USER"],
            "password": os.environ["SCRAPER_PASSWORD"],
        },
        timeout=20,
    ).raise_for_status()
    save_jar(session)
    return session


if __name__ == "__main__":
    probe = "https://httpbin.org/cookies"
    client = requests.Session()
    client.headers.update(HEADERS)
    client.get("https://httpbin.org/cookies/set/demo_session/abc123", timeout=15)
    save_jar(client)

    revived = requests.Session()
    revived.headers.update(HEADERS)
    print("restored:", load_jar(revived))
    print(revived.get(probe, timeout=15).json())

The __main__ block runs against httpbin.org so you can watch a cookie survive a process boundary without needing credentials anywhere. build_session is the shape you would use with a real login: try the jar, verify it with one cheap probe, and fall back to authenticating only when the probe fails.

Note os.open with mode 0o600 rather than a plain open. Creating the file with the right permissions from the start avoids a window in which the jar exists world-readable, and it means the mode is correct even when the process umask is permissive — which it usually is inside containers.

The Saved Jar Is a Credential

A session cookie is a bearer token. Anyone holding the file can replay it and the server will treat them as the authenticated account until the cookie expires or is revoked. It is not "less sensitive" than the password because it is temporary; for the window in which it is valid it is strictly more useful to an attacker, since it bypasses any second factor that guarded the original login.

Four routine mistakes leak it. Committing it to version control is the most common, and git keeps it in history even after you delete the file — add the path to .gitignore before the first run, not after. Leaving default permissions makes it readable by every account on the host; 0600 and a directory the process owns is the minimum. Copying it into a container image bakes it into a layer that gets cached, pushed to a registry and shared. And printing the session or a response object during debugging writes cookie values into logs that are usually retained far longer and read far more widely than the file itself.

The safer defaults are easy to adopt. Keep the jar outside the repository entirely, under ~/.cache/ or an explicit XDG_CACHE_HOME path. Mount it at runtime rather than building it in. In CI, prefer the platform's secret storage and write the jar to a path that is cleaned at the end of the job. Delete it on any authentication failure so a revoked token is not retried indefinitely. And keep one jar per account, never a shared file that concurrent workers write to, because interleaved writes produce a truncated JSON document and a jar that silently restores nothing.

Edge Cases and Caveats

• Concurrent writers corrupt the file. Two processes calling save_jar at once can interleave. Write to a temporary file in the same directory and os.replace it, which is atomic on POSIX and Windows.
• A domain leading dot changes scope. .example.com matches subdomains, example.com does not. Round-tripping through dict_from_cookiejar loses the distinction and can silently narrow or widen where a cookie is sent.
• __Host- and __Secure- prefixes carry rules. Restoring a __Host- cookie with a domain attribute or over plain HTTP means the browser-equivalent constraints no longer hold and the server may reject it.
• The jar is only half the identity. A session restored from a different address or with a different User-Agent than it was issued to is frequently invalidated on purpose; keep the exit and the header set stable across runs, as covered in how to scrape a static website without getting blocked.
• CSRF tokens are not cookies. Many frameworks pair a cookie with a per-form token embedded in the HTML. Restoring the jar alone leaves POST requests failing until you fetch a fresh token from the page.
• Playwright storage_state includes origins. The file records localStorage per origin, so a state file captured on a staging host will not apply on production — the origins simply do not match.

Frequently Asked Questions

Can I just pickle the requests Session object? It works until a dependency updates, then breaks with a confusing error deep in the HTTP stack — and unpickling executes code, so a jar file an attacker can modify becomes code execution in your process. Serialise the cookie jar to JSON and build a fresh Session around it instead.

How do I know whether a restored jar is still valid? Filter out cookies whose expires timestamp has passed, apply your own maximum age using a saved_at field you write alongside them, and then make one cheap probe request. Server-side revocation is invisible from the file, so always keep a re-authentication fallback.

Does saving cookies work for Playwright too? Use context.storage_state(path=...), which writes cookies and per-origin localStorage as JSON, then reload with browser.new_context(storage_state=...). The localStorage part matters because many applications keep the access token there rather than in a cookie.

Is a saved cookie file really as sensitive as a password? For as long as it is valid, it is more useful to an attacker, because replaying it skips the login and any second factor that protected it. Keep it out of version control and container images, create it with mode 0600, and never let a session or response object reach a log.

Related

• Managing Cookies and Sessions — the parent guide on session objects and cookie scoping
• Understanding HTTP Requests and Responses — how Set-Cookie and redirects shape the state you are saving
• Logging In with Python Requests — the authentication step the saved jar lets you skip
• Scheduling Scrapers with Cron and GitHub Actions — where a persisted jar has to live in an automated run