Python Web Scraping
Reading layout

HTTP Caching with requests-cache

requests-cache turns a normal requests session into one that transparently stores and replays responses, and this page — part of Caching and Incremental Crawling — covers the parts that decide whether it helps or quietly does nothing.

Path of a request through a cached session A get call is turned into a cache key, which either hits a fresh stored entry and returns with no network traffic, or misses and goes to the network before being written back. Both paths return the same response object. session.get(url)Cache key builtmethod, URL, body, headersFresh entryno socket is openedMiss or expiredfetch, then write backResponse objectfrom_cache tells you whichhitmiss
A cached session intercepts the call before the transport layer: a fresh entry returns without a socket being opened, and only a miss or an expired entry costs a request.

Swap requests.Session() for requests_cache.CachedSession() and every GET you have already made returns from local storage without opening a socket. The two settings that matter are the backend, which decides where entries live and who can read them, and expire_after, which decides how long an entry counts as fresh. The failure mode to watch for is a poisoned cache key: if a value that changes on every request is folded into the key, the cache fills up and never returns a hit.

How the Cache Key Is Built

An entry is addressed by a digest of the request, not by the URL alone. By default that digest covers the HTTP method, the URL after normalisation, and the request body — which is what lets a POST with a JSON payload be cached correctly while a different payload to the same endpoint stays separate.

Anatomy of an HTTP cache key The key is built from the method, the normalized URL, the request body and any headers you opt into. Three examples show a constant header being safe, a rotating User-Agent destroying the hit rate, and an Authorization header splitting per account. What the key is built fromGETmethodNormalized URLhost lowered, query sortedRequest bodyPOST payloadsMatched headersonly if you opt inEffect of matching on a headerAccept: text/htmlconstant across the whole crawlone key per URLUser-Agent, rotateda new value on every requesthit rate falls to zeroAuthorizationsplits data per accountmatch it on purpose
Every value folded into the key multiplies the number of distinct entries. A rotating header turns a cache into a write-only log.

URL normalisation is not a trivial string comparison. requests-cache lower-cases the scheme and host, drops a default port, removes the fragment, and sorts query parameters, so https://Example.com/api?b=2&a=1 and https://example.com/api?a=1&b=2 share one entry. It does not know which of your query parameters are meaningless, so tracking parameters such as utm_source still split the key — strip those yourself before the request if the page ignores them.

Headers are excluded from the key unless you ask for them with match_headers. That default is deliberate and it is the single most important thing to understand about the library. A scraper that rotates its User-Agent per request, as recommended for evasion in How to Rotate User Agents in Python, would get a hit rate of exactly zero if that header were part of the key. Turn header matching on only when the response genuinely varies by the header:

import requests_cache

session = requests_cache.CachedSession(
    "api_cache",
    backend="sqlite",
    match_headers=["Accept", "Accept-Language"],
)

Matching on Authorization is the legitimate case: it keeps one account's data out of another's entry. Matching on User-Agent, Cookie or a per-request proxy header is almost always a mistake.

Choosing a Backend

The backend is the second constructor argument and changes nothing about the API.

Response cache backends compared Four backends scored on whether the cache survives a restart, whether several processes can share it, and what it costs to set up. Memory loses everything on restart, SQLite and the filesystem persist locally, and Redis needs a server. BackendSurvives restartShared by workersCost to runMemorydict in the processlost on exitnononeSQLitethe default backendyesonly with WALone fileFilesystemone file per entryyesyesinode churnRedisshared across hostswith AOF onyesruns a server
Pick the backend by the two questions that actually bite: does the cache have to outlive the process, and will several workers write to it at once.

SQLite is the default and the right answer for a single-machine crawl: one file, transactional, survives a restart, and readable from sqlite3 on the command line when you want to see what is in there. Set fast_save=True if you can tolerate losing the last few writes on a hard kill — it turns off the per-write fsync and roughly triples write throughput.

The memory backend costs nothing and disappears with the process, which makes it right for a short script or a test suite. filesystem writes one file per entry and is the easiest thing to inspect and to hand to a colleague, but a million small files is hostile to most filesystems. redis is the one to use when several workers or several machines share a crawl, which is the setup described in Distributed Crawling with Celery and Redis; note that Redis keeps everything in RAM, so a cache of full HTML bodies gets expensive fast unless you also set a short expire_after.

Expiry: Global, Per-URL, and Per-Request

expire_after accepts seconds, a timedelta, -1 for never, or 0 for "do not cache". urls_expire_after maps URL glob patterns to their own values and is evaluated in order, which is how you implement a page-class TTL policy in one place.

from datetime import timedelta

import requests_cache

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}

session = requests_cache.CachedSession(
    "books_cache",
    backend="sqlite",
    expire_after=timedelta(hours=12),
    urls_expire_after={
        "books.toscrape.com/catalogue/page-*": timedelta(minutes=15),
        "books.toscrape.com/catalogue/*/index.html": timedelta(days=7),
        "*": timedelta(hours=12),
    },
    allowable_codes=(200,),
    stale_if_error=True,
)

def fetch(url: str) -> str:
    response = session.get(url, headers=HEADERS, timeout=15)
    response.raise_for_status()
    origin = "cache" if response.from_cache else "network"
    print(f"{origin:>7}  {response.status_code}  {url}")
    return response.text

if __name__ == "__main__":
    listing = "https://books.toscrape.com/catalogue/page-1.html"
    fetch(listing)
    fetch(listing)
    print("entries:", len(session.cache.responses))

Run it and the second call prints cache. Three of those arguments deserve a note. allowable_codes=(200,) stops a transient 503 being stored and replayed for twelve hours — the default already excludes error codes, but being explicit prevents a later edit from widening it by accident. stale_if_error=True does the opposite where it is useful: if the network fails, an expired entry is returned rather than raising, which keeps a long crawl alive through a brief outage. And from_cache is the attribute to log; a hit rate you have never measured is a hit rate you do not have.

To respect the server's own Cache-Control and Expires headers instead of your own numbers, pass cache_control=True. The library will then honour max-age, treat no-store as "do not cache this", and use ETag/Last-Modified to revalidate expired entries with a conditional request rather than a full download. This is the closest requests-cache gets to being a correct HTTP cache, and it is worth turning on for APIs, which usually send sensible headers. For scraped HTML it is often worth leaving off, because a great many sites send Cache-Control: no-cache on every page as a blanket CDN policy and you would end up caching nothing.

Filtering What Gets Stored

Not every response deserves a slot. A filter_fn receives the response and returns True to store it, which lets you reject soft errors that arrive with a 200 — an anti-bot interstitial, a login wall, a truncated page.

import requests
import requests_cache

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
}
BLOCK_MARKERS = ("Just a moment", "Access Denied", "captcha-delivery")

def worth_caching(response: requests.Response) -> bool:
    if "text/html" not in response.headers.get("Content-Type", ""):
        return False
    if len(response.content) < 2048:
        return False
    body = response.text
    return not any(marker in body for marker in BLOCK_MARKERS)

session = requests_cache.CachedSession(
    "filtered_cache", backend="sqlite", filter_fn=worth_caching
)

if __name__ == "__main__":
    response = session.get(
        "https://books.toscrape.com/catalogue/page-2.html",
        headers=HEADERS,
        timeout=15,
    )
    print("stored:", not response.from_cache and worth_caching(response))

Caching a challenge page is worse than not caching at all: the entry looks like a success, your parser extracts nothing, and the TTL guarantees you keep extracting nothing until it expires. The block-page markers above are the same signatures discussed in Bypassing Cloudflare and Akamai Protections.

The Same Thing for httpx: hishel

requests-cache only wraps requests. For asyncio-based crawlers built on HTTPX, the equivalent is hishel, which plugs in as a transport rather than as a session subclass and implements RFC 9111 semantics properly.

import asyncio

import hishel
import httpx

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
}

storage = hishel.AsyncSQLiteStorage(ttl=3600)
controller = hishel.Controller(
    cacheable_methods=["GET"],
    cacheable_status_codes=[200],
    allow_stale=True,
    always_revalidate=False,
)

async def main() -> None:
    transport = hishel.AsyncCacheTransport(
        transport=httpx.AsyncHTTPTransport(),
        storage=storage,
        controller=controller,
    )
    async with httpx.AsyncClient(transport=transport, headers=HEADERS) as client:
        for _ in range(2):
            response = await client.get(
                "https://books.toscrape.com/catalogue/page-1.html", timeout=15
            )
            print(response.status_code, response.extensions.get("from_cache"))

if __name__ == "__main__":
    asyncio.run(main())

The differences worth knowing: hishel reports a hit through response.extensions["from_cache"] rather than an attribute, its TTL lives on the storage object instead of the session, and by default it follows the server's caching directives rather than a blanket expiry — so a site that sends no-store will not be cached at all until you relax the Controller. Setting force_cache=True on the controller overrides the server and caches regardless, which is what you usually want for a development fixture cache.

Inspecting and Maintaining the Cache

The cache is an object you can query, which matters more than it sounds: most of the time something goes wrong with a cached crawl, the fastest diagnosis is to look at what is actually stored.

from datetime import datetime, timezone

import requests_cache

session = requests_cache.CachedSession("books_cache", backend="sqlite")


def summarise(session: requests_cache.CachedSession) -> None:
    cache = session.cache
    urls = list(cache.urls())
    print(f"{len(urls)} entries")
    for url in urls[:5]:
        response = cache.get_response(cache.create_key(method="GET", url=url))
        if response is None:
            continue
        created = response.created_at.replace(tzinfo=timezone.utc)
        age = datetime.now(timezone.utc) - created
        print(f"  {age.total_seconds():8.0f}s old  expired={response.is_expired}  {url}")


def prune(session: requests_cache.CachedSession) -> None:
    session.cache.delete(expired=True)
    if session.cache.__class__.__name__ == "SQLiteCache":
        session.cache.responses.vacuum()


if __name__ == "__main__":
    summarise(session)
    prune(session)
    print("after prune:", len(list(session.cache.urls())))

cache.urls() lists what is stored, create_key() rebuilds the digest for a request you describe, and every cached response carries created_at, expires and is_expired. If create_key() for a URL you know you fetched does not find an entry, the key is being built from something you did not expect — usually a header you added to match_headers, or a query parameter you strip in one code path and not another.

Pruning is a separate operation from expiring. delete(expired=True) removes entries whose TTL has passed; without it a long-lived SQLite cache grows monotonically, and because SQLite never returns pages to the filesystem on its own, the file stays large until you VACUUM. On a crawl that caches full HTML, budget roughly the compressed size of every page you have ever fetched and schedule the prune weekly.

One more habit worth building: keep the cache name in a constant that encodes the parser version. When the extraction logic changes shape, bump it. The alternative — remembering to delete a file by hand after a refactor — fails exactly once, and the symptom is a dataset silently built from the previous release's code.

Edge Cases and Caveats

  • Redirects consume entries. A 301 chain is cached per hop, so a site that redirects every URL doubles your entry count. Set allowable_codes=(200,) and let requests follow redirects normally so only the final response is stored.
  • Expired is not deleted. Entries stay on disk after they expire. Call session.cache.delete(expired=True) on a schedule, and run VACUUM on a SQLite backend afterwards — the file does not shrink on its own.
  • Streaming responses. stream=True responses are read into memory to be cached, which defeats the point of streaming. Do not cache large binary downloads through this layer.
  • Cookies and sessions. A cached response replays its Set-Cookie headers, which can resurrect a dead session identifier. Exclude authenticated endpoints, or match on the cookie header so each session gets its own entry.
  • Parser changes invalidate nothing. The cache stores bytes, not your extraction. After changing a parser, version the cache name — books_cache_v2 — so a bug fix does not replay records built by the broken code.
  • Thread and process safety. The SQLite backend is safe across threads but needs WAL mode for concurrent processes; the memory backend is per-process by definition and gives a near-zero hit rate under multiprocessing.
  • Time zones and clock skew. Expiry is computed against the local clock. A container whose clock drifts, or one that runs in a different zone from the machine that wrote the cache, can treat fresh entries as expired.

Frequently Asked Questions

How do I tell whether a response actually came from the cache? Check response.from_cache, which requests-cache adds to every response object. Log it per request and track the ratio; a hit rate near zero almost always means a rotating value has been folded into the cache key.

Can I force a single request to bypass the cache? Yes. Wrap the call in with session.cache_disabled(): to skip the cache entirely for that block, or pass expire_after=0 on the individual request to fetch fresh and refuse to store the result.

Does requests-cache respect Cache-Control headers from the server? Only if you construct the session with cache_control=True. Without it the library ignores server directives and uses your expire_after values exclusively, which is usually what you want when scraping HTML from sites that mark everything as uncacheable.

Should I use requests-cache or hishel? Use requests-cache if your crawler is built on requests, and hishel if it is built on httpx — especially async httpx, which requests-cache cannot wrap at all. hishel follows the HTTP caching specification more closely, so expect it to cache less by default until you configure the controller.

Back to home