Caching and Incremental Crawling

The cheapest request is the one you do not send, and this guide — part of Scaling & Deploying Python Web Scrapers — is about arranging a crawler so that most of the URLs it considers never reach the network at all. The scope is the suppression machinery: a response cache during development, conditional requests in production, a definition of "changed" that matches your dataset, and the state you have to persist between runs to make any of it work.

Three separate mechanisms do the suppressing, and they are usually confused with one another. Frontier state answers "have I already queued this URL, and is it due yet?" without touching the network. An HTTP cache answers "do I already hold a usable copy of this response?" and may still send a cheap validation request. A content hash answers "did the bytes I just downloaded differ from the bytes I stored last time?" — it is the only one that costs a full download, and its payoff is skipping the parse, the validation and the database write rather than the transfer.

When to Use Caching and When to Use Incremental Crawling

The two are not alternatives. Caching is about not paying twice for the same response inside a short window; incremental crawling is about a long-running dataset where each scheduled run should only touch what moved.

A useful test for whether incremental crawling is worth the engineering: measure the fraction of pages whose extracted record was identical to the previous run. Below about 30 per cent unchanged, the bookkeeping rarely pays for itself. Above 80 per cent — which is normal for product catalogues, documentation sites, legislative archives and job boards older than a week — the saving dominates everything else you could optimise.

The cost side is worth stating plainly, because "add caching" is often treated as free. Every layer adds state that can go stale, get corrupted, or disagree with reality, and each one adds a class of bug that produces missing data rather than an exception. A crawler that fetches too much is visible in a bandwidth graph; a crawler that fetches too little looks identical to a healthy one until someone compares the dataset against the live site. Budget for the reconciliation runs that catch that, and treat them as part of the feature rather than as an optional extra.

Prerequisites

Python 3.10 or newer. The examples use the standard library plus three small packages:

python -m pip install "requests>=2.32" "requests-cache>=1.2" "httpx>=0.27" "hishel>=0.0.30" "pybloom-live>=4.0"

sqlite3 ships with CPython, so the frontier and validator stores below need no extra dependency. If you intend to share state between machines you will also want a Redis server reachable from every worker; the setup for that is covered in Distributed Crawling with Celery and Redis.

Step-by-Step: Building a Crawl That Skips Its Own Work

1. Cache Everything While You Develop

Parser work is an edit-run-inspect loop, and each iteration should cost zero requests. Put a response cache in front of the client on the first day of a project, before you write a single selector. With requests-cache this is two lines and changes nothing else about your code.

import requests_cache

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}

session = requests_cache.CachedSession("dev_cache", expire_after=86_400)

def fetch(url: str) -> str:
    response = session.get(url, headers=HEADERS, timeout=15)
    response.raise_for_status()
    return response.text

if __name__ == "__main__":
    html = fetch("https://books.toscrape.com/catalogue/page-1.html")
    print(len(html), "bytes")
    html_again = fetch("https://books.toscrape.com/catalogue/page-1.html")
    print("served from cache:", session.get(
        "https://books.toscrape.com/catalogue/page-1.html", headers=HEADERS
    ).from_cache)

The first run writes dev_cache.sqlite in the working directory; every run after that reads from it. The practical effect is that a twenty-page sample becomes a fixture you can iterate against a hundred times in an afternoon without the target site ever noticing. It also makes your parser tests deterministic, because the bytes stop moving underneath them.

Two habits keep this honest. Delete the cache file deliberately when you want to confirm the crawl still works against the live site, and never ship the development expiry into production — a day-long TTL that is right for parser work is wrong for a nightly job.

2. Send Conditional Requests in Production

Once the parser is stable, the expensive part is transfer. HTTP already has a mechanism for "send me this only if it changed": you store the ETag or Last-Modified value the server returned, and echo it back on the next visit as If-None-Match or If-Modified-Since. An unchanged resource comes back as a 304 Not Modified with headers and no body. The full mechanics, including servers that hand out validators that never change, are in Incremental Crawls with ETag and Last-Modified.

import sqlite3
import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
}

db = sqlite3.connect("validators.sqlite")
db.execute("CREATE TABLE IF NOT EXISTS validators (url TEXT PRIMARY KEY, etag TEXT)")
db.commit()

def conditional_get(url: str) -> str | None:
    row = db.execute("SELECT etag FROM validators WHERE url = ?", (url,)).fetchone()
    headers = dict(HEADERS)
    if row and row[0]:
        headers["If-None-Match"] = row[0]

    response = requests.get(url, headers=headers, timeout=15)
    if response.status_code == 304:
        return None

    response.raise_for_status()
    etag = response.headers.get("ETag")
    db.execute(
        "INSERT INTO validators (url, etag) VALUES (?, ?) "
        "ON CONFLICT(url) DO UPDATE SET etag = excluded.etag",
        (url, etag),
    )
    db.commit()
    return response.text

if __name__ == "__main__":
    target = "https://books.toscrape.com/catalogue/page-1.html"
    print("changed" if conditional_get(target) else "unchanged")
    print("changed" if conditional_get(target) else "unchanged")

Run that twice and the second call prints unchanged. The request still happens — you still pay DNS, TCP, TLS and about 300 bytes of headers — but you do not pay for the body, and on a page averaging 48 kB that is a 99 per cent reduction in bytes for that URL.

3. Decide What "Changed" Means for Your Dataset

This is the decision that determines whether the whole scheme works, and it is a data question rather than an HTTP one. A page can change without your record changing — a rotating advert, a "23 people viewed this today" counter, a CSRF token in a form, a build hash in a script tag. If you treat any byte difference as a change, a site that stamps a timestamp into its footer will report 100 per cent churn every run and you will have built an expensive no-op.

Define change on the extracted record, not the response body:

import hashlib
import json

def record_fingerprint(record: dict[str, str]) -> str:
    """Stable digest of the fields you actually store."""
    payload = json.dumps(record, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()

def has_changed(record: dict[str, str], previous_digest: str | None) -> bool:
    return record_fingerprint(record) != previous_digest

if __name__ == "__main__":
    old = {"title": "A Light in the Attic", "price": "51.77", "stock": "In stock"}
    new = {"title": "A Light in the Attic", "price": "51.77", "stock": "In stock"}
    print("changed:", has_changed(new, record_fingerprint(old)))
    new["price"] = "49.99"
    print("changed:", has_changed(new, record_fingerprint(old)))

Sorting the keys matters: dictionary iteration order is stable in CPython 3.7+, but a field added in the middle of a parser refactor would otherwise reorder the JSON and invalidate every digest you hold. Note also what this excludes — if you later start extracting a new field, every fingerprint changes and the next run is a full re-crawl. That is correct behaviour, but schedule it deliberately rather than discovering it at 03:00.

For a body-level digest, normalise before hashing: strip the fragment-y parts you know are volatile, or hash only the container element your extractor reads. hashlib.blake2b(html.encode(), digest_size=16).hexdigest() is roughly twice as fast as SHA-256 on large documents and 16 bytes is ample for change detection, where a collision costs a missed update rather than a security failure.

4. Give Each Page Class Its Own TTL

A single site-wide expiry is always wrong somewhere: too short for archive pages and too long for the listing that tells you which archive pages exist. Classify by how fast the content moves.

import re
from datetime import timedelta

TTL_RULES: list[tuple[re.Pattern[str], timedelta]] = [
    (re.compile(r"/catalogue/page-\d+\.html$"), timedelta(minutes=15)),
    (re.compile(r"/catalogue/category/"), timedelta(hours=6)),
    (re.compile(r"/catalogue/[^/]+/index\.html$"), timedelta(days=7)),
    (re.compile(r"/archive/\d{4}/"), timedelta(days=3650)),
]
DEFAULT_TTL = timedelta(hours=12)

def ttl_for(url: str) -> timedelta:
    for pattern, ttl in TTL_RULES:
        if pattern.search(url):
            return ttl
    return DEFAULT_TTL

if __name__ == "__main__":
    for candidate in [
        "https://books.toscrape.com/catalogue/page-3.html",
        "https://books.toscrape.com/catalogue/category/books/travel_2/index.html",
        "https://books.toscrape.com/archive/2019/notice.html",
    ]:
        print(f"{ttl_for(candidate)}\t{candidate}")

The rule of thumb that keeps this from drifting: a page's TTL should be shorter than the interval at which a missed change would actually hurt you. If a price update discovered six hours late is acceptable, six hours is the right TTL and any shorter value is waste. Listing pages get the shortest TTL not because they change most but because they are how you discover new URLs at all — a stale listing means new detail pages are invisible, which is a silent data-completeness bug rather than a staleness bug.

5. Persist the Frontier Between Runs

The frontier is the set of URLs known, queued, in flight and done. Held in memory it evaporates on restart; held on disk it turns a crash into a resumption. SQLite is enough for a single-machine crawl of a few million URLs and gives you crash safety for free.

import sqlite3
import time

SCHEMA = """
CREATE TABLE IF NOT EXISTS frontier (
    url        TEXT PRIMARY KEY,
    state      TEXT NOT NULL DEFAULT 'queued',
    next_due   REAL NOT NULL DEFAULT 0,
    fingerprint TEXT
);
CREATE INDEX IF NOT EXISTS frontier_due ON frontier (state, next_due);
"""

class Frontier:
    def __init__(self, path: str = "frontier.sqlite") -> None:
        self.db = sqlite3.connect(path)
        self.db.execute("PRAGMA journal_mode=WAL")
        self.db.executescript(SCHEMA)
        self.db.commit()

    def add(self, url: str) -> bool:
        cur = self.db.execute(
            "INSERT OR IGNORE INTO frontier (url) VALUES (?)", (url,)
        )
        self.db.commit()
        return cur.rowcount == 1

    def due(self, limit: int = 100) -> list[str]:
        rows = self.db.execute(
            "SELECT url FROM frontier WHERE next_due <= ? ORDER BY next_due LIMIT ?",
            (time.time(), limit),
        ).fetchall()
        return [r[0] for r in rows]

    def complete(self, url: str, fingerprint: str, ttl_seconds: float) -> None:
        self.db.execute(
            "UPDATE frontier SET state='done', next_due=?, fingerprint=? WHERE url=?",
            (time.time() + ttl_seconds, fingerprint, url),
        )
        self.db.commit()

if __name__ == "__main__":
    frontier = Frontier(":memory:")
    print("new:", frontier.add("https://books.toscrape.com/catalogue/page-1.html"))
    print("duplicate:", frontier.add("https://books.toscrape.com/catalogue/page-1.html"))
    print("due now:", frontier.due())

PRAGMA journal_mode=WAL is not optional if more than one process will touch the file — without it a second writer meets database is locked under trivial load. The next_due column is what makes the scheduler incremental: a completed URL is not deleted, it is pushed into the future by its TTL, so the next run naturally picks up only what has come due.

At tens of millions of URLs the PRIMARY KEY index alone becomes the memory problem, and a probabilistic membership structure replaces it — see Deduplicating URLs with Bloom Filters for the sizing arithmetic and the Redis-backed variant.

6. Choose Where the State Lives

Three storage choices cover almost every crawl. SQLite is the default: one file, transactional, no server, comfortable to a few million rows, and trivial to inspect with a shell. Redis is the answer when several machines share one crawl — a SET for the seen-set, a sorted set keyed on next_due for the schedule, and hashes for validators — at the cost of running a server and of everything being in RAM. The filesystem, one file per response keyed by a digest of the URL, is the crudest option and the easiest to reason about; it survives anything, but a million small files punish ext4 at the inode level and directory listings become unusable, so shard into two levels of prefix directories if you go this way.

Keep the response cache and the crawl state separate even if they share a backend. They have different lifetimes: you will want to wipe a response cache after a parser change without losing the record of which URLs you have ever seen.

7. Measure the Suppression Instead of Assuming It

Every layer described so far fails silently when it is misconfigured. A cache with a poisoned key still returns correct data — it just fetches everything. A validator store against a server that emits a fresh ETag every time still works — it just never gets a 304. Neither raises, neither logs, and both look exactly like a healthy crawl in the process list. The only defence is to count the outcomes and compare them against what you expected.

from collections import Counter
from dataclasses import dataclass, field


@dataclass
class CrawlStats:
    outcomes: Counter[str] = field(default_factory=Counter)

    def record(self, outcome: str) -> None:
        self.outcomes[outcome] += 1

    def report(self) -> str:
        total = sum(self.outcomes.values()) or 1
        lines = [f"{'outcome':<16}{'count':>8}{'share':>9}"]
        for name, count in self.outcomes.most_common():
            lines.append(f"{name:<16}{count:>8}{count / total:>8.1%}")
        suppressed = total - self.outcomes["fetched"]
        lines.append(f"{'suppressed':<16}{suppressed:>8}{suppressed / total:>8.1%}")
        return "\n".join(lines)


if __name__ == "__main__":
    stats = CrawlStats()
    for outcome in ["not-due"] * 41_200 + ["cache-hit"] * 2_800 + \
                   ["not-modified"] * 4_600 + ["same-hash"] * 900 + ["fetched"] * 500:
        stats.record(outcome)
    print(stats.report())

Five buckets is enough: not-due (frontier said no), cache-hit (served locally), not-modified (a 304), same-hash (downloaded but identical) and fetched (a genuine change). Log the table at the end of every run and alert on movement rather than on absolute values. A crawl whose not-modified share falls from 60 per cent to 4 per cent overnight has almost certainly hit a server-side change — a new CDN, a rebuilt template, a rotated ETag scheme — and you want to know that on the morning it happens, not when someone notices the bandwidth bill a month later.

The one number worth an outright threshold is the fetched share. If it exceeds roughly twice the site's plausible change rate, one of the layers is not doing its job, and the ordering of the buckets tells you which: a collapsed not-due share is a frontier bug, a collapsed cache-hit share is a key problem, and a collapsed not-modified share is the server's doing.

Performance and Scaling Considerations

Incremental crawling changes the shape of a run rather than making each request faster. The first run costs exactly what a full crawl costs, because it has to populate every store. From the second run on, the request budget collapses to the change rate.

The second-order effect is the one that matters for politeness. If your nightly job used to send 50,000 requests spread over six hours at roughly 2.3 requests per second, and incremental crawling drops it to 6,000 requests, the naive reaction is to keep the same delay and finish in forty minutes. The better reaction is to keep the same rate and finish early, because your concurrency budget was negotiated against the site's capacity, not against your deadline. The saving should be spent on being a smaller fraction of the target's traffic, and only afterwards on shortening the window.

That said, 304 responses are genuinely cheap for the origin — they usually short-circuit before any template rendering or database query — so a validated request is worth perhaps a tenth of a full one in server cost. A reasonable policy is to raise the request rate for the validation sweep and drop back to the original rate for URLs that actually returned a body. The concurrency plumbing for this is the same as in Asynchronous Scraping with Asyncio and HTTPX: one semaphore for validations, a tighter one for full fetches.

Memory is the other constraint. A Python set of 10 million URL strings costs on the order of 1.2 GB — roughly 60 bytes of str overhead plus the characters, plus the set's own table. Hashing to a 16-byte digest cuts that to about 800 MB, which is still one Python object per URL. Only a bit array gets it into the tens of megabytes.

Two cache-specific numbers are worth measuring rather than assuming. SQLite as a response cache sustains a few thousand reads per second on a single connection and far fewer concurrent writes; if your workers are blocking on cache writes, either batch them or move to Redis. And a cache is only useful if it is hit — instrument from_cache and log the ratio, because a cache with a 2 per cent hit rate is a bug, not an optimisation. Emitting that ratio as a metric is exactly the kind of signal covered in Monitoring and Alerting for Scrapers.

Common Errors and Fixes

sqlite3.OperationalError: database is locked — two processes writing the same cache or frontier file with the default rollback journal. Enable write-ahead logging and set a busy timeout so writers queue instead of failing:

import sqlite3

db = sqlite3.connect("frontier.sqlite", timeout=30)
db.execute("PRAGMA journal_mode=WAL")
db.execute("PRAGMA busy_timeout=30000")
db.commit()

Hit rate of zero with a cache that is clearly populated — the cache key includes a header you rotate. Rotating the User-Agent per request, which is the standard defence described in How to Rotate User Agents in Python, creates a brand-new key on every call if that header is part of the key. Match on headers only when the response genuinely varies by them.

Every page reports as changed — you are hashing the whole body and the site stamps something volatile into it. Hash the extracted record instead, or hash only the subtree your extractor reads.

304 responses being parsed as HTML — requests gives a 304 an empty response.text, and a parser handed an empty string usually returns zero items rather than raising. That silently empties your dataset. Branch on response.status_code == 304 before anything touches the body, and treat the count of 304s as a first-class metric.

Cache grows without bound — an expiry is not a deletion. requests-cache keeps expired rows until you call session.cache.delete(expired=True); a filesystem cache keeps them forever. Schedule a vacuum, and remember that a SQLite file does not shrink on DELETE unless you run VACUUM.

Stale data after a parser fix — the response cache correctly returned yesterday's bytes for a URL whose extraction logic you just changed. Version your cache: put a schema version in the cache name (dev_cache_v3) and bump it whenever the parser's output shape changes, so a fix never silently reuses records built by the old code. The same versioning discipline belongs in whatever you write records into, as discussed in Storing and Exporting Scraped Data.

Scrapy's cache appears to do nothing — HTTPCACHE_ENABLED = True uses the dummy policy by default, which ignores all cache-control headers and caches everything for HTTPCACHE_EXPIRATION_SECS. Switch to scrapy.extensions.httpcache.RFC2616Policy if you want validators and Cache-Control respected; the framework details are in Web Scraping with Scrapy.

Frequently Asked Questions

Does a response cache make my crawler impolite or unfair to the site? The opposite. A cache reduces the number of requests the site has to serve, and conditional requests let it answer with a header-only 304 instead of rendering a page. The one thing to watch is that a long TTL can make you act on stale data, which is your problem rather than the site's.

Should I cache during development and in production, or only one of them? Both, with different settings. In development you want a long expiry and a single local file so that parser iteration costs nothing. In production you want short, per-page-class TTLs and conditional revalidation so that the crawl reflects reality within a known lag.

What happens if a site returns no ETag and no Last-Modified header? You fall back to downloading the body and comparing a digest against the one you stored. That saves parsing, validation and database writes but not bandwidth, so pair it with a longer TTL for page classes you know are slow-moving.

How do I share a cache across several machines? Point the cache backend at Redis and give every worker the same connection settings, then make sure the cache key does not include anything machine-specific such as a per-worker proxy or a rotated header. A shared cache also needs a shared expiry policy, otherwise one worker keeps resurrecting entries another has just retired.

Is a Bloom filter a replacement for the frontier database? No — it replaces only the "have I seen this URL" membership test. You still need durable state for scheduling, validators and fingerprints, because a Bloom filter cannot tell you when a URL is next due or what its last digest was.

Related

• Scaling & Deploying Python Web Scrapers — the parent section this guide belongs to
• HTTP Caching with requests-cache — backends, expiry rules and a development workflow that costs no requests
• Incremental Crawls with ETag and Last-Modified — conditional requests and a SQLite validator store
• Deduplicating URLs with Bloom Filters — membership testing when a set no longer fits
• Distributed Crawling with Celery and Redis — sharing frontier and cache state across workers