Incremental Crawls with ETag and Last-Modified

A conditional request asks the server to send a page only if it has changed since you last saw it, and this page — part of Caching and Incremental Crawling — shows how to store the validators, send them and survive the servers that get them wrong.

The mechanism is small. On the first fetch the server may return an ETag header (an opaque token identifying that exact representation) or a Last-Modified header (a timestamp). You store whichever you got, keyed by URL. On the next visit you send it back as If-None-Match or If-Modified-Since, and if nothing has changed the server answers 304 Not Modified with headers and an empty body. A 48 kB page becomes roughly 300 bytes of response headers.

Why the Two Validators Are Not Equivalent

ETag is the stronger signal because it is derived from the representation itself. A strong entity tag such as "a1b2c3" promises byte equality; a weak one, written W/"a1b2c3", promises only semantic equivalence, which is what a server sends when it gzips at different levels or reorders a header but keeps the content the same. For change detection weak tags are fine — you care about the content, not the octets.

Last-Modified carries a Mon, 14 Jul 2025 09:02:11 GMT timestamp with one-second resolution, and that resolution is its defect. A page edited twice in the same second, or edited within the same second as your fetch, can be reported as unchanged. It also breaks on any resource assembled from multiple sources, because the server has to pick one modification time and often picks the template's.

When a response carries both, send both. If-None-Match takes precedence per RFC 9110 — a server that understands entity tags must ignore If-Modified-Since when If-None-Match is present — but sending both costs nothing and covers servers that only implement one.

A SQLite Validator Store

The state you need is one row per URL. SQLite handles it with no server, survives a crash, and stays inspectable.

import hashlib
import sqlite3
import time
from dataclasses import dataclass

import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}

SCHEMA = """
CREATE TABLE IF NOT EXISTS validators (
    url           TEXT PRIMARY KEY,
    etag          TEXT,
    last_modified TEXT,
    body_hash     TEXT,
    last_checked  REAL NOT NULL
);
"""


@dataclass
class FetchResult:
    changed: bool
    reason: str
    body: str | None


class ValidatorStore:
    def __init__(self, path: str = "validators.sqlite") -> None:
        self.db = sqlite3.connect(path)
        self.db.execute("PRAGMA journal_mode=WAL")
        self.db.executescript(SCHEMA)
        self.db.commit()

    def load(self, url: str) -> tuple[str | None, str | None, str | None]:
        row = self.db.execute(
            "SELECT etag, last_modified, body_hash FROM validators WHERE url = ?",
            (url,),
        ).fetchone()
        return row if row else (None, None, None)

    def save(
        self,
        url: str,
        etag: str | None,
        last_modified: str | None,
        body_hash: str | None,
    ) -> None:
        self.db.execute(
            "INSERT INTO validators (url, etag, last_modified, body_hash, last_checked) "
            "VALUES (?, ?, ?, ?, ?) "
            "ON CONFLICT(url) DO UPDATE SET "
            "  etag=excluded.etag, last_modified=excluded.last_modified, "
            "  body_hash=excluded.body_hash, last_checked=excluded.last_checked",
            (url, etag, last_modified, body_hash, time.time()),
        )
        self.db.commit()


def digest(body: str) -> str:
    return hashlib.blake2b(body.encode("utf-8"), digest_size=16).hexdigest()


def conditional_fetch(
    session: requests.Session, store: ValidatorStore, url: str
) -> FetchResult:
    etag, last_modified, body_hash = store.load(url)

    headers = dict(HEADERS)
    if etag:
        headers["If-None-Match"] = etag
    if last_modified:
        headers["If-Modified-Since"] = last_modified

    response = session.get(url, headers=headers, timeout=20)

    if response.status_code == 304:
        store.save(url, etag, last_modified, body_hash)
        return FetchResult(changed=False, reason="304", body=None)

    response.raise_for_status()
    new_hash = digest(response.text)
    store.save(
        url,
        response.headers.get("ETag"),
        response.headers.get("Last-Modified"),
        new_hash,
    )
    if body_hash is not None and new_hash == body_hash:
        return FetchResult(changed=False, reason="same-hash", body=response.text)
    return FetchResult(changed=True, reason="new-body", body=response.text)


if __name__ == "__main__":
    store = ValidatorStore("validators.sqlite")
    with requests.Session() as session:
        target = "https://books.toscrape.com/catalogue/page-1.html"
        for attempt in (1, 2):
            result = conditional_fetch(session, store, target)
            print(f"run {attempt}: changed={result.changed} reason={result.reason}")

Run it twice. The first pass stores whatever validators the server offered plus a digest of the body; the second either gets a 304 (reason="304") or, if the server ignores conditional headers, downloads the page again and reports reason="same-hash". Both outcomes correctly say changed=False, which is the point: the caller does not need to know which mechanism did the work.

The body_hash column is what makes this robust. It costs one extra 16-byte string per URL and it is the only signal available against a server that never sends validators, which in practice is a large share of dynamically rendered sites.

Reading the Response Correctly

Three details cause most of the bugs in code like the above.

A 304 has no body. requests gives it an empty response.text, and response.raise_for_status() does not raise for a 3xx status, so an unguarded parser receives an empty string and silently extracts zero items. Branch on the status code before anything touches the body.

requests follows redirects by default, so the status you inspect belongs to the final response. If a URL 301s somewhere else, the ETag you store belongs to the target, not the URL you keyed on. Either key the store on response.url or record the redirect separately so the next run starts at the destination.

The ETag header must be echoed verbatim, quotes included. ETag: "a1b2c3" is sent back as If-None-Match: "a1b2c3" with the double quotes in the header value. Stripping them produces a malformed condition that most servers answer with a full 200, which looks like "the page changed" and quietly destroys your hit rate. The same applies to the W/ prefix on weak tags — leave it in place.

When the Server Lies

Not every origin implements this properly, and the failure modes are specific:

• A new ETag on every response. Common behind load balancers where each node hashes the response including a per-node timestamp or a rotating nonce in the HTML. You always get 200 and the tag is never reused. Detect it by comparing your body_hash with the new one: if the body is identical but the ETag differs, stop sending If-None-Match for that host and rely on hashing.
• Last-Modified set to the request time. Some frameworks emit Last-Modified: <now> for dynamically generated pages, which makes If-Modified-Since permanently true. Same detection, same fallback.
• 304 returned for a page that did change. Rarer and much worse, because you never see the update. It usually comes from an over-aggressive CDN rule. Guard against it by forcing an unconditional fetch for a random sample — say 2 per cent of URLs per run — and alerting when a sampled page's hash differs from what the 304 implied.
• Conditional headers ignored entirely. You get a full 200 every time. Nothing is broken, you just get no bandwidth saving, and the content hash is doing all the work.
• ETag varies with Accept-Encoding. A server that generates the tag from the compressed bytes hands out a different tag to a client that requests gzip than to one that requests br. Pin Accept-Encoding to a single value for the whole crawl so the tag you stored is the tag you will be compared against.

A practical policy is to track, per host, the ratio of 304 responses to conditional requests sent. If a host is below a few per cent after a couple of runs, its validators are not usable and you should skip the extra headers entirely — they cost nothing but they also buy nothing, and the log noise hides real problems. Emitting that ratio is the kind of health signal covered in Monitoring and Alerting for Scrapers.

Running the Revalidation Sweep Separately

Once validators work, the crawl naturally splits into two phases with different economics. The sweep sends conditional requests for every URL that has come due and produces a list of URLs that actually changed. The fetch phase then downloads and parses only that list. Separating them lets each phase have its own concurrency limit, which is the point: a 304 costs the origin almost nothing, while a full render can cost it hundreds of milliseconds of CPU.

import asyncio

import httpx

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
}

SWEEP_CONCURRENCY = 24
FETCH_CONCURRENCY = 6


async def probe(
    client: httpx.AsyncClient, url: str, etag: str | None, gate: asyncio.Semaphore
) -> tuple[str, bool]:
    headers = dict(HEADERS)
    if etag:
        headers["If-None-Match"] = etag
    async with gate:
        response = await client.head(url, headers=headers, timeout=15)
    return url, response.status_code != 304


async def sweep(known: dict[str, str | None]) -> list[str]:
    gate = asyncio.Semaphore(SWEEP_CONCURRENCY)
    async with httpx.AsyncClient(follow_redirects=True) as client:
        results = await asyncio.gather(
            *(probe(client, url, etag, gate) for url, etag in known.items()),
            return_exceptions=False,
        )
    return [url for url, changed in results if changed]


if __name__ == "__main__":
    known = {
        "https://books.toscrape.com/catalogue/page-1.html": None,
        "https://books.toscrape.com/catalogue/page-2.html": None,
    }
    stale = asyncio.run(sweep(known))
    print(f"{len(stale)} of {len(known)} URLs need a full fetch")
    print(f"sweep at {SWEEP_CONCURRENCY} in flight, fetch at {FETCH_CONCURRENCY}")

A HEAD request is worth considering for the sweep because it never returns a body even on a 200, so a server that ignores conditional headers still costs you only headers. The catch is that a meaningful minority of sites answer HEAD with a 405, and a few answer it with a 200 and stale headers generated by a different code path. Probe a sample of the target's URLs both ways before committing; if HEAD disagrees with GET, use a conditional GET and accept the occasional full body. The concurrency mechanics here are the same ones described in Asynchronous Scraping with Asyncio and HTTPX.

The output of a sweep is also the most useful thing you can put on a dashboard. The count of changed URLs per run is a direct measurement of the site's churn, and it moves for real reasons — a catalogue refresh, a template deploy, a seasonal repricing. A run where that count jumps to 100 per cent almost always means the site changed something structural rather than that every page genuinely changed, and it is worth blocking the fetch phase until a human looks.

Edge Cases and Caveats

• Weak tags and byte comparison. W/"abc" means semantically equivalent, so a 304 against a weak tag does not promise identical bytes. For extraction that is fine; for archiving raw HTML it is not.
• Compression changes the body you hash. Hash response.text after requests has decoded the transfer encoding, never response.content from a stream you decoded differently between runs.
• Character encoding drift. If the server omits a charset and requests guesses differently on two runs, the decoded text differs and the hash changes even though the bytes did not. Set response.encoding explicitly when you know it; the failure modes are covered in Fixing Common Unicode Errors in Python Scraping.
• APIs versus HTML. JSON APIs implement validators far more reliably than rendered pages. If the site has an endpoint behind the HTML, as described in Reverse-Engineering Private APIs, conditional requests against it will usually work where they fail against the page.
• Clock skew on If-Modified-Since. The timestamp must be the one the server gave you, echoed verbatim. Generating your own from a local clock invites a skewed container to declare everything modified.
• A 304 still costs a round trip. DNS, TCP and TLS are unchanged, so the saving is bandwidth and origin CPU, not latency. On a crawl bounded by connection setup rather than transfer, conditional requests help less than the byte counts suggest.
• Storage growth. The validator table only ever grows. Delete rows whose last_checked is older than the point at which you would treat the URL as gone, otherwise a site with rotating URL parameters slowly fills the disk.

Frequently Asked Questions

Do I need both If-None-Match and If-Modified-Since? Send both when you have both. A server that supports entity tags is required to ignore the timestamp condition, and a server that only implements timestamps will use the one it understands, so sending both costs a few dozen bytes and covers more implementations.

What should my crawler do when it receives a 304? Treat it as "no work to do": skip parsing, skip the database write, refresh the last_checked timestamp and push the URL's next due time forward by its TTL. Do not overwrite the stored validators with the ones from the 304 unless the server sent new ones.

Is content hashing a replacement for conditional requests? No, it is the fallback. A conditional request avoids transferring the body at all, while hashing requires you to download it first and only saves the parse and the write. Use validators when the server supports them and hashing when it does not.

Why does my crawler get 200 on every request even though I send the validators? Check three things in order: that you are echoing the ETag with its quotes and any W/ prefix intact, that you are storing the validator against the URL you actually end up requesting after redirects, and that Accept-Encoding is identical between runs. If all three are right, the server is generating a fresh tag each time and you should fall back to hashing.

Related

• Caching and Incremental Crawling — the parent topic and the layer model this fits into
• HTTP Caching with requests-cache — a library that can drive these validators for you
• Deduplicating URLs with Bloom Filters — the membership half of the same problem