Detecting Silent Scraper Failures

The failure mode that costs the most data is not a crash — it is a run that returns 200 OK, parses valid HTML, extracts nothing and exits zero, which is why it gets its own page under Monitoring and Alerting for Scrapers.

Both rows describe a page that produced no data. Only the top row raises an exception, and an exception is the only thing most schedulers know how to notice.

The only reliable defence is to make the run assert its own output before it exits. Five checks cover nearly every real case: an expected-row-count band, per-field null-rate thresholds, schema validation at the write boundary, a comparison against a rolling median of recent runs, and a canary URL whose correct values you already know. Run all five, exit non-zero if any fails, and a silent failure becomes an ordinary loud one that your scheduler already knows how to report.

Why a Broken Scraper Still Returns 200

Nothing in the HTTP layer knows what your parser expects. A response is successful if the server produced a document, and every one of these produces a document:

A front-end redesign. div.product_pod became article[data-testid="product-card"] overnight. The page is fine; your selector is not.
A soft 404. Many sites return 200 with a "we couldn't find that" page rather than a real 404, so both raise_for_status() and a status-code check pass.
A bot wall that renders. An interstitial or a JavaScript challenge returns 200 with a small, valid, entirely dataless document.
Client-side rendering. The server returns a 4 KB shell and the data arrives by XHR. requests sees a valid page with no products in it.
A silent login expiry. The session cookie lapsed, the site redirects to a login form, and the login form is a 200.
Geographic or A/B variation. A new datacentre IP range gets a different template with different class names.
Pagination that stops. Page 2 returns page 1's content, or an empty results container, so the crawl finishes early with a plausible-looking subset.

In every case the request layer is healthy, the parse layer produces an empty list, and the storage layer dutifully writes zero rows. No exception is raised anywhere, so retries do not trigger, error rates stay flat, and run duration actually improves because there was less work to do. A faster-than-usual run is a warning sign, not a win.

The Five Checks

Expected-row-count band. The cheapest check there is: a run that produced 40 items when it usually produces 5,000 is broken. Express it as a band rather than a floor, because an unexplained tripling is also worth knowing about — it usually means duplicate extraction or a pagination loop.

Per-field null-rate. Row count alone misses partial breakage, which is the more common case. A redesign that moves only the price element leaves you with 5,000 rows where price is None. Track the fraction of saved items missing each required field and threshold each one separately. A field that is normally 0.1% null and is suddenly 40% null is the single clearest signal that a selector broke.

Schema assertion on write. Validate at the boundary where data leaves the crawl, so nothing malformed reaches storage in the first place. Types, ranges and required-ness catch a price parsed as "£51.77" instead of 51.77 and a rating that came back as the string "Three". This overlaps with Cleaning and Validating Scraped Data, which covers the validation library choices; here it is used as a monitoring signal — count the rejections rather than only dropping them.

Comparison against a rolling median. Absolute thresholds rot. A site that grows 5% a month will breach a hand-picked floor eventually, and a site that shrinks will make it meaningless. Keep the item count of the last N runs, take the median, and require the current run to land within a tolerance of it.

Absolute thresholds age badly as a site grows. A band derived from the last few runs moves with the target and still catches a collapse within one run.

The median matters more than the mean here: one already-broken run near zero drags a mean down far enough that the next broken run looks acceptable, whereas a median over seven runs needs four bad runs before it moves. Use a window long enough to smooth weekday and weekend variation — seven runs for a daily crawl, twenty-four for an hourly one.

Canary URLs. The only check that catches a subtle change rather than a total one. Pick two or three stable pages, record their correct values by hand, and assert them at the end of every run. If https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html no longer yields a price of 51.77, something changed in the extraction path regardless of what the aggregate counts say. Canaries catch the case where the selector still matches but now matches the wrong node — a list price instead of a sale price, a decimal separator misread, a currency symbol left in the string. No statistical check can see that; every value is present and plausible.

Each gate is a cheap assertion over the run's own output. Any one of them failing should turn a clean exit code into a non-zero one before the data is published.

A Runnable Yield Gate

This module implements all five checks and returns a process exit code. It stores the run history in a small JSON file, which is enough for a single scheduled job; for a fleet, put the same three columns in the database you already write to.

# scraper/yieldgate.py
import json
import statistics
import sys
from dataclasses import dataclass
from pathlib import Path

HISTORY = Path("run_history.json")
WINDOW = 7
REQUIRED_FIELDS = ("title", "price", "availability")
MAX_NULL_RATE = 0.05
BAND = 0.4  # allow 40% either side of the rolling median

@dataclass(frozen=True)
class CheckResult:
    ok: bool
    problems: tuple[str, ...]

def _load_history() -> list[int]:
    if not HISTORY.exists():
        return []
    return json.loads(HISTORY.read_text(encoding="utf-8"))[-WINDOW:]

def _save_history(history: list[int], count: int) -> None:
    HISTORY.write_text(json.dumps((history + [count])[-WINDOW:]), encoding="utf-8")

def check_run(
    items: list[dict[str, object]],
    canaries: dict[str, dict[str, object]],
    min_rows: int = 20,
) -> CheckResult:
    problems: list[str] = []
    count = len(items)

    # 1. absolute floor — catches the total collapse with no history at all
    if count < min_rows:
        problems.append(f"only {count} items, floor is {min_rows}")

    # 2. rolling-median band
    history = _load_history()
    if len(history) >= 3:
        median = statistics.median(history)
        low, high = median * (1 - BAND), median * (1 + BAND)
        if not low <= count <= high:
            problems.append(f"{count} items outside band {low:.0f}-{high:.0f} (median {median:.0f})")

    # 3. per-field null-rates
    for name in REQUIRED_FIELDS:
        missing = sum(1 for item in items if not item.get(name))
        rate = missing / count if count else 1.0
        if rate > MAX_NULL_RATE:
            problems.append(f"null-rate for {name} is {rate:.1%} (max {MAX_NULL_RATE:.0%})")

    # 4. schema assertion on the sample that will be written
    for item in items[:200]:
        price = item.get("price")
        if not isinstance(price, (int, float)) or not 0 < float(price) < 100_000:
            problems.append(f"price failed schema check: {price!r}")
            break

    # 5. canaries — known URLs with known-good values
    by_url = {str(item.get("url")): item for item in items}
    for url, expected in canaries.items():
        actual = by_url.get(url)
        if actual is None:
            problems.append(f"canary {url} produced no item")
            continue
        for key, want in expected.items():
            if actual.get(key) != want:
                problems.append(f"canary {url}: {key} was {actual.get(key)!r}, expected {want!r}")

    if not problems:
        _save_history(history, count)
    return CheckResult(ok=not problems, problems=tuple(problems))

def main(items: list[dict[str, object]], canaries: dict[str, dict[str, object]]) -> int:
    result = check_run(items, canaries)
    for problem in result.problems:
        print(f"YIELD GATE FAILED: {problem}", file=sys.stderr)
    return 0 if result.ok else 1

if __name__ == "__main__":
    sample = [
        {"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
         "title": "A Light in the Attic", "price": 51.77, "availability": "In stock"},
    ] * 40
    expected = {
        "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html":
            {"title": "A Light in the Attic", "price": 51.77},
    }
    sys.exit(main(sample, expected))

Two design choices carry most of the value. History is only appended when the run passed, so a sequence of broken runs cannot drag the median down to meet them — the band stays anchored to the last known-good behaviour. And the absolute floor runs first and independently of history, so the very first run and any run after the history file is lost still get a check.

The order of the checks is also deliberate: cheapest and most decisive first. A run with 40 items instead of 5,000 will trip the floor, the band and every null-rate at once, and reporting all of them is noise. In practice you want the first problem in the list to be the one that names the failure, which is why the absolute floor precedes the field-level detail. Collecting every problem rather than returning on the first is still worth it, because the combination is diagnostic: a normal row count with one field at 100% null is a moved selector, while a collapsed row count with normal null-rates is a blocked or truncated crawl.

Wire the exit code into whatever runs the crawl: a non-zero exit already fails a scheduled workflow, marks a container task unhealthy and stops a downstream load step. If the gate fails, the correct action is usually to keep the run's output in a staging table and not promote it, rather than to write nothing — inspecting the bad rows is how you find out which selector moved.

Edge Cases and Caveats

Incremental crawls have no stable yield. A crawl that only fetches pages changed since the last run legitimately produces 4,000 items one day and 12 the next. Band the ratio of items to pages fetched instead of the raw count, or track the two separately, as discussed in Caching and Incremental Crawling.
Seasonality breaks a short window. A retail catalogue in the week before a holiday genuinely triples. A seven-run window over a daily crawl spans one week and will alert on it; either widen the window or compare against the same weekday a week ago.
Canary pages get deleted. A canary that 404s is a broken check, not a broken scraper, and it will fail every run until someone notices. Treat a missing canary as a distinct, lower-severity signal and keep three so a single deletion does not blind you.
Partial-field checks need a denominator you trust. A null-rate computed over items that were already dropped by validation looks perfect while most of the data is being discarded. Count rejections at the validation step and add them back before dividing.
Some fields are legitimately sparse. discount_price may be null for 90% of products. Threshold each field against its own historical rate, not against a global 5%.
The first three runs have no history. The band check is skipped below three data points, which is correct but means the absolute floor is doing all the work for a day or two. Seed the history file from a manual run rather than leaving it empty.
A history file on ephemeral disk resets every run. In a container or a scheduled runner, run_history.json disappears with the filesystem, so the band check never accumulates three data points and silently never runs. Persist the history where the data goes — a small table alongside the output, or an object-store key — rather than next to the code.
Retries can hide a partial outage. If half the pages fail and are retried into eventual success, the yield is fine and the run took twice as long. Watch run duration alongside yield; the pattern is covered by the backoff behaviour in Retrying Failed Requests with Tenacity.

Frequently Asked Questions

How large should the tolerance band be? Start at plus or minus 40% of the rolling median and tighten it after observing thirty runs. Compute the actual spread of your own history — if the middle 90% of runs land within 12% of the median, a 40% band is far too loose and a real collapse could sit inside it for days.

Should a failed check stop the data being written at all? Write it, but do not promote it. Keeping the output of a failed run in a staging table is how you diagnose which selector moved, and it costs nothing. Blocking the promotion step is what protects downstream consumers, which is a different decision from blocking the write.

How do I pick canary values that will not change legitimately? Choose attributes the site has no reason to edit: a product's title and identifier rather than its price or stock level. If the target has no such page, assert structure instead of value — that the page yields exactly one item and that every required field is non-null.

Can I detect a silent failure without changing the scraper? Partly. A query over the stored data — rows written per run, null-rate per column, maximum row timestamp — catches most of it from outside and needs no code change, which makes it a good first move on a scraper you inherited. It cannot catch canary drift, because that requires knowing what the right answer was.

Monitoring and Alerting for Scrapers — the topic this page belongs to
Structured Logging for Python Scrapers — logging the missing-field names that identify the broken selector
Exporting Scrapy Metrics to Prometheus — expressing the same yield checks as alert rules
Writing Selectors That Survive a Redesign — reducing how often the collapse happens in the first place