Monitoring and Alerting for Scrapers

This is the observability layer of Scaling and Deploying Python Web Scrapers, and it exists because a scraper almost never crashes — it degrades. The process keeps exiting zero, the target keeps returning 200 OK, and the fields quietly come back empty for three weeks before anyone downstream notices. This guide covers what to measure, what to log, where to send it, and how to set thresholds that catch a collapse without waking you for a single 404.

Ordinary application monitoring assumes that failure is loud: a request throws, a process dies, a health check goes red. Scrapers break the assumption. The failure that costs you a month of data is a CSS selector that stopped matching after a front-end deploy, and nothing about that is an exception. The parse function returns an empty list, the pipeline writes nothing, the job finishes early because there was less work to do, and every conventional signal reads green. Monitoring a scraper means monitoring its output, not just its process.

When to Add Monitoring, and How Much

Instrumentation has a cost in code and in attention. Match the depth to what the crawl is worth:

Two rules of thumb decide the rest. First, if nobody would notice for more than one run that the data stopped arriving, you need an automated check — humans do not notice absence. Second, if you would not get out of bed for it, it must not page you; put it in a digest instead. Most scraper monitoring failures are not missing signals, they are signals routed at the wrong volume until everyone mutes the channel.

The other thing worth deciding early is where the crawl runs, because that constrains the transport. A crawl on a scheduled runner has no persistent process to expose an endpoint, so it must push. A long-lived container behind a queue can expose /metrics and be scraped normally. Deploying Scrapers to the Cloud covers the deployment shapes; this guide assumes you have picked one.

Prerequisites

Python 3.10 or newer. The core pattern needs nothing outside the standard library — logging, json, time and sys are enough for structured logs and exit codes. Metrics export and HTTP need three small packages:

python -m pip install "httpx>=0.27" "prometheus-client>=0.20" "tenacity>=8.2"

If you are exporting from a short-lived job you will also want a push target reachable from the crawl. For Prometheus that is a Pushgateway (docker run -d -p 9091:9091 prom/pushgateway) or a node exporter textfile directory. Hosted collectors — Grafana Cloud, Datadog, Honeycomb — accept the same counters over their own endpoints and remove the need to run either.

Step-by-Step: Instrumenting a Crawl

1. Give Every Run an Identity

Nothing else works without this. Generate one identifier when the process starts and attach it to every log line, every metric sample and every row you write. It turns "the scraper is broken" into "run a1f9c2 fetched 12,481 pages and saved 40 items", which is a question you can answer.

# scraper/context.py
import os
import time
import uuid
from dataclasses import dataclass, field

@dataclass
class RunContext:
    run_id: str = field(default_factory=lambda: uuid.uuid4().hex[:12])
    spider: str = "books"
    started_at: float = field(default_factory=time.monotonic)
    version: str = field(default_factory=lambda: os.environ.get("GIT_SHA", "dev"))

    def elapsed_s(self) -> float:
        return round(time.monotonic() - self.started_at, 3)

    def as_fields(self) -> dict[str, str]:
        return {"run_id": self.run_id, "spider": self.spider, "version": self.version}

Use time.monotonic() for durations and wall-clock time only for timestamps. A machine that syncs its clock mid-run will otherwise report a negative duration, which then poisons any average built on top of it.

2. Log One Structured Event per Request

Free-text logs are unqueryable at any interesting volume. Emit one JSON object per line, with a stable set of keys, and let the collector index the fields. The events that matter in a crawl are narrow: a run started, a URL was fetched, a page was parsed, an item was stored, a URL was abandoned, a run finished.

# scraper/events.py
import logging
from typing import Any

logger = logging.getLogger("scraper")

def emit(event: str, /, **fields: Any) -> None:
    """Log one structured event. Fields land as JSON keys, not inside the message."""
    logger.info(event, extra={"fields": fields})

def fetched(url: str, status: int, elapsed_ms: int, attempt: int) -> None:
    emit("fetch", url=url, status=status, elapsed_ms=elapsed_ms, attempt=attempt)

def parsed(url: str, items: int, missing: list[str]) -> None:
    level = logging.WARNING if items == 0 else logging.INFO
    logger.log(level, "parse", extra={"fields": {"url": url, "items": items, "missing": missing}})

Two details make this worth doing. The missing list — the names of fields the parser expected and did not find — is the single highest-value thing a scraper can log, because it names the broken selector before anyone opens the database. And a page that yields zero items is logged at WARNING, not INFO, so a level filter alone surfaces it. The formatter that turns these records into JSON, the correlation ids, and the fields you must never write are covered in Structured Logging for Python Scrapers.

3. Count the Four Signal Families

Logs tell you what happened to one URL. Metrics tell you what happened to the run. Keep the counters in one object so they can be summarised at the end and exported in the middle.

# scraper/metrics.py
from collections import Counter
from dataclasses import dataclass, field

@dataclass
class RunMetrics:
    pages: int = 0
    items: int = 0
    by_status: Counter[int] = field(default_factory=Counter)
    nulls: Counter[str] = field(default_factory=Counter)
    retries: int = 0
    give_ups: int = 0

    def record_response(self, status: int) -> None:
        self.pages += 1
        self.by_status[status] += 1

    def record_item(self, item: dict[str, object], fields: tuple[str, ...]) -> None:
        self.items += 1
        for name in fields:
            if not item.get(name):
                self.nulls[name] += 1

    def error_rate(self) -> float:
        bad = sum(n for status, n in self.by_status.items() if status >= 400)
        return round(bad / self.pages, 4) if self.pages else 0.0

    def null_rate(self, name: str) -> float:
        return round(self.nulls[name] / self.items, 4) if self.items else 1.0

    def items_per_page(self) -> float:
        return round(self.items / self.pages, 3) if self.pages else 0.0

The five numbers worth emitting from this are pages per minute, error rate by status, items per page, null-rate per field, and run duration. Everything else is derivable. Note that null_rate returns 1.0 when no items were saved at all: a run that produced nothing has a hundred percent null-rate by definition, and defaulting it to zero would let the worst possible run pass a null-rate threshold.

Each of the four families answers a different question, and the order matters when you are diagnosing rather than alerting. Run health answers "did it execute" — exit code, wall-clock duration, peak memory, whether the scheduler fired at all. HTTP outcomes answer "did the target cooperate" — the share of 2xx, the appearance of 403 or 429 that means a block, the 5xx that means the site is struggling. Extraction yield answers "did the parser still work" — items per page and null-rate per field, the only two numbers that move when a selector breaks. Data freshness answers "is anyone downstream getting stale data" — the age of the newest row, which is the one signal that survives the scraper disappearing entirely. A green reading on the first two with a collapsed third is the classic silent failure; a green reading on the first three with a stale fourth means the crawl is running fine and the write path is broken.

4. Send the Numbers Somewhere

There are three destinations, and most projects end up using two of them.

Standard output, as JSON. In a container this is not a fallback — it is the correct answer. The runtime already captures stdout, ships it, timestamps it and retains it, and every platform from Docker Compose to ECS to a scheduled CI runner does this without configuration. Writing to a log file inside a container means writing to a disk that disappears. Never open a file handle for logs in a containerised scraper.

A Prometheus registry. Counters and gauges pushed to a Pushgateway or written to a node exporter textfile directory, then scraped, stored and alerted on with the same rules as the rest of your infrastructure. This is the right choice when a crawl is one job among many and the on-call rotation already lives in Alertmanager. The Scrapy-specific wiring, the stats keys worth exporting and the push-versus-scrape problem are in Exporting Scrapy Metrics to Prometheus.

A hosted collector. Grafana Cloud, Datadog, Honeycomb and their peers all accept counters over HTTP with an API key, which removes an entire piece of infrastructure. The trade is cost per series: label a metric by URL rather than by status code and a hosted bill grows without limit. Label by things with a small, bounded set of values — spider name, status code, field name, finish reason — and never by anything drawn from the pages you crawl.

# scraper/export.py
from prometheus_client import CollectorRegistry, Counter, Gauge, push_to_gateway

def push_run(gateway: str, spider: str, pages: int, items: int, duration_s: float) -> None:
    registry = CollectorRegistry()
    Counter("scraper_pages_total", "Pages fetched", registry=registry).inc(pages)
    Counter("scraper_items_total", "Items stored", registry=registry).inc(items)
    Gauge("scraper_duration_seconds", "Wall-clock run time", registry=registry).set(duration_s)
    Gauge("scraper_last_success_timestamp", "Unix time of last good run", registry=registry).set_to_current_time()
    push_to_gateway(gateway, job="scraper", grouping_key={"spider": spider}, registry=registry)

scraper_last_success_timestamp deserves special attention. It is the freshness signal, and it is the only metric that keeps working when the scraper stops running entirely. Every other counter goes missing along with the process; a timestamp gauge sitting in the Pushgateway keeps aging, so time() - scraper_last_success_timestamp > 90000 fires whether the crawl failed, the scheduler died, or somebody deleted the cron entry. Build that alert first and the rest is refinement.

5. Assert the Run Before It Exits

This is the step that converts a silent failure into a loud one. Before the process returns, check the run against expectations and exit non-zero if it does not meet them. Schedulers, CI runners and container orchestrators all already know how to alert on a non-zero exit; you are simply giving them something true to react to.

# scraper/gate.py
import sys
from scraper.metrics import RunMetrics

REQUIRED = ("title", "price", "availability")

def gate(metrics: RunMetrics, expected_items: int, tolerance: float = 0.4) -> int:
    problems: list[str] = []
    if metrics.items < expected_items * (1 - tolerance):
        problems.append(f"yield {metrics.items} below floor {int(expected_items * (1 - tolerance))}")
    if metrics.error_rate() > 0.05:
        problems.append(f"error rate {metrics.error_rate():.1%} above 5%")
    for name in REQUIRED:
        if metrics.null_rate(name) > 0.05:
            problems.append(f"null-rate for {name} is {metrics.null_rate(name):.1%}")
    for problem in problems:
        print(f"GATE FAILED: {problem}", file=sys.stderr)
    return 1 if problems else 0

The gate is deliberately boring: four comparisons and an exit code. The subtle part is choosing expected_items, because a fixed constant rots the moment the target grows or shrinks. Deriving it from recent runs, and the other detection patterns — schema assertions on write, expected-row-count bands, canary URLs with known-good values — are the subject of Detecting Silent Scraper Failures.

6. Set Thresholds That Do Not Page You at 3am

An alert that fires for something you will not act on is worse than no alert, because it trains everyone to ignore the channel that carries the real one. Sort every condition onto a rung and route each rung differently.

Four practices keep the top rung quiet:

• Alert on rates and ratios, never on counts. "Six errors" means nothing; "6% of responses were 5xx" means something. A count-based threshold breaks the day the crawl doubles in size.
• Require persistence. Use for: 30m in a Prometheus rule, or require two consecutive failed runs, before anything escalates. A single bad run is usually the target having a bad minute.
• Set the threshold from history, not from taste. Look at the last thirty runs, take the median and the spread, and put the line outside the observed range. A threshold picked from intuition either never fires or fires constantly.
• Suppress the derivative alerts. When a run fails outright, its yield alert, null-rate alert and freshness alert will all fire too. Route them so the run-failure alert inhibits the others, or you will get four messages describing one incident.

One more asymmetry is worth internalising. False negatives in scraping are far more expensive than false positives, because a missed collapse silently corrupts a dataset for as long as nobody checks, while a spurious notification costs someone ninety seconds. Bias the notify rung towards sensitivity and the page rung towards specificity.

7. Post a Run Report

Metrics answer questions you thought to ask. A run report answers the question you did not: does this look like yesterday? Post one message per run, containing eight numbers and a verdict, to a channel or a mailbox.

# scraper/report.py
import json
import os
import urllib.request
from scraper.context import RunContext
from scraper.metrics import RunMetrics

def build_report(ctx: RunContext, m: RunMetrics, verdict: str) -> str:
    minutes = max(ctx.elapsed_s() / 60, 1e-9)
    lines = [
        f"*{ctx.spider}* run `{ctx.run_id}` — {verdict}",
        f"pages {m.pages:,} in {ctx.elapsed_s() / 60:.1f} min ({m.pages / minutes:.0f}/min)",
        f"items {m.items:,} ({m.items_per_page():.2f} per page)",
        f"HTTP errors {m.error_rate():.2%} · retries {m.retries} · gave up {m.give_ups}",
        "null-rates: " + ", ".join(f"{k} {m.null_rate(k):.2%}" for k in ("title", "price")),
    ]
    return "\n".join(lines)

def post_report(text: str) -> None:
    webhook = os.environ.get("REPORT_WEBHOOK_URL")
    if not webhook:
        print(text)
        return
    body = json.dumps({"text": text}).encode("utf-8")
    request = urllib.request.Request(
        webhook,
        data=body,
        headers={"Content-Type": "application/json", "User-Agent": "scraper-report/1.0"},
    )
    with urllib.request.urlopen(request, timeout=10) as response:
        response.read()

Note the fallback: with no webhook configured the report prints, so the same code works on a laptop, in CI and in production without branching. That property matters more than it sounds, because monitoring code that only runs in production is monitoring code that has never been tested.

Wire the report into the scheduled job itself. If the crawl runs from a scheduled workflow, the report step runs whether the crawl succeeded or failed — see Scheduling Scrapers with Cron and GitHub Actions for the if: always() pattern that makes that work.

Performance and Scaling Considerations

Instrumentation is cheap until it is not, and the places it becomes expensive are predictable.

Logging is I/O. One JSON line per request at 500 pages a minute is 30,000 lines an hour, which is fine. One line per request at 50,000 pages a minute across twelve workers is not: the logging module serialises through a lock, and stdout in a container is a pipe with a finite buffer. Above roughly a thousand events a second, drop per-request logs to DEBUG, keep them off in production, and rely on counters for the aggregate picture. Sampling also works — log every request that errored plus one percent of the successes.

Counters are free, labels are not. Incrementing a Counter is a few nanoseconds. Creating a new label combination allocates a new time series, and a series is roughly 3 KB of memory in the collector plus storage forever. Counter("pages", labelnames=["url"]) on a million-URL crawl is how you take down a Prometheus server. Bound every label to a set you can name in advance.

Do not compute metrics twice. In an async crawl, resist wrapping every coroutine in a timing decorator. Measure at the boundaries — one timer around the request, one around the parse — and let the difference account for the rest. The pattern for keeping those measurements consistent under concurrency is the same one used for rate limiting, described in Asynchronous Scraping with Asyncio and HTTPX.

An incremental crawl needs a different baseline. If the job only fetches what changed since last time, as in Caching and Incremental Crawling, then pages fetched and items saved both swing by an order of magnitude between runs and neither can be banded directly. Band the ratio of items to pages instead, and monitor cache hit rate as a signal in its own right — a hit rate that jumps to 100% means the conditional-request logic started matching everything, which produces an empty run that looks efficient.

Aggregate before you export. A distributed crawl with fifty workers should not push fifty times a second. Have each worker keep local counters and push once at the end of its shard, or expose them for a single scrape, and let the collector sum. Pushing per-item is the most common cause of a monitoring stack costing more than the crawl.

Retries distort every rate you compute. A crawl with automatic retries fetches more responses than it has URLs, so error_rate computed over responses understates the fraction of URLs that failed. Track both: responses by status for the target's health, and give-ups by URL for yours. If you use the backoff patterns from Retrying Failed Requests with Tenacity, count a retry and a final abandonment as separate events.

Common Errors and Fixes

Alert never fires because the metric disappeared. A Prometheus rule like rate(scraper_errors_total[1h]) > 0.05 evaluates to nothing when the scraper stops running, and "nothing" is not "greater than 0.05". The rule is silently dead exactly when you need it. Fix it with an absent() companion rule:

- alert: ScraperMissing
  expr: absent(scraper_last_success_timestamp) or (time() - scraper_last_success_timestamp) > 90000
  for: 15m
  labels: { severity: page }
  annotations:
    summary: "No successful crawl in over 25 hours"

ValueError: Duplicated timeseries in CollectorRegistry. Raised when the same metric is registered twice — typically because a module-level Counter(...) is imported by a worker that reloads, or because a test suite imports the module more than once. Create a fresh CollectorRegistry() per run and pass it explicitly, as in the push_run example above, rather than relying on the default global registry.

Logs vanish in the container but appear locally. Python buffers stdout when it is not a TTY, so up to 8 KB of log lines sit in the buffer when the process is killed. Set PYTHONUNBUFFERED=1 in the image, or ENV PYTHONUNBUFFERED=1 in the Dockerfile. This is also why a crawl that is SIGKILLed appears to have stopped logging minutes before it actually died.

The gate fails every run after a legitimate site change. A target that genuinely shrank from 5,000 products to 2,000 will trip a yield floor forever. The fix is not to raise the tolerance — it is to derive the expectation from a rolling window so it adapts within a few runs, and to make the tolerance asymmetric: a 60% drop is an incident, a 60% rise is a note.

push_to_gateway raises urllib.error.URLError and kills the run. Monitoring must never be able to fail the thing it monitors. Wrap every export call in try/except Exception, log the failure as a warning, and continue. The one exception is the gate in step 5, which is supposed to fail the run.

Every field shows a 100% null-rate but items were saved. Almost always a shape mismatch: record_item was passed a Scrapy Item or a Pydantic model rather than a dict, so item.get(name) returns None for everything. Normalise to a plain dict at the metrics boundary, or validate first — see Cleaning and Validating Scraped Data.

The duration metric is negative or absurdly large. Using time.time() for elapsed measurement means an NTP correction during the run lands directly in the number, and a container whose clock jumps at start-up can produce a run that apparently took minus four seconds. time.monotonic() is immune to clock adjustments and is the only correct choice for a duration; keep wall-clock time for timestamps you intend to display.

Alerts fire correctly but nobody can tell which spider broke. This happens when metrics carry no identifying label and the log lines carry no run identifier, so the alert says "yield collapsed" and answering "for what" means opening a database. Label every metric with the spider name from the first line of instrumentation, and make sure that label matches the spider field in the logs so a fired alert can be pivoted straight into a log query.

Frequently Asked Questions

What is the single most valuable thing to monitor on a scraper? Items per page, compared against recent runs. It is the one number that moves when a selector breaks, when a site adds a bot wall that serves a valid-looking empty page, and when pagination silently stops after page one. Error rate and run duration both stay green through all three.

How do I monitor a scraper that only runs once a day? Push the metrics rather than exposing them, and lean on a last-success timestamp gauge so the absence of a run is itself an alertable condition. A daily crawl has a monitoring window measured in days, so add a run report as well — with one data point per day, a human reading eight numbers catches drift that a threshold would take a week to see.

Should the scraper alert, or should the monitoring system? Both, at different levels. The scraper should assert its own output and exit non-zero, because it is the only thing that knows what a good run looks like for that target. The monitoring system should alert on trends across runs, absence, and anything that spans more than one job. Putting cross-run logic inside the scraper means it cannot detect its own failure to start.

Will logging every request slow the crawl down? Not at ordinary scraper volumes. A JSON log line costs roughly 10 to 30 microseconds, which is invisible next to a 200-millisecond HTTP round trip. It becomes measurable above about a thousand requests a second in one process, at which point move per-request detail to DEBUG and keep counters.

How do I avoid alert fatigue with many spiders? Alert on the aggregate first and the individual spider second. One rule on "share of spiders whose last run failed" catches systemic breakage — a proxy pool expiring, a rotated credential — while per-spider rules stay on the digest rung. Label every metric with the spider name so you can drill in after the aggregate fires.

Related

• Scaling and Deploying Python Web Scrapers — the parent section this guide belongs to
• Structured Logging for Python Scrapers — a JSON formatter, correlation ids and what never to log
• Detecting Silent Scraper Failures — null-rate thresholds, yield bands, schema assertions and canary URLs
• Exporting Scrapy Metrics to Prometheus — a stats-collector extension, push targets and alert rules
• Storing and Exporting Scraped Data — the write path your freshness and schema checks sit on