Exporting Scrapy Metrics to Prometheus

Scrapy already counts everything a crawl does; the work is getting those counters out of a process that is about to exit, which is the export half of Monitoring and Alerting for Scrapers.

The shape of the answer is a Scrapy extension that reads crawler.stats when the spider closes and writes the numbers to a Prometheus push target. A short-lived crawl cannot be scraped — it is gone before the next collection tick — so use a Pushgateway, or write a .prom file into a node exporter's textfile directory. Export four things and you cover almost everything: responses by status code, items scraped, error-level log lines, and the finish reason.

What Scrapy Already Records

Every Scrapy component increments a shared StatsCollector, reachable as crawler.stats and dumped at INFO level when a crawl ends. Running any spider prints roughly forty keys. Most are noise for alerting purposes; four carry the diagnostic weight.

downloader/response_status_count/<code> is one key per status code the crawl actually saw. The ratio of 4xx and 5xx to the total is the target's health and your block rate in one number, and a sudden appearance of 403 or 429 is the earliest sign that a proxy pool or fingerprint has been recognised.

item_scraped_count counts items that made it through every pipeline, not items yielded by the spider. That distinction matters: if a validation pipeline drops malformed records, this number falls while the spider looks busy, which is exactly the signal you want.

log_count/ERROR counts calls to logger.error across the whole process, including Scrapy's own. It rises when a callback raises, when a pipeline fails to write, and when the retry middleware finally gives up on a URL. It does not rise for individual retries, which makes it a decent proxy for lost work rather than for transient noise.

finish_reason is a string — finished, closespider_timeout, closespider_itemcount, shutdown, cancelled. It is the difference between "the crawl completed" and "the crawl was cut off partway through with a plausible-looking partial dataset", and it is the field most often left unexported.

Useful supporting keys include downloader/request_count, downloader/response_bytes, retry/count, retry/max_reached, dupefilter/filtered, elapsed_time_seconds, and memusage/max. Everything else can wait until you have a question it answers.

Two of those supporting keys pull more weight than their obscurity suggests. retry/max_reached counts URLs abandoned after the retry middleware exhausted its attempts, which is the true count of lost work — retry/count counts individual attempts and will happily sit in the hundreds during a healthy crawl of a flaky target. And dupefilter/filtered counts requests the scheduler discarded as already-seen; a sudden jump usually means a pagination link started pointing back at itself, which caps the crawl at one page while every other statistic looks entirely normal.

You can see the full set for any spider without writing code by reading the Dumping Scrapy stats block at the end of a run, or by opening crawler.stats.get_stats() in a scrapy shell session. Doing that once against a healthy crawl is the fastest way to decide which keys are worth exporting for your particular target, because the interesting ones vary: a crawl behind a proxy pool cares about status codes, a crawl through a validation-heavy pipeline cares about the gap between items yielded and items scraped.

Scrape or Push

Prometheus is a pull system: it fetches /metrics from each target on a fixed interval, typically every 15 to 60 seconds. That model assumes the target outlives several intervals. A crawl that runs for 34 seconds and exits inside a 60-second interval is never scraped at all, and a longer crawl that exposes an endpoint still loses its final counter values, because the process dies before the last collection.

Three ways out, in order of how often they are the right answer:

Pushgateway. The crawl POSTs its final counters to a long-lived gateway, which holds them until Prometheus scrapes it. Purpose-built for batch jobs, one small container, and the values persist across crawls so a missing run shows as a stale timestamp rather than as a gap. The caveats are real: values persist forever unless deleted, so a decommissioned spider keeps reporting its last numbers, and counters pushed by successive runs replace rather than accumulate — group them by a grouping_key that identifies the spider, and treat the gauges as last-value.

Node exporter textfile collector. The crawl writes a .prom file into a directory a node exporter is already watching, and the exporter serves it alongside host metrics. No extra network service and no extra failure mode, but it only works when the crawl and the exporter share a filesystem — which rules out most serverless and many container setups. Write to a temporary file and os.replace() it so the exporter never reads a half-written file.

A long-lived exporter process. If your crawls are continuous rather than scheduled — a worker pulling from a queue for hours — then the ordinary pull model applies. Call start_http_server(8000) once and let Prometheus scrape it. Do not do this for scheduled jobs; it adds a port and a race condition without solving the fundamental problem.

The Extension

A Scrapy extension is any class with a from_crawler classmethod that connects to signals. This one reads the stats dict once, at spider_closed, and pushes it. Add it to settings.py with EXTENSIONS = {"myproject.extensions.PrometheusStatsExtension": 500}.

# myproject/extensions.py
import logging
import os
from typing import Any

from prometheus_client import CollectorRegistry, Counter, Gauge, push_to_gateway
from scrapy import signals
from scrapy.crawler import Crawler
from scrapy.exceptions import NotConfigured

logger = logging.getLogger(__name__)

FINISH_REASONS = ("finished", "closespider_timeout", "closespider_itemcount", "shutdown", "cancelled")

class PrometheusStatsExtension:
    """Push Scrapy's stats to a Pushgateway when the spider closes."""

    def __init__(self, gateway: str, job: str) -> None:
        self.gateway = gateway
        self.job = job

    @classmethod
    def from_crawler(cls, crawler: Crawler) -> "PrometheusStatsExtension":
        gateway = crawler.settings.get("PROMETHEUS_PUSHGATEWAY")
        if not gateway:
            raise NotConfigured("PROMETHEUS_PUSHGATEWAY is not set")
        extension = cls(gateway, crawler.settings.get("PROMETHEUS_JOB", "scrapy"))
        crawler.signals.connect(extension.spider_closed, signal=signals.spider_closed)
        return extension

    def spider_closed(self, spider: Any, reason: str) -> None:
        stats: dict[str, Any] = spider.crawler.stats.get_stats()
        registry = CollectorRegistry()

        responses = Counter(
            "scrapy_responses_total", "Responses by HTTP status code",
            ["status"], registry=registry,
        )
        for key, value in stats.items():
            if key.startswith("downloader/response_status_count/"):
                responses.labels(status=key.rsplit("/", 1)[1]).inc(value)

        Counter("scrapy_items_total", "Items that cleared every pipeline",
                registry=registry).inc(stats.get("item_scraped_count", 0))
        Counter("scrapy_log_errors_total", "ERROR-level log records",
                registry=registry).inc(stats.get("log_count/ERROR", 0))
        Counter("scrapy_requests_total", "Requests issued by the downloader",
                registry=registry).inc(stats.get("downloader/request_count", 0))
        Counter("scrapy_retries_total", "Requests retried",
                registry=registry).inc(stats.get("retry/count", 0))

        Gauge("scrapy_duration_seconds", "Wall-clock crawl duration",
              registry=registry).set(stats.get("elapsed_time_seconds", 0.0))
        Gauge("scrapy_last_finish_timestamp", "Unix time the crawl ended",
              registry=registry).set_to_current_time()

        finish = Gauge("scrapy_finish_reason", "1 for the reason this crawl ended",
                       ["reason"], registry=registry)
        for candidate in FINISH_REASONS:
            finish.labels(reason=candidate).set(1 if candidate == reason else 0)

        try:
            push_to_gateway(
                self.gateway,
                job=self.job,
                grouping_key={"spider": spider.name, "instance": os.environ.get("HOSTNAME", "local")},
                registry=registry,
                timeout=10,
            )
        except Exception:  # never let monitoring fail the crawl
            logger.warning("prometheus push failed", exc_info=True)

Three details are load-bearing. A fresh CollectorRegistry() per crawl avoids Duplicated timeseries in CollectorRegistry, which is what you get from the default global registry when a process runs more than one spider. The grouping_key includes the spider name so two spiders do not overwrite each other's series in the gateway. And the push is wrapped in a bare except Exception — an unreachable gateway must never turn a successful crawl into a failed one.

The textfile variant replaces the push_to_gateway call with an atomic write:

# myproject/textfile.py
import os
from prometheus_client import CollectorRegistry, write_to_textfile

def dump(registry: CollectorRegistry, directory: str, spider: str) -> None:
    path = os.path.join(directory, f"scrapy_{spider}.prom")
    tmp = f"{path}.tmp"
    write_to_textfile(tmp, registry)
    os.replace(tmp, path)  # atomic: the exporter never sees a partial file

Alert Rules That Fire

Metrics without rules are a dashboard nobody opens. These four cover the failure modes that matter, and each is written against the metric names above.

groups:
  - name: scrapy
    rules:
      - alert: ScrapyCrawlMissing
        expr: absent(scrapy_last_finish_timestamp) or (time() - scrapy_last_finish_timestamp) > 93600
        for: 15m
        labels: { severity: page }
        annotations:
          summary: "No crawl has finished in over 26 hours"

      - alert: ScrapyYieldCollapsed
        expr: scrapy_items_total < 0.5 * avg_over_time(scrapy_items_total[7d])
        for: 10m
        labels: { severity: notify }
        annotations:
          summary: "{{ $labels.spider }} produced under half its weekly average"

      - alert: ScrapyBlockRateHigh
        expr: |
          sum by (spider) (scrapy_responses_total{status=~"403|429"})
            / sum by (spider) (scrapy_responses_total) > 0.05
        for: 30m
        labels: { severity: notify }
        annotations:
          summary: "{{ $labels.spider }} is seeing over 5% 403/429 responses"

      - alert: ScrapyCrawlTruncated
        expr: scrapy_finish_reason{reason!="finished"} == 1
        for: 5m
        labels: { severity: digest }
        annotations:
          summary: "{{ $labels.spider }} ended as {{ $labels.reason }}"

ScrapyCrawlMissing is the one to build first. Every other rule evaluates to nothing when the crawl stops running entirely, because there is no series to compare — absent() is what turns silence into a signal. The 26-hour threshold on a daily crawl gives two hours of slack for a late start without letting a genuinely missed day pass.

ScrapyBlockRateHigh is a ratio rather than a count, so it survives the crawl changing size. Sustaining it for: 30m avoids firing on the brief 429 burst that any polite crawler occasionally provokes; if it stays above 5% for half an hour, the proxy or fingerprint side of the setup needs attention.

Edge Cases and Caveats

• Pushed values never expire. Delete a spider's group when you retire it — curl -X DELETE http://gateway:9091/metrics/job/scrapy/spider/books — or its stale numbers will keep satisfying alert rules that check for presence.
• Counters reset every crawl. rate() and increase() assume a monotonically increasing series within one process lifetime. Pushed per-run counters are effectively last-value gauges, so compare them with avg_over_time and direct comparisons rather than rate().
• Never label by URL. Counter(..., ["url"]) creates one series per page and will exhaust a Prometheus server's memory on any real crawl. Bound every label to a small enumerable set: status code, spider name, finish reason.
• elapsed_time_seconds is not in older Scrapy. It was added in Scrapy 2.5; before that, subtract start_time from finish_time in the stats dict. Both are timezone-aware datetime objects.
• spider_closed does not fire on SIGKILL. A container killed for exceeding its memory limit exports nothing at all, which is the case ScrapyCrawlMissing exists to catch. It does fire on SIGTERM with reason="shutdown".
• Multiple spiders in one process share the default registry. Running spiders sequentially under one CrawlerProcess will raise on the second registration unless each crawl builds its own CollectorRegistry, as the extension above does.
• item_scraped_count is absent, not zero, when nothing was scraped. stats.get("item_scraped_count", 0) supplies the default; reading the key directly raises KeyError on exactly the run you most need to measure.
• The gateway is a single point of failure you can tolerate. Because the push is wrapped in a try block, a gateway outage loses metrics but not crawls. Pair it with the run-report pattern so a human still sees the numbers when the metrics path is down.

Frequently Asked Questions

Do I need a Pushgateway, or can Prometheus scrape the spider directly? Scraping works only if the crawl reliably outlives two scrape intervals and you accept losing the final values. For anything scheduled — nightly, hourly, per-commit — push. For a continuously running worker consuming from a queue, start_http_server and ordinary scraping is simpler and better.

Should I use scrapy-prometheus-exporter instead of writing this? Third-party extensions save an hour and cost you control over metric names and cardinality, and they lag Scrapy releases. The extension above is about sixty lines and exports exactly the four things worth alerting on, which is usually the better trade for production.

How do I get the same metrics out of a plain requests or HTTPX scraper? Keep your own counters and push them the same way — there is nothing Scrapy-specific about the export path, only about where the numbers come from. Count responses by status, items stored and give-ups yourself, then reuse the registry and push call unchanged.

Why is my yield alert never firing even though the crawl is broken? Almost always because the series stopped being pushed and the expression has nothing to evaluate. A comparison against a missing series is not false, it is empty, and an empty result never fires. Add the absent() rule alongside every threshold rule.

Related

• Monitoring and Alerting for Scrapers — the topic this page belongs to
• Detecting Silent Scraper Failures — the in-process checks these alert rules mirror
• Web Scraping with Scrapy — the framework whose stats collector this reads
• Writing Scrapy Item Pipelines — where item_scraped_count is decided