Scaling Python Web Scrapers

A script that fetches one page and a system that fetches a million are not the same program with a different loop bound. Somewhere between those two numbers the constraints change: the runtime stops being dominated by parsing and starts being dominated by waiting, the process stops fitting in memory, a crash stops being an inconvenience and starts costing six hours of work, and "did the run succeed?" stops having an obvious answer. This path is about that transition — the engineering layer that turns extraction code into a pipeline you can leave running.

It assumes you can already write the extraction itself, as covered in The Complete Guide to Python Web Scraping. If your problem is that the target blocks you rather than that the crawl is slow, the sibling path on Advanced Scraping Techniques and Anti-Bot Evasion is the one to read first — scaling a scraper that gets 403s simply produces 403s faster.

When One Script Stops Being Enough

Growth in a crawl does not degrade performance smoothly; it retires one design and forces the next. Recognising which wall you have hit saves you from applying the wrong fix, which is usually "add more concurrency" to a scraper whose actual problem is that it cannot resume.

At a few hundred URLs, a serial loop with a one-second delay finishes over a coffee and needs nothing else. At a few thousand, the wall-clock time becomes hours and rate limits start appearing, so you need backoff and a delay policy. At tens of thousands, two things break at once: the list of results no longer fits comfortably in memory, and a crash at 80% completion costs the whole run because there is no checkpoint. Beyond that, a single host is the ceiling — one egress IP, one process, one failure domain — and the work has to be distributed.

Three capabilities separate a script from a pipeline. Concurrency, so the process is not idle while a server thinks. Durability, so a transient failure retries and a crash resumes. Observability, so you learn that a run collected zero rows from a change in the target's markup, rather than discovering it a week later in a report.

import json
import pathlib

STATE = pathlib.Path("crawl_state.json")

def load_done() -> set[str]:
    if STATE.exists():
        return set(json.loads(STATE.read_text(encoding="utf-8")))
    return set()

def mark_done(done: set[str], url: str) -> None:
    done.add(url)
    STATE.write_text(json.dumps(sorted(done)), encoding="utf-8")

def crawl(urls: list[str]) -> None:
    done = load_done()
    remaining = [u for u in urls if u not in done]
    print(f"{len(done)} already done, {len(remaining)} to go")
    for url in remaining:
        # fetch and store here; the point is that progress survives a crash
        mark_done(done, url)

crawl([f"https://books.toscrape.com/catalogue/page-{n}.html" for n in range(1, 6)])

That is the cheapest possible checkpoint, and it is the difference between a crash costing six hours and costing one URL. Run it twice: the second run reports that everything is already done.

Concurrency with asyncio and HTTPX

Scraping is an I/O-bound workload. In a typical serial loop the process spends over 95% of its time blocked on a socket, which means the CPU is idle almost the entire run. asyncio reclaims that time by keeping many requests in flight from a single thread, and HTTPX provides the async HTTP client to do it with.

The gain is large and then it stops. Going from serial to twenty concurrent requests is roughly a twentyfold improvement; going from twenty to two hundred against the same origin usually improves nothing, because you are now bounded by the server rather than by your client — and it will get you rate-limited or blocked. Concurrency must be capped, and the cap belongs to the target, not to your machine. An asyncio.Semaphore is the standard mechanism, covered in Limiting Concurrency with Semaphores, with the wider pattern in Asynchronous Scraping with Asyncio and HTTPX.

# pip install "httpx==0.27.2"
import asyncio
import httpx

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
}

async def fetch(client: httpx.AsyncClient, url: str, gate: asyncio.Semaphore) -> tuple[str, int]:
    async with gate:                                  # never more than N in flight
        try:
            response = await client.get(url, timeout=15.0)
            return url, response.status_code
        except httpx.HTTPError as exc:
            return url, -1 if not isinstance(exc, httpx.HTTPStatusError) else 0

async def crawl(urls: list[str], concurrency: int = 8) -> list[tuple[str, int]]:
    gate = asyncio.Semaphore(concurrency)
    limits = httpx.Limits(max_connections=concurrency, max_keepalive_connections=concurrency)
    async with httpx.AsyncClient(headers=HEADERS, limits=limits, follow_redirects=True) as client:
        return await asyncio.gather(*(fetch(client, url, gate) for url in urls))

pages = [f"https://books.toscrape.com/catalogue/page-{n}.html" for n in range(1, 21)]
for url, status in asyncio.run(crawl(pages)):
    print(status, url.rsplit("/", 1)[-1])

Two details make this production-shaped rather than a demo. The semaphore and the client's connection limits are set to the same number, so the pool is never the accidental bottleneck. And fetch returns a status instead of raising, so one dead URL cannot abort the whole gather — the default behaviour of asyncio.gather is to propagate the first exception and discard the rest of the results.

When the Bottleneck Moves from the Network to the CPU

Concurrency solves waiting. It does nothing for work. Once twenty requests are in flight, a fraction of crawls discover that the event loop is now saturated by parsing — a 400 KB page through BeautifulSoup with html.parser can take 200 ms of pure CPU, and twenty of those per second is more than one core has to give. The symptom is distinctive: adding concurrency stops improving throughput while CPU sits at 100% and the target's response times are unchanged.

Three fixes apply, in increasing order of effort. Switch the parser backend to lxml, which is typically 3–5× faster than html.parser on real documents for a one-word change. Parse less by scoping the parse to the fragment you need rather than building a tree for the whole document. And when neither is enough, move parsing off the event loop into a process pool, because asyncio gives you no parallelism for CPU work — a long parse blocks every other coroutine in the loop, including the ones waiting on sockets.

import asyncio
from concurrent.futures import ProcessPoolExecutor

from bs4 import BeautifulSoup

def parse_titles(html: str) -> list[str]:
    """Pure CPU work: runs in a worker process, not on the event loop."""
    soup = BeautifulSoup(html, "lxml")
    return [a.get("title", "") for a in soup.select("article.product_pod h3 a")]

async def parse_many(pages: list[str], workers: int = 4) -> list[str]:
    loop = asyncio.get_running_loop()
    with ProcessPoolExecutor(max_workers=workers) as pool:
        results = await asyncio.gather(
            *(loop.run_in_executor(pool, parse_titles, page) for page in pages)
        )
    return [title for group in results for title in group]

sample = "<article class='product_pod'><h3><a title='Example Book'></a></h3></article>"
print(asyncio.run(parse_many([sample] * 8)))

run_in_executor with a ProcessPoolExecutor sidesteps the global interpreter lock, so parsing genuinely runs on multiple cores while the loop stays free to keep fetching. Note the cost it introduces: the HTML is pickled and shipped to the worker, so this only pays off when parsing takes appreciably longer than serialising the document — measure before adopting it.

Adopting Scrapy Instead of Rebuilding It

There is a recognisable moment in every hand-rolled crawler's life where it grows a scheduler, a duplicate-URL filter, a retry policy, a per-domain throttle, and a chain of post-processing steps. That is Scrapy, rewritten badly. Scrapy provides all of it — an async engine, request scheduling and deduplication, AutoThrottle, middleware hooks, and item pipelines — and asks you to write only the parsing.

Its cost is a project layout and a set of conventions to learn, which is real but front-loaded. Adopt it when the crawl follows links rather than iterating a known list, when it runs repeatedly on a schedule, or when it needs per-domain politeness across many domains at once. The full workflow is in Web Scraping with Scrapy; the boundary against lighter tools is in Scrapy vs BeautifulSoup: Which to Use and, for rendered applications, Scrapy vs Playwright for Single-Page Apps.

# pip install "Scrapy==2.11.2" && scrapy runspider books_spider.py -o books.jsonl
import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]
    custom_settings = {
        "USER_AGENT": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                      "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
        "AUTOTHROTTLE_ENABLED": True,
        "AUTOTHROTTLE_TARGET_CONCURRENCY": 4.0,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 8,
        "RETRY_TIMES": 3,
        "HTTPCACHE_ENABLED": True,
    }

    def parse(self, response):
        for card in response.css("article.product_pod"):
            yield {
                "title": card.css("h3 a::attr(title)").get(),
                "price": card.css("p.price_color::text").get(),
            }
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

AUTOTHROTTLE_ENABLED is the setting that matters most here: Scrapy measures the target's response latency and adjusts its own concurrency to keep the server comfortable, which is both politer and more robust than a fixed delay you guessed. HTTPCACHE_ENABLED means re-running the spider while you iterate on selectors does not re-request anything. Cleaning and persistence belong in an item pipeline, covered in Writing Scrapy Item Pipelines.

Caching and Incremental Crawling

The fastest request is the one you do not send. Two related ideas cut a recurring crawl down by an order of magnitude and simultaneously make you a better client. Caching stores responses so repeated fetches during development and across runs are served locally. Incremental crawling uses the server's own change signals — ETag and Last-Modified — to ask "has this changed since I last saw it?" and receive a 304-byte 304 Not Modified instead of a full page.

For a daily crawl over a catalogue where 2% of pages change, incremental fetching removes 98% of the bytes and most of the parsing. It also removes almost all of the load you place on the target, which does more for your long-term access than any evasion technique. The full treatment is in Caching and Incremental Crawling, with the drop-in library approach in HTTP Caching with requests-cache, the validator mechanics in Incremental Crawls with ETag and Last-Modified, and memory-efficient URL deduplication in Deduplicating URLs with Bloom Filters.

import json
import pathlib
import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
}
VALIDATORS = pathlib.Path("validators.json")

def load() -> dict[str, dict[str, str]]:
    return json.loads(VALIDATORS.read_text(encoding="utf-8")) if VALIDATORS.exists() else {}

def conditional_get(url: str) -> str | None:
    """Return the body, or None when the server says nothing changed."""
    store = load()
    headers = dict(HEADERS)
    known = store.get(url, {})
    if "etag" in known:
        headers["If-None-Match"] = known["etag"]
    if "last_modified" in known:
        headers["If-Modified-Since"] = known["last_modified"]

    response = requests.get(url, headers=headers, timeout=15)
    if response.status_code == 304:
        return None
    response.raise_for_status()
    entry: dict[str, str] = {}
    if "ETag" in response.headers:
        entry["etag"] = response.headers["ETag"]
    if "Last-Modified" in response.headers:
        entry["last_modified"] = response.headers["Last-Modified"]
    store[url] = entry
    VALIDATORS.write_text(json.dumps(store, indent=2), encoding="utf-8")
    return response.text

url = "https://books.toscrape.com/catalogue/page-1.html"
print("first run:", "changed" if conditional_get(url) else "unchanged")
print("second run:", "changed" if conditional_get(url) else "unchanged")

Whether the second run prints unchanged depends on whether the origin emits validators — many static hosts do, many application servers do not. Checking is a two-line experiment and the payoff on a recurring crawl is enormous.

Storing What You Collect

Extraction produces records; the pipeline has to land them somewhere queryable without holding the entire dataset in RAM. The choice of sink follows from volume and use. JSON Lines is the right default for a crawl in progress — one record per line, appendable, streamable, and readable by everything. CSV is for handing a small result to a spreadsheet. SQLite gives you indexes and UPSERT semantics with no server. PostgreSQL is for concurrent writers and real relational work; Parquet is for columnar analytics over tens of millions of rows.

The property that matters more than the format is incrementality. Append each record as it is produced rather than accumulating a list and writing once at the end, so that a crash costs the last record instead of the whole run. Format trade-offs are in Storing and Exporting Scraped Data, the relational case in Saving Scraped Data to PostgreSQL, and the analytical formats in Exporting Scraped Data to CSV and Parquet. Field-level normalisation before storage belongs to Cleaning and Validating Scraped Data.

import json
import pathlib
import sqlite3
from contextlib import closing

def append_jsonl(record: dict[str, object], path: str = "items.jsonl") -> None:
    with open(path, "a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, ensure_ascii=False) + "\n")

def upsert_sqlite(records: list[dict[str, object]], db: str = "items.db") -> int:
    with closing(sqlite3.connect(db)) as conn:
        conn.execute(
            "CREATE TABLE IF NOT EXISTS books ("
            " url TEXT PRIMARY KEY, title TEXT NOT NULL, price REAL, seen_at TEXT)"
        )
        conn.executemany(
            "INSERT INTO books (url, title, price, seen_at) VALUES (?, ?, ?, datetime('now'))"
            " ON CONFLICT(url) DO UPDATE SET"
            " title=excluded.title, price=excluded.price, seen_at=excluded.seen_at",
            [(r["url"], r["title"], r["price"]) for r in records],
        )
        conn.commit()
        return conn.total_changes

rows = [
    {"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
     "title": "A Light in the Attic", "price": 51.77},
]
for row in rows:
    append_jsonl(row)
print("rows written:", upsert_sqlite(rows), "file:", pathlib.Path("items.jsonl").stat().st_size, "bytes")

The ON CONFLICT ... DO UPDATE clause is what makes a repeated crawl idempotent. Without it, running the same job twice produces two copies of every record, and deduplicating after the fact is far more painful than preventing it with a primary key.

Distributing Across Machines

A single process has one egress IP, one memory limit, and one failure domain. Past a few million URLs, or when you need geographic distribution, the work has to be spread across workers that pull from a shared queue. The architecture is a broker holding the URL queue, a set of stateless workers consuming it, and a shared store for results and for the deduplication set.

The design constraint people underestimate is idempotence. Any distributed queue will occasionally deliver the same task twice — a worker dies after processing but before acknowledging — so every task must be safe to run again. That is exactly what the UPSERT above buys you. The task design, fan-out patterns, and coordination are in Distributed Crawling with Celery and Redis, and the broker choice in Celery vs RQ for Scraping Task Queues.

# pip install "celery[redis]==5.4.0"   # then: celery -A tasks worker --loglevel=info
import requests
from celery import Celery

app = Celery("scraper", broker="redis://localhost:6379/0", backend="redis://localhost:6379/1")
HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
}

@app.task(
    bind=True,
    autoretry_for=(requests.RequestException,),
    retry_backoff=True,
    retry_backoff_max=300,
    retry_jitter=True,
    max_retries=4,
    acks_late=True,               # re-queue if the worker dies mid-task
)
def fetch_page(self, url: str) -> dict[str, object]:
    response = requests.get(url, headers=HEADERS, timeout=20)
    response.raise_for_status()
    return {"url": url, "status": response.status_code, "bytes": len(response.content)}

def enqueue(urls: list[str]) -> None:
    for url in urls:
        fetch_page.delay(url)

acks_late=True together with autoretry_for is the combination that makes a crawl survive worker loss: a task is acknowledged only after it completes, so a killed worker's task returns to the queue instead of vanishing. retry_jitter prevents a synchronised retry storm when a target briefly returns errors to every worker at once.

Knowing Whether the Run Actually Worked

The characteristic production failure of a scraper is not a crash. It is a run that exits zero, logs nothing unusual, and writes 0 rows because the target changed a class name. Every scaled crawl needs to answer three questions automatically: did it finish, did it collect a plausible number of records, and did the error rate change.

The cheapest effective alert is a floor on the record count relative to the previous run. If yesterday produced 48,000 rows and today produced 40, something is broken regardless of the exit code. Structured logs make the rest diagnosable after the fact. The full approach is in Monitoring and Alerting for Scrapers, with log structure in Structured Logging for Python Scrapers, the silent-failure case in Detecting Silent Scraper Failures, and metric export in Exporting Scrapy Metrics to Prometheus.

import json
import logging
import sys
from dataclasses import asdict, dataclass

logging.basicConfig(stream=sys.stdout, level=logging.INFO, format="%(message)s")
log = logging.getLogger("crawl")

@dataclass
class RunStats:
    requested: int = 0
    ok: int = 0
    failed: int = 0
    records: int = 0

def emit(event: str, **fields: object) -> None:
    log.info(json.dumps({"event": event, **fields}))

def check_run(stats: RunStats, previous_records: int, tolerance: float = 0.5) -> bool:
    emit("run_finished", **asdict(stats))
    if stats.requested and stats.failed / stats.requested > 0.2:
        emit("alert", reason="error_rate", value=stats.failed / stats.requested)
        return False
    if previous_records and stats.records < previous_records * tolerance:
        emit("alert", reason="record_count_drop", got=stats.records, expected=previous_records)
        return False
    return True

stats = RunStats(requested=1000, ok=996, failed=4, records=40)
print("healthy:", check_run(stats, previous_records=48000))

That prints False and emits a record_count_drop alert, which is exactly the failure a green exit code would have hidden. JSON-per-line output means the same logs are greppable by a human and parseable by a log aggregator without a custom format.

Deploying and Scheduling the Crawl

Eventually the job leaves your laptop. The three realistic homes are a container on a small always-on machine, a serverless function invoked on a schedule, and a scheduled runner such as GitHub Actions. Each trades control against operational cost: a container gives you a stable egress IP and unlimited runtime but you maintain it; a function scales to zero but caps execution time and gives you an unpredictable, frequently-flagged IP; a scheduled runner is free for light jobs and unsuitable for anything that must run every few minutes.

The concerns that follow the crawl wherever it goes are secrets (never in the image), egress IP (fixed or rotating, and whether the target has already flagged your provider's range), and timeouts. Options are compared in Deploying Scrapers to the Cloud, with the serverless specifics in Running Scrapers on AWS Lambda and the scheduling patterns in Scheduling Scrapers with Cron and GitHub Actions.

# .github/workflows/crawl.yml — a scheduled crawl with artifacts and failure surfacing
name: nightly-crawl
on:
  schedule:
    - cron: "17 3 * * *"      # 03:17 UTC; avoid :00, everyone schedules there
  workflow_dispatch:
jobs:
  crawl:
    runs-on: ubuntu-latest
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt
      - run: python -m crawler.run --out items.jsonl
        env:
          PROXY_URL: ${{ secrets.PROXY_URL }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: items
          path: items.jsonl

if: always() on the upload step is the detail worth copying: when a run fails halfway you still get the partial output, which is usually enough to see what the target changed.

Common Pitfalls

These are the failures that show up specifically at scale — none of them appear when you are testing against twenty URLs on a laptop, and all of them are cheaper to design against than to retrofit at three in the morning.

• Unbounded concurrency. Firing every request at once gets your IP blocked and can degrade the target. Cap it with a semaphore or Scrapy's AutoThrottle, and set the cap from the target's observed behaviour rather than from your core count.
• Accumulating results in a list. A million records in RAM is a memory error at 3 a.m. Append to disk or a database as each record is produced.
• No resume point. A six-hour crawl with no checkpoint means a single transient failure costs six hours. Persist the completed-URL set, even if only to a JSON file.
• Non-idempotent tasks. Distributed queues deliver at least once. If running a task twice produces two rows, your dataset will silently double. Use a primary key and an upsert.
• Treating exit code zero as success. The most common production failure writes nothing and exits cleanly. Assert on a minimum record count.
• Re-fetching unchanged pages every night. Conditional requests and a response cache remove most of the traffic on a recurring crawl at almost no implementation cost.
• Rebuilding Scrapy by hand. Once you have written a scheduler, a dedupe filter, a retry policy, and a pipeline chain, you have paid Scrapy's learning cost twice over with none of the testing.
• Scaling before fixing the block rate. Twenty workers producing 403s produce twenty times the 403s, and they burn through a proxy pool while doing it. Resolve detection first, then add throughput.
• Logging in prose instead of structure. A message such as failed to fetch page 12 cannot be aggregated, counted, or alerted on. Emit one JSON object per event with stable field names and let the aggregator do the analysis.

Frequently Asked Questions

Is asyncio faster than threads for scraping? For I/O-bound work, asyncio sustains far more concurrent requests per process with much lower memory overhead, because a coroutine costs a few kilobytes where a thread costs a megabyte of stack. Threads remain the pragmatic choice when you must call a synchronous library that has no async equivalent, and a ThreadPoolExecutor of 20–50 workers is perfectly adequate for moderate volumes.

How many concurrent requests should I use? Set it from the target, not from your hardware. Start at 5–10 per domain, watch for 429 and 503 responses and for rising latency, and increase only while both stay flat. Rising response times under increasing concurrency mean you are the load, and pushing further gets you blocked rather than finishing sooner.

When is Scrapy worth the learning curve? When the crawl follows links rather than iterating a known list, when it runs on a schedule for months, or when you need per-domain throttling across many domains. For a one-off extraction of a few hundred known URLs it is overhead. The moment you start writing your own retry queue and duplicate filter, you have crossed the line.

What format should I store scraped data in? JSON Lines while crawling, because it is appendable and streamable and survives a crash mid-write. Then load into SQLite or PostgreSQL for querying, or Parquet for columnar analytics over large volumes. CSV is only for handing a small result to someone who will open it in a spreadsheet — it has no types and no reliable escaping story for scraped text.

Do I need Celery, or is asyncio enough?asyncio scales one process; Celery scales across machines. If a single host can finish the crawl within your time budget and one egress IP is acceptable, stay with asyncio — a broker adds real operational surface. Move to a queue when you need multiple IPs, more throughput than one host provides, or resilience to losing a machine mid-run.

How do I know a scheduled crawl is still working? Compare each run's record count against the previous run and alert on a large drop, track the ratio of failed to total requests, and alert when a scheduled run does not report at all. Those three checks catch the overwhelming majority of real failures, including the silent ones where the code is fine and the target changed.

Related

• Asynchronous Scraping with Asyncio and HTTPX — concurrency, semaphores, and retry policy
• Web Scraping with Scrapy — spiders, settings, middleware, and pipelines
• Caching and Incremental Crawling — conditional requests, response caches, and URL dedupe
• Storing and Exporting Scraped Data — formats, schemas, and incremental writes
• Distributed Crawling with Celery and Redis — brokers, task design, and idempotence
• Monitoring and Alerting for Scrapers — structured logs, metrics, and silent-failure detection
• Deploying Scrapers to the Cloud — containers, functions, schedulers, and secrets
• The Complete Guide to Python Web Scraping — the extraction fundamentals this path builds on
• Advanced Scraping Techniques and Anti-Bot Evasion — fix detection before adding throughput
• Data Extraction Patterns and APIs — cleaner sources that make a large crawl cheaper