Web Scraping with Scrapy

Scrapy is the only widely used Python tool that treats crawling — rather than fetching — as the primary problem, and this guide, part of Scaling Python Web Scrapers, covers building a spider that survives contact with a real site. The scope is the framework itself: how a request travels through the engine, which settings actually govern throughput, and where to put logic so it stays testable.

The distinction that matters is between a library and a framework. A requests script gives you a response and leaves the frontier, retry policy, deduplication, concurrency limits, and output serialisation to you. Scrapy already has all of those, wired together by an asynchronous engine built on Twisted, and asks you to fill in two things: which URLs to start from and what to extract. Every other behaviour is a setting or a middleware hook. That is a good trade above roughly a hundred pages and a poor one below ten.

When to Use Scrapy

The honest threshold is around a few hundred pages, or any crawl you will run more than twice. Below that, the project scaffolding, the settings file and the pipeline indirection are overhead you pay without collecting. Above it, you will otherwise re-implement a request queue, a seen-URL set, a retry policy with backoff, and a concurrency limiter — badly, because they are not the part you were interested in. The direct comparison for the small end is in Scrapy vs BeautifulSoup: Which to Use.

The one case where Scrapy is genuinely the wrong tool is a site whose content only exists after JavaScript runs. Scrapy fetches bytes; it does not execute scripts. That is a solvable problem with a browser integration, but the trade-offs are real and covered in Scrapy vs Playwright for Single-Page Apps.

Prerequisites

Scrapy 2.11 and later require Python 3.8 or newer; use 3.11 or 3.12 for the best Twisted performance. Install into a virtual environment, because Scrapy pins several transitive dependencies that you do not want colliding with a system Python.

python3 -m venv .venv
source .venv/bin/activate
pip install "scrapy==2.12.0" "itemadapter==0.9.0" "itemloaders==1.3.2"
scrapy version

On Linux, lxml and cryptography normally install as wheels with no compiler needed. If pip falls back to building from source, install the headers first:

sudo apt-get install -y python3-dev libxml2-dev libxslt1-dev libssl-dev

Step-by-Step: Building a Production Spider

1. Scaffold the Project

scrapy startproject bookstore
cd bookstore

That produces a fixed layout, and the layout is the framework's opinion about where code belongs:

bookstore/
├── scrapy.cfg
└── bookstore/
    ├── items.py          # the record schema
    ├── middlewares.py    # request/response interception
    ├── pipelines.py      # what happens to each scraped item
    ├── settings.py       # concurrency, delays, enabled components
    └── spiders/          # one file per crawl target

Resisting the urge to collapse this into one file pays off the first time you need to change where data is written without touching parsing, or add a proxy without touching either.

2. Write the Spider

A spider names itself, declares start URLs, and implements a callback that turns a response into items, new requests, or both. The parse method is a generator: whatever it yields goes back into the engine.

# bookstore/spiders/books.py
import scrapy
from scrapy.http import Response


class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]

    custom_settings = {
        "DOWNLOAD_DELAY": 0.25,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 8,
    }

    def parse(self, response: Response):
        for card in response.css("article.product_pod"):
            detail_url = card.css("h3 a::attr(href)").get()
            yield response.follow(
                detail_url,
                callback=self.parse_book,
                cb_kwargs={"list_price": card.css("p.price_color::text").get()},
            )

        next_page = response.css("li.next a::attr(href)").get()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)

    def parse_book(self, response: Response, list_price: str):
        yield {
            "url": response.url,
            "title": response.css("div.product_main h1::text").get(),
            "list_price": list_price,
            "availability": " ".join(
                response.css("p.availability::text").getall()
            ).strip(),
            "upc": response.xpath(
                "//th[text()='UPC']/following-sibling::td/text()"
            ).get(),
        }

Three details are doing real work here. allowed_domains stops an errant relative link from sending the crawl onto a different site. response.follow resolves relative URLs against the current page, so ../../book_1/index.html becomes an absolute URL without any urljoin calls. And cb_kwargs carries data from the listing page into the detail page's callback, which is how you join fields that live on two different pages without global state.

The //th[text()='UPC']/following-sibling::td/text() expression is XPath because CSS cannot select an element by its text content or reach sideways to a sibling — see Selecting Elements with XPath and CSS Selectors for when each language is the right one.

Run it and export in one command:

scrapy crawl books -O books.jsonl

Use -O (overwrite) rather than -o (append) during development, or a re-run silently doubles the file.

3. Understand Where Your Code Sits in the Request Lifecycle

Debugging Scrapy is mostly a matter of knowing which layer a problem lives in. A request leaves the spider, is deduplicated and queued by the scheduler, passes through every downloader middleware on the way out, is fetched, then travels back through those middlewares, through the spider middlewares, and into your callback.

That ordering explains behaviour that otherwise looks arbitrary. Retries happen in a downloader middleware, so a retried request never reaches your callback and never counts as a parse failure. robots.txt is checked in a downloader middleware, which is why a blocked URL produces a log line but no response object. And the dupefilter runs in the scheduler before the download, which is why yielding the same URL twice silently drops the second one unless you pass dont_filter=True.

To inspect any of it interactively, use the shell — it gives you a real response object with the project's settings and middlewares applied:

scrapy shell "https://books.toscrape.com/catalogue/page-1.html"

4. Define Items and a Pipeline

Yielding raw dictionaries is fine for a prototype and a liability in production, because a typo in a key creates a new field instead of an error. An Item declares the schema; a pipeline is where every record goes on its way out.

# bookstore/items.py
import scrapy


class BookItem(scrapy.Item):
    url = scrapy.Field()
    title = scrapy.Field()
    list_price = scrapy.Field()
    availability = scrapy.Field()
    upc = scrapy.Field()
    price_gbp = scrapy.Field()

# bookstore/pipelines.py
import re

from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem

PRICE_RE = re.compile(r"(\d+(?:\.\d{1,2})?)")


class NormalisePricePipeline:
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        raw = adapter.get("list_price") or ""
        match = PRICE_RE.search(raw)
        if match is None:
            raise DropItem(f"unparseable price {raw!r} at {adapter.get('url')}")
        adapter["price_gbp"] = float(match.group(1))
        return item


class DropDuplicateUpcPipeline:
    def __init__(self) -> None:
        self.seen: set[str] = set()

    def process_item(self, item, spider):
        upc = ItemAdapter(item).get("upc")
        if upc in self.seen:
            raise DropItem(f"duplicate UPC {upc}")
        self.seen.add(upc)
        return item

# bookstore/settings.py (excerpt)
ITEM_PIPELINES = {
    "bookstore.pipelines.NormalisePricePipeline": 300,
    "bookstore.pipelines.DropDuplicateUpcPipeline": 400,
}

The integers are ordering, low to high, not priority levels — normalise before deduplicating, because the duplicate check should run on cleaned values. Raising DropItem removes the record and increments a stat you can assert on later. Pipelines that open database connections, batch writes, or talk to object storage need the open_spider and close_spider hooks as well; those patterns are the subject of Writing Scrapy Item Pipelines, and the field-level rules for what counts as valid are covered in Cleaning and Validating Scraped Data.

5. Tune the Settings That Actually Govern Throughput

Four settings decide almost everything about how fast a Scrapy crawl runs and how likely it is to get blocked. The rest are noise by comparison.

# bookstore/settings.py
BOT_NAME = "bookstore"
SPIDER_MODULES = ["bookstore.spiders"]
NEWSPIDER_MODULE = "bookstore.spiders"

ROBOTSTXT_OBEY = True

CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_DELAY = 0.25
DOWNLOAD_TIMEOUT = 30

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0

RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]

HTTPCACHE_ENABLED = False
FEED_EXPORT_ENCODING = "utf-8"
REQUEST_FINGERPRINTER_IMPLEMENTATION = "2.7"

CONCURRENT_REQUESTS is the global in-flight ceiling; CONCURRENT_REQUESTS_PER_DOMAIN is what a single site actually feels, and it is the one to lower when a target starts returning 429. DOWNLOAD_DELAY is not a fixed pause — Scrapy randomises it between 0.5× and 1.5× the value to avoid a machine-perfect request cadence.

AutoThrottle is worth enabling on any unfamiliar site. It measures the latency of each response and adjusts the delay to keep roughly AUTOTHROTTLE_TARGET_CONCURRENCY requests in flight, so the crawl automatically slows down when the server is struggling and speeds up when it is idle. The cost is that your measured throughput becomes a function of the target's health rather than your settings, which makes benchmarking harder but production far safer.

6. Make the Crawl Resumable

A crawl that dies at hour three and restarts from zero is not a production crawl. JOBDIR persists the scheduler queue and the dupefilter to disk, so a stopped job resumes where it left off.

scrapy crawl books -s JOBDIR=jobs/books-2026-08 -O books.jsonl

Stop it with a single Ctrl-C, which triggers a graceful shutdown that flushes state; a second Ctrl-C kills the process immediately and the job directory will be inconsistent. To run the spider from Python rather than the CLI — which is what you need when a scheduler or a task queue invokes it — use CrawlerProcess:

# run_crawl.py
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

from bookstore.spiders.books import BooksSpider


def main() -> None:
    settings = get_project_settings()
    settings.set("JOBDIR", "jobs/books-scheduled")
    settings.set("FEEDS", {"books.jsonl": {"format": "jsonlines", "overwrite": True}})
    process = CrawlerProcess(settings)
    process.crawl(BooksSpider)
    process.start()


if __name__ == "__main__":
    main()

Wiring that into a timetable is covered in Scheduling Scrapers with Cron and GitHub Actions.

7. Intercept Requests with a Downloader Middleware

Anything that must apply to every outgoing request — a realistic header set, a proxy, a per-request cookie jar, a custom retry rule — belongs in a downloader middleware rather than in the spider. The middleware sees the request on the way out and the response on the way back, and returning None from process_request means "carry on down the stack".

# bookstore/middlewares.py
import random

from scrapy.http import Request, Response

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
]


class RealisticHeadersMiddleware:
    def process_request(self, request: Request, spider) -> None:
        request.headers.setdefault("User-Agent", random.choice(USER_AGENTS))
        request.headers.setdefault(
            "Accept",
            "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        )
        request.headers.setdefault("Accept-Language", "en-GB,en;q=0.9")
        request.headers.setdefault("Sec-Fetch-Mode", "navigate")
        return None

    def process_response(self, request: Request, response: Response, spider) -> Response:
        if response.status == 403:
            spider.logger.warning("403 on %s — headers may be insufficient", request.url)
        return response

# bookstore/settings.py (excerpt)
DOWNLOADER_MIDDLEWARES = {
    "bookstore.middlewares.RealisticHeadersMiddleware": 400,
    "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
}

Setting the built-in UserAgentMiddleware to None disables it so it cannot overwrite the header you just set. The numbers order the stack: on the way out, middlewares run low to high; on the way back, high to low. Putting a header middleware at 400 places it before the retry middleware at 550, so retried requests get fresh headers.

8. Lock the Selectors Down with Contracts

Selectors break silently. Scrapy's contracts turn a docstring on a callback into an executable check, so a schema change fails in CI rather than producing empty rows for a week.

    def parse_book(self, response: Response, list_price: str = "£10.00"):
        """Extract one book detail page.

        @url https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
        @returns items 1 1
        @scrapes url title list_price availability upc
        """
        yield {
            "url": response.url,
            "title": response.css("div.product_main h1::text").get(),
            "list_price": list_price,
            "availability": " ".join(
                response.css("p.availability::text").getall()
            ).strip(),
            "upc": response.xpath(
                "//th[text()='UPC']/following-sibling::td/text()"
            ).get(),
        }

scrapy check books

@returns items 1 1 asserts exactly one item, and @scrapes asserts those fields are present and non-empty. It is a live network test, so keep it out of the fast unit suite and run it on a schedule — a failure means the site changed, which is exactly the alert you want.

Performance and Scaling Considerations

One Scrapy process saturates about one core. The Twisted reactor is single-threaded, so TLS, HTML parsing and item processing all compete for the same core. Realistic throughput on plain HTML is 200–800 pages per minute per process. To go faster, shard the URL space and run several processes rather than raising CONCURRENT_REQUESTS past about 100, where the reactor starts spending more time scheduling than downloading.

Memory is dominated by the scheduler queue and the dupefilter. The default dupefilter keeps a 40-byte fingerprint per seen request in a Python set — about 400 MB at ten million URLs, plus set overhead. With JOBDIR the queue itself spills to disk, but the fingerprint set stays resident. For very large frontiers, a probabilistic filter trades a tiny false-positive rate for constant memory, as described in Deduplicating URLs with Bloom Filters.

Pipelines are on the critical path. process_item runs on the reactor thread, so a synchronous database INSERT per item stalls the entire crawl for the duration of every write. Buffer items in the pipeline and flush in batches of a few hundred, or hand the write to a thread via deferToThread. Sink choices and batching strategy are covered in Storing and Exporting Scraped Data.

Feed exports are cheaper than pipelines for plain files. FEEDS writes with a buffered exporter and supports post-processing and remote destinations, so for CSV, JSON Lines or Parquet output there is rarely a reason to write file I/O by hand.

The stats collector is your instrumentation. Every run ends with downloader/response_status_count/*, item_scraped_count, dupefilter/filtered and finish_reason. Watching those across runs catches silent breakage — an unchanged page count with a collapsed item count means selectors broke, not that the site shrank. Shipping them somewhere durable is covered in Exporting Scrapy Metrics to Prometheus.

Blocks are a settings problem before they are a proxy problem. Lower per-domain concurrency and enable AutoThrottle first. When a site blocks by address regardless of pace, the answer is address diversity — see Rotating Proxies and Managing IP Blocks.

Common Errors and Fixes

ModuleNotFoundError: No module named 'bookstore' when running scrapy crawl Scrapy resolves the project from scrapy.cfg in the working directory. You are one level too deep or too shallow. Run the command from the directory containing scrapy.cfg, not from inside the inner package.

ValueError: Missing scheme in request url: /catalogue/page-2.html A relative href was passed to scrapy.Request directly. Use response.follow(href, callback=...), which joins against the current response URL, or response.urljoin(href) if you need the string.

Filtered offsite request to 'cdn.example.com'allowed_domains excludes the host. Add it, or if the request must escape the allow-list deliberately, set dont_filter=True on that request. Note that allowed_domains entries are hostnames, not URLs — https://books.toscrape.com/ there will never match.

DEBUG: Forbidden by robots.txt: <GET https://example.com/page>ROBOTSTXT_OBEY is on and the path is disallowed. This is a middleware decision, so no response reaches your spider. Decide deliberately whether the path is one you are entitled to fetch before changing the setting.

twisted.internet.error.ReactorNotRestartableCrawlerProcess.start() was called twice in one interpreter, typically by looping over spiders. Pass every spider to a single process with repeated process.crawl(...) calls before one process.start(), or use CrawlerRunner with your own reactor management.

Items are scraped but the output file is empty Either a pipeline is raising DropItem for every record — check item_dropped_count in the run stats — or you used -o against an existing file and are reading the stale head of it. Check the stats block printed at the end of every crawl before suspecting the exporter.

TimeoutError: User timeout caused connection failure on every requestDOWNLOAD_TIMEOUT elapsed before the server responded. Against a slow origin, raise it to 60. If it happens on all hosts at once, the process cannot reach the network at all — test with scrapy fetch "https://books.toscrape.com/", which exercises the same downloader stack.

Frequently Asked Questions

Does Scrapy execute JavaScript? No. It performs HTTP requests and parses the returned markup, so anything injected by client-side code is invisible to it. The usual answers are to find the underlying JSON endpoint the page itself calls, or to plug in a browser through scrapy-playwright — which is far slower per page, so use it only for the URLs that need it. The browser side of that is covered in Using Playwright for Modern Web Automation.

Why is my crawl much slower than the concurrency setting suggests? Almost always DOWNLOAD_DELAY combined with CONCURRENT_REQUESTS_PER_DOMAIN on a single-domain crawl, or AutoThrottle having raised the delay because the server is slow. The global CONCURRENT_REQUESTS is an upper bound that a one-domain crawl will never reach.

Can a spider start from a database or a file instead of start_urls? Yes. Override start_requests and yield scrapy.Request objects from any source. That is the normal way to feed a spider from a queue, a seed table, or a sitemap.

How do I pass arguments to a spider at run time? Use -a, as in scrapy crawl books -a category=fiction. The value arrives as an instance attribute on the spider, so self.category is available in start_requests and every callback. Everything arrives as a string, so cast it yourself.

Should the crawl and the parsing live in the same spider? For most sites, yes — the callback that receives a response is the natural place to extract from it. Split them only when parsing is expensive enough to stall the reactor, in which case store raw responses and run extraction as a separate offline pass.

Related

• Scaling Python Web Scrapers — how Scrapy compares with async and distributed approaches
• Writing Scrapy Item Pipelines — validation, batching and database writes
• Scrapy vs BeautifulSoup: Which to Use — picking the right level of tooling
• Scrapy vs Playwright for Single-Page Apps — handling JavaScript-rendered content
• Exporting Scrapy Metrics to Prometheus — turning run stats into alerts