Scrapy vs BeautifulSoup: Which to Use

This is the most common opening question in Python scraping and it is slightly miscast, so this page — part of Web Scraping with Scrapy — reframes it into the decision you are actually making and gives a rule for settling it.

BeautifulSoup is a parsing library: it turns a string of HTML into a searchable tree and does nothing else. Scrapy is a crawling framework: an asynchronous download engine, a scheduler, retry and throttling middleware, its own selectors and a pipeline for output. They are not alternatives to each other. The genuine comparison is requests plus BeautifulSoup, where you assemble the crawl yourself, against Scrapy, where the framework supplies it. Reach for the first when the job is a handful of pages or a script embedded in a larger application; reach for Scrapy when the crawl spans many linked pages, runs repeatedly, and needs concurrency, retries and throttling that you would otherwise write and debug yourself.

What Each Tool Is Responsible For

Laying the pipeline out stage by stage makes the size of the gap obvious. A crawl has five jobs: decide what to fetch next, fetch it, parse it, discover more URLs, and store the result.

requests owns one box. BeautifulSoup owns one box. The other three are yours to write, and they are the three that get hard as the crawl grows — the scheduler that avoids re-fetching, the concurrency that keeps the network busy without flooding the target, the retry policy that distinguishes a 503 from a 404, the throttle that keeps you welcome, and the output layer that batches writes rather than opening a database connection per page.

Scrapy owns all five. That is the whole substance of the comparison, and the reason "which is better" has no answer independent of how many boxes you need.

There is a second-order point worth knowing: BeautifulSoup is not itself a parser. It is an API over one — html.parser from the standard library, lxml, or html5lib — and which one you pass changes both speed and how broken markup is repaired. That choice is measured in BeautifulSoup vs lxml: Which Parser Is Faster. Scrapy's selectors are parsel, which wraps lxml directly, so on parsing speed alone Scrapy starts from the fast option by default.

Choose requests and BeautifulSoup When

• The job is one page, one endpoint, or a few dozen URLs you already have.
• It is a one-off pull, or a step inside a larger application where a project scaffold would be intrusive.
• You are learning, and want to see the HTTP request and the tree walk without a framework in between.
• The extraction is unusual enough that you want the full BeautifulSoup navigation API — find_parent, next_sibling, find_all(string=...) — rather than selectors.

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/125.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}


def scrape(url: str) -> list[dict[str, str]]:
    response = requests.get(url, headers=HEADERS, timeout=15)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "lxml")
    rows: list[dict[str, str]] = []
    for book in soup.select("article.product_pod"):
        link = book.select_one("h3 a")
        price = book.select_one("p.price_color")
        if link is None or price is None:
            continue
        rows.append({"title": link["title"], "price": price.get_text(strip=True)})
    return rows


if __name__ == "__main__":
    for row in scrape("https://books.toscrape.com/"):
        print(row["title"], row["price"])

Twenty lines, no scaffold, immediately readable. It is also strictly sequential, has no retry policy, no delay between requests, no deduplication, and will follow no links unless you write that loop.

Choose Scrapy When

• The crawl walks many linked pages or a whole section of a site.
• It runs on a schedule and has to survive partial failures without manual intervention.
• You want concurrency, retries and polite throttling without owning that code.
• You want fetching, parsing and storage separated so each can be tested and changed independently.

# books_spider.py — run with: scrapy runspider books_spider.py -o books.jsonl
from collections.abc import Iterator

import scrapy


class BookSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/"]
    custom_settings = {
        "USER_AGENT": (
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
            "(KHTML, like Gecko) Chrome/125.0 Safari/537.36"
        ),
        "CONCURRENT_REQUESTS_PER_DOMAIN": 8,
        "DOWNLOAD_DELAY": 0.25,
        "AUTOTHROTTLE_ENABLED": True,
        "RETRY_TIMES": 3,
    }

    def parse(self, response: scrapy.http.Response) -> Iterator[dict[str, str] | scrapy.Request]:
        for book in response.css("article.product_pod"):
            yield {
                "title": book.css("h3 a::attr(title)").get(),
                "price": book.css("p.price_color::text").get(),
                "url": response.urljoin(book.css("h3 a::attr(href)").get()),
            }
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

The parsing logic is barely longer than the BeautifulSoup version. Everything else — the settings block — is configuration for machinery you did not write: eight concurrent requests per domain, a quarter-second delay, adaptive throttling that widens the delay when the server slows down, three retries on transient failures, automatic deduplication of URLs already seen, and JSON Lines output from a command-line flag. Storage beyond a flat file belongs in an item pipeline, covered in Writing Scrapy Item Pipelines.

Both examples use CSS selectors, and both would accept XPath instead; the trade-off between the two languages is the same in either tool and is covered in Selecting Elements with XPath and CSS Selectors.

Where the Time Actually Goes

The performance question is usually asked as though it were about parsing. It is not. Parsing a 50 KB page with lxml takes single-digit milliseconds; fetching it over the internet takes hundreds. The difference between the two approaches is entirely whether requests overlap.

Those figures come from a single laptop on a home connection against a site with roughly half a second of average response time, and they are indicative arithmetic from that response time rather than a controlled benchmark — the point is the shape, not the digits. A sequential loop can never beat pages × response_time. Adding threads to requests closes most of the gap; Scrapy's event loop closes it with less code and adds the throttling that keeps the concurrency polite. Below about fifty pages the absolute difference is a few seconds and should not drive the decision at all — pick on maintainability instead.

Memory goes the other way. A requests script holds one response at a time; a Scrapy crawl holds a scheduler queue, a duplicate-request filter and whatever is in flight, so a crawl of a million URLs needs attention to queue growth (JOBDIR for on-disk queues) in a way a small script never does.

You Can Use Both

They are not exclusive. Inside a Scrapy callback, BeautifulSoup(response.text, "lxml") works perfectly, and is occasionally the pragmatic answer for a parse where you want find_parent or text-based navigation that parsel expresses awkwardly. The cost is a second parse of the same document, so do it for the tricky page, not for every page.

The reverse also holds: parsel is packaged separately, so you can use Scrapy's Selector class in a plain requests script without adopting the framework.

Neither executes JavaScript. Both see only the HTML the server sent, so a single-page app returns an empty shell to both — the situation covered in Scrapy vs Playwright for Single-Page Apps, where the usual answer is to call the underlying JSON API rather than to render anything.

The Decision Rule

Ask two questions: how many pages, and how often. Few pages, once: requests and BeautifulSoup. Many pages, repeatedly: Scrapy. The reliable signal for migrating is structural rather than numeric — when your BeautifulSoup script grows a URL queue, a seen set, a time.sleep, a retry counter and a thread pool, you have rebuilt the parts of Scrapy you needed, only without the tests.

Migrating a Script Without Rewriting the Parse

The migration is smaller than it looks, because the extraction logic is the part you keep. In practice the mapping is mechanical:

Two behavioural differences will surprise you during the port. Scrapy selectors return None for a missing match rather than raising, so silent None fields replace the TypeError you used to get from soup.select_one(...)["title"] — check for them explicitly or the crawl will fill a file with nulls. And callbacks are generators driven by the engine rather than functions you call, so the order in which pages are parsed is not the order you yielded requests. Anything that depends on sequence has to carry its state in cb_kwargs or meta rather than in a local variable.

Edge Cases and Caveats

• Scrapy is not a library you import into an existing loop. It runs a Twisted reactor, which can be started once per process and does not coexist comfortably with an existing asyncio application. Embedding it inside a web server or a Jupyter notebook needs CrawlerRunner and care; requests has no such constraint.
• BeautifulSoup is only as good as its parser. html.parser is pure Python and lenient, lxml is fast and stricter, html5lib is slow and matches browser recovery best. Pass one explicitly — the default changes with what is installed, which makes bugs environment-dependent.
• Scrapy obeys robots.txt by default. ROBOTSTXT_OBEY = True is on in generated projects, so a spider may silently fetch nothing on a site that disallows your path. That is usually correct behaviour, but it surprises people migrating from a requests script that never checked.
• Item pipelines are synchronous by default. A blocking database write in a pipeline stalls the reactor and caps your throughput regardless of CONCURRENT_REQUESTS. Batch writes, or use an async pipeline.
• response.follow handles relative URLs, requests does not. Hand-written loops routinely break on ../index.html; use urljoin rather than string concatenation if you are assembling URLs yourself.
• Neither is a stealth tool. Both send whatever headers you configure, and a default Scrapy user agent identifies itself as Scrapy. On defended sites the relevant work is in Advanced Scraping Techniques and Anti-Bot Evasion, not in the framework choice.

Frequently Asked Questions

Is Scrapy faster than BeautifulSoup? For multi-page crawls, substantially, but the reason is concurrency rather than parsing. Scrapy's engine keeps many requests in flight while a naive requests loop waits for each response in turn, so wall-clock scales with the response time multiplied by the page count. For a single page the two are within milliseconds of each other, since both end up in lxml.

Can I use BeautifulSoup inside Scrapy? Yes — passing response.text to BeautifulSoup inside a spider callback works fine, and is a reasonable escape hatch when a parse is easier to express with its navigation API. It parses the document a second time, so use it selectively rather than as the default for every page.

Which is better for beginners?requests and BeautifulSoup, because there is almost no machinery between your code and the HTTP request, so the mechanics of headers, status codes and tree traversal stay visible. Move to Scrapy once those are second nature and the crawl needs concurrency, retries and scheduling.

Do either of them handle JavaScript-rendered pages? No. Both operate on the HTML the server returned, so content injected by client-side JavaScript is simply absent. The usual fix is not a browser but the JSON API the page's own JavaScript calls; when rendering genuinely is required, a headless browser such as Playwright can be plugged into a Scrapy crawl for the pages that need it.

Related

• Web Scraping with Scrapy — the parent topic covering spiders, settings and project layout.
• Parsing HTML with BeautifulSoup — the other side of the comparison, in depth.
• Writing Scrapy Item Pipelines — the storage stage that a plain script has to build by hand.
• Scrapy vs Playwright for Single-Page Apps — what to do when neither tool can see the content.