BeautifulSoup vs lxml: Which Parser Is Faster

The parser choice you make in Parsing HTML with BeautifulSoup sets the floor on how much CPU each page costs, and it also quietly decides what tree your selectors will be querying.

Raw lxml.html is the fastest option by a wide margin, typically several times quicker than BeautifulSoup over the same lxml backend and roughly an order of magnitude quicker than html5lib. But the comparison is not BeautifulSoup versus lxml — BeautifulSoup uses lxml. The real question is how much you pay for BeautifulSoup's object model, and the answer is that the overhead only matters once parsing dominates your runtime, which for a network-bound scraper it almost never does.

Where the Time Actually Goes

BeautifulSoup is a facade. Given BeautifulSoup(markup, "lxml") it hands the bytes to lxml's C parser, then walks the resulting tree and builds a parallel structure of Python Tag and NavigableString objects. The C parse is fast; the object construction is not, because every element becomes a full Python object with a dict of attributes, parent and sibling references, and a NavigableString for each text node. On a page with 20,000 elements that is 20,000 heap allocations plus their children, all of them tracked by the cyclic garbage collector because the tree is full of reference cycles.

lxml.html.fromstring skips that entirely. Elements stay in the C tree and Python only sees lightweight proxy objects created on demand as you traverse. That is why the gap widens with document size rather than staying a constant factor: BeautifulSoup's cost is roughly linear in node count, and lxml's Python-side cost is roughly linear in the number of nodes you actually touch.

The four options line up on one axis. html5lib implements the HTML5 tree-construction algorithm in pure Python, reproducing exactly what a browser would build, and it is the slowest by a large multiple. html.parser is the standard library's own tokenizer, also pure Python, moderately fast, and notable mainly for having no C dependency at all. lxml as a BeautifulSoup backend gives you C parsing with a Python tree. lxml used directly gives you C parsing and a C tree.

Measuring It on Your Own Documents

Benchmarks published on other people's pages are close to worthless here, because the ratio depends on how deep and how text-heavy the markup is. Run this against a page you actually scrape:

"""Compare parse time across backends on a document you care about."""
import statistics
import time

import requests
from bs4 import BeautifulSoup
from lxml import html as lxml_html

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}


def timed(label: str, fn, markup, rounds: int = 20) -> None:
    samples: list[float] = []
    for _ in range(rounds):
        start = time.perf_counter()
        fn(markup)
        samples.append(time.perf_counter() - start)
    best = min(samples)
    median = statistics.median(samples)
    print(f"{label:<22} best {best * 1000:7.2f} ms   median {median * 1000:7.2f} ms")


def main() -> None:
    response = requests.get("https://books.toscrape.com/", headers=HEADERS, timeout=20)
    response.raise_for_status()
    markup = response.text
    print(f"document: {len(markup) / 1024:.1f} KiB")

    timed("lxml direct", lambda m: lxml_html.fromstring(m), markup)
    timed("bs4 + lxml", lambda m: BeautifulSoup(m, "lxml"), markup)
    timed("bs4 + html.parser", lambda m: BeautifulSoup(m, "html.parser"), markup)
    timed("bs4 + html5lib", lambda m: BeautifulSoup(m, "html5lib"), markup)


if __name__ == "__main__":
    main()

Report the minimum as well as the median: the minimum is the cleanest estimate of the parser's cost, while the spread between them tells you how much garbage collection is interfering. On one run of that script on a 2023 laptop against a synthetic 2 MB document of nested div elements, lxml direct came in near 25 ms, bs4 over lxml near 260 ms, html.parser near 700 ms, and html5lib past 2.5 s. Those numbers are indicative of the ordering on that setup only — treat the ratios as the transferable part and re-measure before you optimise anything.

Two measurement mistakes are worth avoiding. Parsing the same string repeatedly warms CPU caches in a way a real scraper never sees, so a microbenchmark flatters every parser roughly equally but exaggerates the fastest. And timing BeautifulSoup(...) alone measures construction, not querying — if your workload is one parse followed by 200 select() calls, the query engine matters more than the constructor.

Memory, Not Just Milliseconds

For long-running workers the more important number is peak resident memory, because that is what decides how many concurrent parsers fit on a box. Measure it with the standard library rather than guessing:

import tracemalloc

from bs4 import BeautifulSoup
from lxml import html as lxml_html

MARKUP = "<html><body>" + '<div class="row"><span>cell</span></div>' * 30_000 + "</body></html>"


def peak_kib(fn) -> float:
    tracemalloc.start()
    tree = fn(MARKUP)
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    del tree
    return peak / 1024


print(f"lxml direct : {peak_kib(lambda m: lxml_html.fromstring(m)):9.0f} KiB")
print(f"bs4 + lxml  : {peak_kib(lambda m: BeautifulSoup(m, 'lxml')):9.0f} KiB")

tracemalloc only sees Python-level allocations, so it under-reports lxml, whose tree lives in C memory that the module never observes — read the two figures as "Python heap pressure" rather than total process usage, and cross-check with resource.getrusage or the process RSS if the absolute number matters. Even with that caveat the direction is clear: the BeautifulSoup tree costs several hundred bytes per node in Python objects, and that is the memory a concurrent worker pool multiplies.

The Trees Are Not the Same

Speed is the visible difference; tree shape is the one that causes bugs. Backends disagree about how to repair broken markup, and a selector tuned against one tree can silently return nothing against another.

The canonical case is an implied <tbody>. Given <table><tr><td>1</table>, html5lib follows the HTML5 spec and inserts a <tbody> between the table and the row, exactly as a browser does. lxml and html.parser do not. A selector of table > tr therefore matches under two backends and returns an empty list under the third — with no exception, no warning, and a scraper that just stops producing rows.

from bs4 import BeautifulSoup

MARKUP = "<table><tr><td>1</table>"

for backend in ("html.parser", "lxml", "html5lib"):
    soup = BeautifulSoup(MARKUP, backend)
    direct = len(soup.select("table > tr"))
    descendant = len(soup.select("table tr"))
    print(f"{backend:<12} table > tr: {direct}   table tr: {descendant}")

The lesson is not "use html5lib" but "pin the backend and prefer descendant combinators". Never write BeautifulSoup(markup) without the second argument: with no backend named, bs4 picks the best one installed, so adding lxml to a requirements file months later changes the parse tree of code nobody edited. Writing table tr instead of table > tr also makes the selector immune to the difference. Choosing anchors that survive this kind of structural change is the subject of Writing Selectors That Survive a Redesign.

Getting the Speed Without Losing the API

If a profile shows parsing is genuinely your bottleneck, move the hot path to lxml and keep BeautifulSoup for the awkward corners. lxml supports both selector languages — XPath natively through libxml2, and CSS through the cssselect package, which compiles the selector to XPath once. The choice between them is covered in XPath vs CSS Selectors: Which Should You Use?; the practical note here is that cssselect adds a translation step per unique selector string, so hoist compiled selectors out of loops.

import requests
from lxml import html as lxml_html

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
    "Accept-Language": "en-GB,en;q=0.9",
}


def scrape_catalogue(url: str) -> list[dict[str, str]]:
    response = requests.get(url, headers=HEADERS, timeout=20)
    response.raise_for_status()

    # Pass bytes, not text: lxml then honours the meta charset itself.
    tree = lxml_html.fromstring(response.content)
    tree.make_links_absolute(url)

    books: list[dict[str, str]] = []
    for card in tree.xpath('//article[@class="product_pod"]'):
        title = card.xpath("./h3/a/@title")
        price = card.xpath('.//p[@class="price_color"]/text()')
        link = card.xpath("./h3/a/@href")
        books.append(
            {
                "title": title[0] if title else "",
                "price": price[0].strip() if price else "",
                "url": link[0] if link else "",
            }
        )
    return books


if __name__ == "__main__":
    for book in scrape_catalogue("https://books.toscrape.com/")[:3]:
        print(book)

Three things in that snippet are the actual performance advice. response.content hands raw bytes to lxml so it can read the document's own <meta charset> rather than re-encoding a string requests already decoded. make_links_absolute resolves hrefs in C instead of a Python urljoin per row. And the per-card relative XPath (./h3/a/@title) restricts the search to one subtree — an absolute // inside a loop rescans the whole document on every iteration, which is the single most common way to make an lxml scraper slower than a BeautifulSoup one.

Edge Cases and Caveats

• Installation. pip install lxml ships manylinux, macOS and Windows wheels, so no compiler is needed on those. Alpine's musl libc does not match manylinux, so the install falls back to building from source and needs libxml2-dev, libxslt-dev and a toolchain.
• XMLSyntaxError from etree.fromstring. Using the XML parser on HTML fails on the first unclosed tag. Use lxml.html.fromstring or pass parser=etree.HTMLParser(); the HTML parser recovers instead of raising.
• Empty document. lxml.html.fromstring("") raises ParserError: Document is empty, while BeautifulSoup returns an empty soup. Guard on response.content before parsing, or a single blank response kills a batch job.
• Fragments versus documents. fromstring returns whatever the fragment's root element is, so a snippet of two sibling <div>s comes back wrapped differently from a full page. Use lxml.html.document_fromstring when you need a guaranteed <html> root.
• Memory on long-lived processes. BeautifulSoup trees are cyclic and only freed by the cycle collector. In a worker that parses thousands of pages, del soup plus an occasional gc.collect() measurably lowers peak resident memory; with lxml, dropping the last reference frees the C tree immediately.
• Threads. libxml2 is thread-safe for parsing but a single etree document must not be shared across threads. Parse inside the worker rather than passing trees between them.
• Encoding. Passing an already-decoded str containing an XML declaration to lxml raises ValueError: Unicode strings with encoding declaration are not supported. Pass bytes instead, which is also the right answer for the problems in Fixing Common Unicode Errors in Python Scraping.

Frequently Asked Questions

Is lxml always faster than BeautifulSoup? For building and querying a tree, yes, because lxml keeps the document in C memory while BeautifulSoup materialises a Python object per node. The exception is a workload where you parse once and then do very little with the result, since there the difference is a few milliseconds against a network fetch measured in hundreds. Configure BeautifulSoup with the lxml backend and the constructor cost drops sharply, though the object-building overhead remains.

Can I use XPath with BeautifulSoup? Not directly — BeautifulSoup exposes find, find_all and CSS selection through soupsieve, but no XPath engine. If you need XPath axes such as parent:: or preceding-sibling::, parse with lxml.html instead, or parse once with lxml and wrap only the awkward subtree in BeautifulSoup.

Which parser handles broken HTML best?html5lib, because it implements the same tree-construction algorithm browsers use, so the tree you get is the tree a browser would show. You pay for it: expect it to be several times slower than html.parser and roughly an order of magnitude slower than lxml. Use it as a targeted fallback for the handful of pages the fast parser mangles, not as the default.

Does the parser choice change what my selectors match? Yes, and this is the most under-appreciated risk in the comparison. Backends repair invalid markup differently — the implied <tbody> is the classic example — so the same CSS selector can return different node counts depending on which parser built the tree. Always name the backend explicitly and pin it in your requirements file.

Related

• Parsing HTML with BeautifulSoup — the parent topic, covering the traversal API in depth.
• Selecting Elements with XPath and CSS Selectors — the query languages both libraries expose.
• Extracting Attributes and Nested Tags — pulling hrefs, data attributes and inner text cleanly.
• Scrapy vs BeautifulSoup: Which to Use — the same trade-off one level up, at framework scale.