Parsing HTML with BeautifulSoup

This guide is the parsing stage of The Complete Guide to Python Web Scraping: everything that happens after a response comes back with a 200 and before the records reach storage. The scope is BeautifulSoup specifically — choosing a backend, querying the tree, reading attributes and text safely, and writing extraction code that fails loudly instead of returning None.

BeautifulSoup parses HTML into a DOM tree you traverse with selectors.

BeautifulSoup does not fetch anything. It takes a string or a byte buffer, hands it to a parser backend, and wraps the resulting tree in an API that is pleasant to work with. That division matters, because roughly half of all "BeautifulSoup problems" are actually parser problems — the tree you are querying is not the tree you saw in the browser's inspector, and no amount of selector tuning will fix it.

When to Use BeautifulSoup

BeautifulSoup is the right tool when you are extracting a handful of fields from HTML you already have in memory and you value readable code over raw speed. It is the wrong tool in three specific situations.

Situation	Use BeautifulSoup?	Alternative
A few dozen to a few thousand HTML pages, tag-and-class extraction	Yes	—
Markup is badly broken and other parsers disagree with the browser	Yes, with the `html5lib` backend	—
You need to walk up or backwards from a matched node	Only via `.parent` / `.find_previous_sibling()`	XPath, via Selecting Elements with XPath and CSS Selectors
Hundreds of thousands of pages, CPU-bound	Marginal	`lxml.html` directly, or Scrapy's `Selector`
The response is JSON, not HTML	No	Parsing JSON and XML Responses
Content appears only after JavaScript runs	No	Using Playwright for Modern Web Automation

The last row is worth restating because it accounts for so many wasted hours: BeautifulSoup sees exactly the bytes the server sent. If a price is written into the DOM by a script after load, it is not in response.text and no selector can find it. Check by searching the raw response for the value, not by looking in the browser inspector — the inspector shows the rendered DOM, which is a different document.

Prerequisites

Python 3.10 or newer, in the environment described in Setting Up Your Python Scraping Environment. Install BeautifulSoup with both fast and forgiving backends so you can switch without another install cycle.

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install "beautifulsoup4>=4.12" "lxml>=5.1" "html5lib>=1.1" "requests>=2.31"

beautifulsoup4 pulls in soupsieve, which is the CSS selector engine behind .select(). lxml needs libxml2 and libxslt but ships wheels for all common platforms, so a compiler is normally not required. html5lib is pure Python and slow, but it is the only backend that follows the WHATWG parsing algorithm exactly, which is what browsers implement.

Step-by-Step: From Response to Clean Records

1. Choose the backend deliberately

The second argument to BeautifulSoup() is not optional in practice. Omit it and the library picks the best available parser and emits a GuessedAtParserWarning — meaning the behaviour of your code depends on which packages happen to be installed on the machine. Name it explicitly, always.

The same broken fragment produces three different trees. Your selectors run against whichever tree the backend built, which is why swapping parsers can break working code.

from bs4 import BeautifulSoup

FRAGMENT = "<a></p>"

for backend in ("html.parser", "lxml", "html5lib"):
    soup = BeautifulSoup(FRAGMENT, backend)
    print(f"{backend:>12}: {soup}")

The three lines of output differ. html.parser produces <a></a>, discarding the orphan close tag. lxml wraps the result in <html><body>, because libxml2 always builds a complete document. html5lib produces <html><head></head><body><a><p></p></a></body></html>, reproducing the browser's recovery rules exactly. A selector such as body > a matches under two backends and not the third — which is why swapping backends "for speed" can break working extraction code.

The practical rule: use lxml by default, and switch to html5lib only when your selectors work in the browser console but not in Python. The measured speed gap is quantified in BeautifulSoup vs lxml: Which Parser Is Faster.

2. Parse from bytes, not from a decoded string

Passing response.content rather than response.text lets BeautifulSoup read the document's own <meta charset> declaration through its Unicode Dammit layer. Passing response.text means the decoding decision was already made by the HTTP client from headers alone, which is a weaker signal — the details of that trade-off are in Understanding HTTP Requests and Responses.

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-GB,en;q=0.9",
}

response = requests.get("https://books.toscrape.com/", headers=HEADERS, timeout=(5, 20))
response.raise_for_status()

soup = BeautifulSoup(response.content, "lxml")
print(soup.title.string.strip() if soup.title else "no title")
print("original encoding:", soup.original_encoding)

3. Pick the right search method

There are four, and the difference that matters is what they return when nothing matches. find() and select_one() return None; find_all() and select() return an empty sequence. Chaining onto None is the single most common crash in scraping code.

The four search methods form a grid: two return the first match or None, two return a list that may be empty — and forgetting which is which is the source of most AttributeError crashes.

from bs4 import BeautifulSoup

HTML = """
<ol class="row">
  <li><article class="product_pod">
    <h3><a href="catalogue/a-light_1/index.html" title="A Light in the Attic">A Light...</a></h3>
    <p class="price_color">£51.77</p>
    <p class="instock availability">In stock</p>
  </article></li>
</ol>
"""

soup = BeautifulSoup(HTML, "lxml")

print(soup.find("h3").a["title"])                 # tag filter, first match
print(soup.select_one("p.price_color").text)      # CSS filter, first match
print(len(soup.find_all("article")))              # tag filter, all matches
print([p.get_text(strip=True) for p in soup.select("article p")])

find_all() also accepts attribute filters directly — soup.find_all("p", class_="price_color") — plus a string= argument for text matching and a callable for anything the other filters cannot express. select() accepts the CSS subset soupsieve supports, which includes :not(), :nth-of-type(), attribute operators and :has().

4. Read attributes and text without crashing

A missing element and a present element with a missing attribute are two different failures, and both must be handled. tag["href"] raises KeyError; tag.get("href") returns None. .get_text() on a None raises AttributeError. Write one small helper and use it everywhere rather than scattering if x is not None through the extraction code.

from bs4 import BeautifulSoup
from bs4.element import Tag


def text_of(node: Tag | None, selector: str, default: str = "") -> str:
    """Return stripped text for the first match, or a default when absent."""
    if node is None:
        return default
    found = node.select_one(selector)
    return found.get_text(" ", strip=True) if found else default


def attr_of(node: Tag | None, selector: str, name: str, default: str = "") -> str:
    """Return an attribute value for the first match, or a default when absent."""
    if node is None:
        return default
    found = node.select_one(selector)
    value = found.get(name) if found else None
    if isinstance(value, list):        # class and rel come back as lists
        return " ".join(value)
    return value if isinstance(value, str) else default


soup = BeautifulSoup(
    '<div class="card"><a href="/p/1" title="Widget">Widget</a><span>£9.99</span></div>',
    "lxml",
)
card = soup.select_one("div.card")
print(text_of(card, "span"), attr_of(card, "a", "href"), attr_of(card, "a", "title"))

Note the list case: multi-valued attributes (class, rel, accept-charset) come back as a list of strings, not a string, because HTML defines them as space-separated token lists. Assuming a string there produces a TypeError deep inside your cleaning code. Nested attribute extraction and the multi-valued rules are covered fully in Extracting Attributes and Nested Tags.

5. Navigate relative to an anchor when the structure is unlabelled

Product tables and specification lists frequently have no useful classes at all: the value you want is simply the cell after the one that says "Price". BeautifulSoup can walk in every direction from a matched node, which lets you anchor on the stable text and move to the unstable value.

from bs4 import BeautifulSoup

HTML = """
<table class="table"><tbody>
  <tr><th>UPC</th><td>a897fe39b1053632</td></tr>
  <tr><th>Price (excl. tax)</th><td>£51.77</td></tr>
  <tr><th>Availability</th><td>In stock (22 available)</td></tr>
</tbody></table>
"""

soup = BeautifulSoup(HTML, "lxml")


def value_for(label: str) -> str:
    """Find the header cell by its visible text, then take the cell beside it."""
    header = soup.find("th", string=lambda s: s and s.strip() == label)
    if header is None:
        return ""
    cell = header.find_next_sibling("td")
    return cell.get_text(strip=True) if cell else ""


print(value_for("Price (excl. tax)"), "|", value_for("Availability"))

The relevant navigation members are .parent and .parents, .children and .descendants, .next_sibling and .previous_sibling, and their filtered forms .find_next_sibling(), .find_previous_sibling() and .find_parent(). Prefer the filtered forms: the raw .next_sibling returns whitespace NavigableString objects between tags, which is a classic source of confusion when a "sibling" turns out to be a newline. When the climb gets more than two steps deep, an XPath expression states the same intent more compactly, which is the argument made in Selecting Elements with XPath and CSS Selectors.

6. Normalise the text you extract

.get_text() concatenates every descendant string. Without a separator, <span>In</span><span>stock</span> becomes Instock. Without strip=True, you inherit every newline and indent from the source. And even with both, HTML text carries non-breaking spaces (U+00A0), zero-width characters and inconsistent whitespace that will break every downstream comparison.

import re
import unicodedata
from bs4 import BeautifulSoup

WHITESPACE = re.compile(r"\s+")


def clean(raw: str) -> str:
    """Collapse HTML whitespace and normalise Unicode to a comparable form."""
    text = unicodedata.normalize("NFKC", raw)   # NBSP -> space, ligatures -> letters
    return WHITESPACE.sub(" ", text).strip()


soup = BeautifulSoup(
    "<p class='availability'>\n  In stock (22 available)\n</p>", "lxml"
)
print(repr(soup.p.get_text(" ", strip=True)))
print(repr(clean(soup.p.get_text(" ", strip=True))))

NFKC normalisation is the step people skip. It turns U+00A0 into a plain space and folds compatibility characters, so "In stock (22 available)" compares equal to itself regardless of which variant the page used.

7. Extract records, not fields

Loop over container elements and pull every field relative to that container. Selecting all prices and all titles separately, then zipping them, breaks silently the moment one card is missing a price — you get correctly-typed, completely wrong data, misaligned by one from that point on.

import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) Chrome/136.0.0.0 Safari/537.36"}


def parse_listing(html: bytes) -> list[dict[str, str]]:
    """One dict per product card, with every field resolved inside its own card."""
    soup = BeautifulSoup(html, "lxml")
    records: list[dict[str, str]] = []
    for card in soup.select("article.product_pod"):
        link = card.select_one("h3 > a")
        price = card.select_one("p.price_color")
        stock = card.select_one("p.availability")
        records.append(
            {
                "title": (link.get("title") or "").strip() if link else "",
                "url": (link.get("href") or "").strip() if link else "",
                "price": price.get_text(strip=True) if price else "",
                "stock": stock.get_text(" ", strip=True) if stock else "",
            }
        )
    return records


page = requests.get("https://books.toscrape.com/", headers=HEADERS, timeout=(5, 20))
page.raise_for_status()
rows = parse_listing(page.content)
print(len(rows), rows[0])

8. Assert the shape before you trust the output

Extraction that returns zero rows should be an error, not an empty file. Two cheap assertions — a minimum row count and a required-field check — convert a silent regression into a loud one, which is the entire difference between noticing a site redesign on the day it happens and noticing it a month later.

def validate(rows: list[dict[str, str]], minimum: int = 10) -> list[dict[str, str]]:
    """Fail fast when a layout change quietly empties the result set."""
    if len(rows) < minimum:
        raise ValueError(f"expected at least {minimum} records, parsed {len(rows)}")
    missing = [i for i, r in enumerate(rows) if not r["title"] or not r["price"]]
    if missing:
        raise ValueError(f"{len(missing)} record(s) missing a required field, first at {missing[0]}")
    return rows

Schema-level validation with types and coercion is a step beyond this; see Cleaning and Validating Scraped Data.

Performance and Scaling Considerations

Parsing is CPU-bound and will become your bottleneck. On a 250 KB listing page, lxml builds the tree in roughly 8–12 ms, html.parser in 45–70 ms, and html5lib in 300–500 ms. Fetching that page over the network takes 100–400 ms. So with lxml you are network-bound and concurrency helps; with html5lib you are CPU-bound and adding threads does nothing, because the GIL serialises the parse.

The tree costs several times the source. A parsed BeautifulSoup document typically occupies 5–10× the byte size of the HTML, because every tag becomes a Python object with dictionaries for attributes and lists for children. A 500 KB page is therefore 3–5 MB live. Parsing 20 pages concurrently in one process peaks near 100 MB before your own data. Delete the soup explicitly inside long loops, and never keep parsed trees in a list.

Parse once, query many times. Each select() call walks the tree, but constructing the tree is what actually costs. Never re-create the soup per field. If you need the same page repeatedly across runs, cache the raw bytes rather than re-fetching — see HTTP Caching with requests-cache.

SoupStrainer cuts memory when you need one region. Passing parse_only builds a tree containing only the matching subtrees, which on a large page with one relevant table can reduce peak memory by an order of magnitude and cut parse time proportionally.

from bs4 import BeautifulSoup, SoupStrainer

only_products = SoupStrainer("article", class_="product_pod")
soup = BeautifulSoup(open("listing.html", "rb").read(), "lxml", parse_only=only_products)
print(len(soup.find_all("article")))

Strip the noise before you read text. Pages carry inline <script>, <style>, <noscript> and SVG blocks whose contents get_text() will happily include, producing records full of minified JavaScript. Removing those subtrees once with decompose() is far cheaper than filtering their text out afterwards, and it also shrinks the tree you keep in memory.

from bs4 import BeautifulSoup

soup = BeautifulSoup(open("page.html", "rb").read(), "lxml")
for junk in soup.select("script, style, noscript, template, svg"):
    junk.decompose()
print(len(soup.get_text(" ", strip=True)))

Scale out with processes, not threads. Because parsing holds the GIL, the way to use more cores is concurrent.futures.ProcessPoolExecutor over the raw HTML strings, or a framework that already separates the two — Scrapy runs parsing in the reactor thread but overlaps it with non-blocking I/O, and the trade-off is examined in Scrapy vs BeautifulSoup: Which to Use.

Common Errors and Fixes

AttributeError: 'NoneType' object has no attribute 'text'.find() or select_one() matched nothing and you chained onto the result. The cause is either a wrong selector or a page variant that genuinely lacks the element. Guard rather than wrapping the whole loop in try, so a missing optional field does not discard the whole record.

title_node = card.select_one("h3 > a")
title = title_node.get_text(strip=True) if title_node else ""

bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. The backend name is valid but the package is not installed in the interpreter that is running. This happens constantly when the editor uses one environment and the terminal another.

python -m pip install lxml
python -c "import lxml.etree; print(lxml.etree.LXML_VERSION)"

GuessedAtParserWarning: No parser was explicitly specified. You called BeautifulSoup(html) with one argument. The warning is real: the chosen parser varies by machine, so results vary by machine. Always pass the backend name as the second argument.

KeyError: 'href' on an element that visibly has an href. You are looking at a different element than you think — commonly the wrapping <h3> rather than the <a> inside it. Use .get() while debugging and print node.name and node.attrs to see what you actually selected.

node = soup.select_one("h3")
print(node.name, node.attrs)          # h3 {}
print(node.a.name, node.a.attrs)      # a {'href': '...', 'title': '...'}

Selectors that work in the browser console return nothing in Python. Two causes, and they need different fixes. Either the content is JavaScript-rendered — check with "price_color" in response.text against the raw response — or the browser's HTML repair differs from your backend's, in which case reparse with html5lib and compare. A third, rarer cause is that the site serves different markup to your User-Agent than to a browser.

UnicodeEncodeError when printing extracted text on Windows. The console codec cannot represent the character, not a parsing fault. Set PYTHONUTF8=1 in the environment or write to a file with an explicit encoding="utf-8". The broader family of encoding failures is covered in Fixing Common Unicode Errors in Python Scraping.

RecursionError: maximum recursion depth exceeded while pickling or deep-copying a soup. BeautifulSoup trees are deeply linked object graphs. Do not pass them across process boundaries or store them; pass the HTML string instead and re-parse on the other side. It is cheaper than it sounds and removes the failure entirely.

Frequently Asked Questions

Can BeautifulSoup handle JavaScript-rendered content? No. It parses the bytes it is given and never executes anything. When the data is injected client-side, either render the page with Playwright or Selenium and hand the rendered HTML to BeautifulSoup, or find the underlying XHR endpoint and read its JSON directly — usually the faster and more stable option.

Which parser backend should I use in production? Use lxml. It is five to forty times faster than the alternatives and handles the great majority of real-world markup correctly. Switch a specific site to html5lib when its markup is broken enough that lxml builds a different tree than the browser, and accept the speed cost for that site only.

Why does my class selector miss elements that clearly have that class?class is a multi-valued attribute. find_all("p", class_="price") matches an element whose class list contains price, but find_all("p", attrs={"class": "price"}) compares the whole attribute string and will miss class="price color". Use the class_ keyword or a CSS selector, never a whole-string attribute comparison.

How do I get data that only exists inside a <script> tag? Select the script element, take its .string, and parse that text as JSON — most sites embed their state as JSON or JSON-LD rather than as generated markup, so this is often cleaner than scraping the rendered HTML. The structured-data case specifically is covered in Extracting JSON-LD and Structured Data, and the general fallback of pulling a substring out with a pattern is in Extracting Data with Regular Expressions.

Should I use BeautifulSoup or lxml directly? Use BeautifulSoup when readability and forgiveness matter more than microseconds, which is most of the time. Use lxml.html directly when you are parsing hundreds of thousands of documents, or when you need XPath, since BeautifulSoup has no XPath support at all.

The Complete Guide to Python Web Scraping — the parent guide this page belongs to.
Extracting Attributes and Nested Tags — multi-valued attributes, nested text and relative URLs.
BeautifulSoup vs lxml: Which Parser Is Faster — the measured backend comparison.
Extracting Data with Regular Expressions — the right way to use regex alongside a parser.
Extracting JSON-LD and Structured Data — reading the machine-readable copy of the page instead.