Extracting Attributes and Nested Tags

Once a selector has returned the right element, the remaining work is reading values out of it without crashing on the one row that is shaped differently; this page expands on the extraction section of Parsing HTML with BeautifulSoup.

Three defaults will save you most of the debugging. Read attributes with tag.get("name") rather than tag["name"], because the first returns None on a missing attribute and the second raises KeyError and ends the run. Expect class, rel and a handful of other attributes to come back as a list, not a string, because HTML defines them as space-separated. And use .get_text(strip=True, separator=" ") rather than .text or .string, because it is the only one of the three that both survives nested markup and returns something you can compare or store without further cleaning.

Attribute Access: .get() versus Square Brackets

A Tag behaves like a dictionary of its attributes, so both tag["href"] and tag.get("href") work when the attribute is present. They differ entirely when it is not: subscript access raises KeyError, while get returns None, or a default you supply as the second argument.

That difference decides whether one unusual row costs you the whole crawl. Real pages are inconsistent — an out-of-stock product with no price link, a promotional tile using the same class as a product card but carrying no data-sku, an image with a srcset but no src. With subscript access, that single row raises an exception a few hundred items into a run and, unless every call is wrapped, the accumulated results go with it. With get, the field comes back None, the row is either skipped or stored with a null, and the run finishes.

tag.attrs is the underlying dictionary, useful when you need to inspect what is actually present. "data-sku" in tag.attrs is the cheap membership test, and iterating tag.attrs.items() during development quickly reveals which attributes the markup really carries versus the ones the DevTools inspector shows after JavaScript has run — a discrepancy that explains a surprising share of "the selector works in the browser but not in Python" reports.

One habit worth adopting: give get an explicit default when a missing value has a natural neutral. tag.get("data-qty", "0") is clearer at the point of use than a None that has to be handled two functions later, and it keeps the shape of your records uniform, which matters when they reach a database or a dataframe.

Multi-Valued Attributes and data-*

HTML specifies certain attributes as taking a space-separated list of tokens, and BeautifulSoup honours that by returning a Python list. class is the one that catches everyone, along with rel, accept-charset, headers and a few others.

So for <article class="product card sale">, tag.get("class") returns ["product", "card", "sale"]. Consequences follow immediately. tag.get("class") == "product" is always False; the correct test is "sale" in tag.get("class", []), and the [] default matters because an element with no class returns None and in None raises TypeError. If you need the original string, " ".join(tag.get("class", [])) rebuilds it. And tag.get("class", [])[0] is not the "main" class — token order is whatever the author wrote and a redesign can reorder it freely.

The behaviour is parser-dependent in one specific case. When BeautifulSoup is constructed with features="xml", no attribute is treated as multi-valued, because XML has no such concept, so class comes back as a plain string. Code that works against lxml in HTML mode and breaks when parsing an XML feed usually breaks here.

data-* attributes have none of that complexity and are frequently the most valuable thing on the page. They are always plain strings, they are the mechanism front-end frameworks use to attach identifiers, prices and state to elements, and they are far more stable across redesigns than presentational classes. article.get("data-product-id"), button.get("data-price-cents"), div.get("data-variant-sku") will typically still work after a visual rebuild that invalidated every class-based selector. Search the raw HTML for data- early on — a great deal of extraction work disappears when the identifiers are already in the markup.

Two are worth special mention because they hold structured payloads. Attributes containing JSON — commonly named data-props, data-state or data-component-props — parse straight into a dict with json.loads, giving you fields that never appear in the rendered text. And srcset holds a comma-separated list of candidate images with width descriptors, which needs splitting on commas and then whitespace to recover the URL for the size you want.

Resolving Relative URLs

An href or src from the page is usually relative, and stitching it to the site root with string concatenation fails on the first document that does something slightly unusual. There are at least five distinct forms in ordinary markup: absolute (https://host/a/b), protocol-relative (//host/a/b), root-relative (/a/b), path-relative (b.html), and dot-relative (../media/x.jpg). A hand-rolled base + href handles one of them.

urllib.parse.urljoin handles all of them, along with query strings and fragments, in one call and with no dependencies. The rules that matter in practice: join against the final URL after redirects (response.url, not the URL you requested), because a redirect from /catalogue/ to /catalogue/page-1.html changes what a path-relative link resolves to; pass an absolute href straight through, since urljoin returns it unchanged and you never need a conditional; and check for a <base href> element in the document, because when present it overrides the response URL for every relative link on the page. That last case is rare enough to forget and common enough to bite, particularly on documentation sites and older CMS templates.

Getting Clean Text out of Nested Markup

BeautifulSoup offers several ways to read text and they are not interchangeable.

.string returns a value only when the tag has exactly one child and that child is a string. <b>24.00</b> gives "24.00"; <p>Price: <b>24.00</b></p> gives None, because the paragraph has two children. This is the classic silent failure — the code works on the tidy element you tested and returns None on the real one. Avoid .string unless you specifically want that "exactly one text child" semantics.

.text (identical to calling .get_text() with no arguments) concatenates every descendant string in document order. It never returns None, but it preserves the source whitespace exactly, so pretty-printed HTML yields '\n Price:\n 24.00\n', and inline tags with no whitespace between them yield run-together words such as "Price:24.00".

.get_text(strip=True, separator=" ") is the version to reach for. strip=True strips each individual string before joining and drops the ones that become empty, so indentation disappears; separator=" " inserts a space between the surviving pieces, so adjacent inline tags do not merge. Together they turn both problem cases above into "Price: 24.00".

.stripped_strings is a generator of the individual stripped strings rather than one joined value. Use it when the pieces are separate data — a definition list where you want the label and the value apart, or a card where the title and price are siblings and you would rather have ["Price:", "24.00"] than one string you must split again.

Excluding a child from the text is the case none of the accessors covers. Suppose a title element contains a "New" badge you do not want in the product name. Two approaches work, and the choice depends on whether you need the original tree afterwards. Calling .decompose() on the unwanted child removes it from the tree permanently and destroys it, which is fine when you are finished with that element. Calling .extract() removes it and returns it, so you can read the badge separately and still have the cleaned parent. When the tree must stay intact — because another part of the code will read the same soup — iterate the parent's direct children and keep only the NavigableString instances, which never touches the tree at all.

A Complete Extraction Function

The script below fetches a real catalogue page and applies everything above: safe attribute reads, a multi-valued class test, urljoin against the final response URL, cleaned text, and a title with an unwanted child element removed.

import json
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup, NavigableString, Tag

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
}


def own_text(tag: Tag) -> str:
    """Text belonging directly to this tag, ignoring any child elements."""
    return " ".join(
        piece.strip()
        for piece in tag.children
        if isinstance(piece, NavigableString) and piece.strip()
    )


def parse_products(html: str, page_url: str) -> list[dict[str, object]]:
    """Extract one record per product card, tolerating missing attributes."""
    soup = BeautifulSoup(html, "lxml")

    base = soup.find("base")
    root = urljoin(page_url, base.get("href", "")) if isinstance(base, Tag) else page_url

    records: list[dict[str, object]] = []
    for card in soup.select("article.product_pod"):
        link = card.find("a", href=True)
        image = card.find("img")
        classes = card.get("class", [])

        payload_raw = card.get("data-props")
        try:
            payload = json.loads(payload_raw) if payload_raw else {}
        except json.JSONDecodeError:
            payload = {}

        records.append(
            {
                "title": link.get("title", "") if isinstance(link, Tag) else "",
                "url": urljoin(root, link.get("href", "")) if isinstance(link, Tag) else "",
                "image": urljoin(root, image.get("src", "")) if isinstance(image, Tag) else "",
                "alt": image.get("alt", "") if isinstance(image, Tag) else "",
                "price": card.select_one(".price_color").get_text(strip=True)
                if card.select_one(".price_color")
                else None,
                "availability": card.select_one(".availability").get_text(
                    strip=True, separator=" "
                )
                if card.select_one(".availability")
                else None,
                "on_sale": "sale" in classes,
                "sku": card.get("data-sku"),
                "extra": payload,
            }
        )
    return records


def title_without_badge(card: Tag) -> str:
    """Read a heading's text after discarding a nested badge element."""
    heading = card.find("h3")
    if not isinstance(heading, Tag):
        return ""
    badge = heading.find("span", class_="badge")
    if isinstance(badge, Tag):
        badge.extract()
    return heading.get_text(strip=True, separator=" ")


if __name__ == "__main__":
    response = requests.get(
        "https://books.toscrape.com/catalogue/page-1.html",
        headers=HEADERS,
        timeout=15,
    )
    response.raise_for_status()
    rows = parse_products(response.text, response.url)
    print(f"{len(rows)} products")
    for row in rows[:3]:
        print(row["title"], row["price"], row["url"])
    first_card = BeautifulSoup(response.text, "lxml").select_one("article.product_pod")
    if first_card is not None:
        print("clean heading:", title_without_badge(first_card))

Two things in there are deliberate rather than defensive noise. response.url is passed into the parser instead of the request URL, so redirects resolve correctly. And the isinstance(x, Tag) checks exist because find returns Tag | NavigableString | None — the check narrows the type for a static checker and guards the None case in the same expression.

Edge Cases and Caveats

• Attribute names are lower-cased in HTML mode. <div DATA-ID="7"> is reachable as data-id. In XML mode case is preserved, so the same lookup fails.
• Boolean attributes have an unhelpful value. <input disabled> gives "" from get("disabled"), which is falsy. Test with "disabled" in tag.attrs, not on the value.
• Entities are already decoded. & in an href arrives as &, so do not unescape a second time — and do not re-escape before comparing against a URL from elsewhere.
• Duplicate attributes keep the first. <a href="/a" href="/b"> yields /a under html.parser and lxml; the second is silently discarded, which can hide a template bug.
• .text on a <script> or <style> returns the code. soup.get_text() on a whole document includes JavaScript source unless you decompose those tags first.
• Whitespace inside values is preserved. strip=True cleans text nodes, not attribute values; tag.get("data-name", "").strip() is still needed.
• Parser choice changes the tree. Malformed markup is repaired differently by each backend, so an element's children can differ between lxml and html5lib. The measured trade-offs are in BeautifulSoup vs lxml: which parser is faster.

Frequently Asked Questions

Why does reading a class attribute give me a list instead of a string? Because HTML defines class as a space-separated list of tokens, and BeautifulSoup reflects that faithfully. Test membership with "sale" in tag.get("class", []), and rebuild the original string with " ".join(...) if you need it — the [] default avoids a TypeError on elements that carry no class at all.

When should I use .string instead of .get_text()? Almost never in scraping. .string returns None as soon as a tag has more than one child, which is the normal case for any element containing inline markup, so it fails silently on real pages. Use .get_text(strip=True, separator=" ") unless you specifically need the "exactly one text child" semantics.

How do I get a heading's text without the nested badge inside it? Call .extract() on the badge to remove it and return it, then read the parent's text; that keeps the badge available if you want it. If the tree must stay untouched, iterate the parent's direct children and keep only the NavigableString pieces.

What is the safest way to turn a relative href into an absolute URL? Use urllib.parse.urljoin with the final response.url after redirects, which handles absolute, protocol-relative, root-relative and dot-relative forms in one call. Check first for a <base href> element, because when one is present it overrides the response URL for every relative link on the page.

Related

• Parsing HTML with BeautifulSoup — the parent guide on parser choice and tree navigation
• Step-by-Step Guide to Extracting Tables from HTML — the same accessors applied to tabular markup
• Selecting Elements with XPath and CSS Selectors — finding the elements before you read them
• Extracting JSON-LD and Structured Data — when the page already carries the data as structured markup