Selecting Elements with XPath and CSS Selectors

Every scraper eventually breaks on a selector rather than on a network error, and this guide — part of The Complete Guide to Python Web Scraping — covers the two selector languages Python gives you, when each one is the right tool, and how to write expressions that survive the next front-end deploy. The scope here is deliberately narrow: locating the right nodes in a parsed document. Turning those nodes into clean fields is covered in Parsing HTML with BeautifulSoup.

Both languages operate on the same parsed tree. Whether you write div.card > a.title or //div[@class="card"]/a, the library walks the document object model that the parser has already built, and both return a list of element nodes. The difference is expressive power, not capability at the fetch layer — so the choice is about which expression states your intent most durably, not about which one is faster in some abstract sense.

When to Use CSS and When to Use XPath

CSS selectors are shorter, more readable, and familiar to anyone who has written a stylesheet. They handle the overwhelming majority of extraction work: match by tag, class, id, attribute, direct child, descendant, and position among siblings. Reach for XPath when you need something CSS cannot express at all.

The practical rule: start with CSS, and switch to XPath the moment your selector would otherwise depend on brittle positional structure. A very common trigger is the anchor-then-climb pattern — you can identify a label reliably ("Price"), but the value you want is in a neighbouring or parent element. CSS has no way to go up or backwards; XPath does.

This asymmetry is the single most useful thing to know about the two languages. CSS moves down the tree (descendants) and forwards among siblings (~, +). XPath adds parent::, ancestor::, preceding-sibling::, and preceding::, which lets you anchor on the most stable thing on the page and then navigate relative to it. When a page is a wall of divs with generated class names, that anchor is often the only stable landmark you have.

Prerequisites

Python 3.10 or newer, plus a parser and a selector engine. lxml provides the XPath engine (a binding to libxml2) and is also the fastest HTML parser available to Python; beautifulsoup4 brings a forgiving API and, through soupsieve, a very complete CSS implementation. Install all three:

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install requests beautifulsoup4 lxml cssselect

cssselect is what lets you run CSS selectors through the lxml tree via .cssselect(); it translates them to XPath internally. If you are working inside Scrapy, all of this is already present — Scrapy's Selector exposes both .css() and .xpath() on the same object, as described in Web Scraping with Scrapy. Set up the environment properly first if you have not already: Setting Up Your Python Scraping Environment.

Step-by-Step: Building a Selector That Lasts

1. Fetch the page and parse it once

Parsing is not free. Build the tree once and run every selector against that object rather than re-parsing per field. Always send a realistic User-Agent; the header conventions are covered in Understanding HTTP Requests and Responses.

import requests
from lxml import html

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-GB,en;q=0.9",
}


def fetch_tree(url: str) -> html.HtmlElement:
    response = requests.get(url, headers=HEADERS, timeout=20)
    response.raise_for_status()
    return html.fromstring(response.text)


tree = fetch_tree("https://books.toscrape.com/catalogue/page-1.html")
print(len(tree.cssselect("article.product_pod")))

html.fromstring returns an HtmlElement that supports both .cssselect() and .xpath(). Keep the parsed tree in a variable and pass it around; re-parsing the same HTML three times to extract three fields is one of the most common avoidable costs in a scraper.

2. Pick the most contract-like anchor available

Before writing anything, look at the markup and rank the available hooks. Attributes that the application itself depends on — id, data-testid, itemprop, name, role — change rarely, because changing them breaks the site's own tests or its structured data. Utility classes change every time someone adjusts spacing. Hashed class names (css-1x7yz9) change on every build of the bundler.

# Best: an attribute the application itself relies on.
tree.cssselect("[data-testid='product-price']")

# Good: a semantic attribute used for structured data.
tree.cssselect("[itemprop='price']")

# Acceptable: a meaningful, human-authored class name.
tree.cssselect("p.price_color")

# Fragile: styling utilities that carry no meaning.
tree.cssselect("div.mt-4 > span.text-sm")

# Worst: a hashed class or an absolute position.
tree.cssselect("div.css-1x7yz9 > div:nth-child(3) > span")

If a page exposes itemprop attributes, prefer them — they are part of the site's own machine-readable contract, and the same markup usually feeds a JSON-LD block you can read directly, as covered in Extracting JSON-LD and Structured Data.

3. Anchor on text, then climb with XPath

When there is no usable attribute, the visible label is often the most stable thing on the page. A definition list, a specification table, or a key-value panel can all be read by finding the label and then stepping to its value.

from lxml import html


def spec_value(tree: html.HtmlElement, label: str) -> str | None:
    """Read the value cell that sits next to a label cell in a spec table."""
    matches = tree.xpath(
        "//th[normalize-space(text())=$label]/following-sibling::td[1]",
        label=label,
    )
    return matches[0].text_content().strip() if matches else None


tree = html.fromstring(
    """
    <table>
      <tr><th>UPC</th><td>a897fe39b1053632</td></tr>
      <tr><th>Availability</th><td>In stock (22 available)</td></tr>
    </table>
    """
)
print(spec_value(tree, "UPC"))
print(spec_value(tree, "Availability"))

Two details make this robust. normalize-space() collapses the whitespace that templating engines scatter through markup, so a label rendered as \n UPC\n still matches. Passing label=label as an XPath variable rather than formatting it into the string avoids quoting bugs the moment a label contains an apostrophe — the XPath equivalent of using parameterised SQL.

4. Scope every selector to its record

The most damaging selector bug is not one that returns nothing; it is one that returns something from the wrong record. Always select the container first, then run field selectors relative to that container. In XPath, the leading dot matters: .// searches inside the current node, while // restarts at the document root and will happily return the first price on the page for every product in your loop.

import requests
from lxml import html

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    )
}


def scrape_listing(url: str) -> list[dict[str, str]]:
    response = requests.get(url, headers=HEADERS, timeout=20)
    response.raise_for_status()
    tree = html.fromstring(response.text)

    rows: list[dict[str, str]] = []
    for card in tree.cssselect("article.product_pod"):
        title = card.xpath("./h3/a/@title")
        price = card.cssselect("p.price_color")
        rows.append(
            {
                "title": title[0].strip() if title else "",
                "price": price[0].text_content().strip() if price else "",
            }
        )
    return rows


for row in scrape_listing("https://books.toscrape.com/catalogue/page-1.html")[:3]:
    print(row)

Note ./h3/a/@title rather than //h3/a/@title. The relative form is bound to card; the absolute form would return the first title in the whole document twenty times. This is the number one cause of "my scraper produced 20 identical rows".

5. Extract text deliberately

.text returns only the text node directly under the element and stops at the first child tag. .text_content() concatenates every descendant text node. Neither is universally correct — a price rendered as <p>£<span>24.00</span></p> gives "£" from .text and "£24.00" from .text_content(), while a card title with a hidden screen-reader span gives you unwanted extra words from .text_content().

from lxml import html

node = html.fromstring('<p class="price">£<span class="amount">24.00</span></p>')
print(repr(node.text))                      # '£'
print(repr(node.text_content()))            # '£24.00'
print(repr(node.xpath("string(.)")))        # '£24.00'
print(repr(node.xpath("normalize-space(.)")))  # '£24.00' with whitespace collapsed

normalize-space(.) is usually what you want for a leaf value: it returns a single string with runs of whitespace collapsed and the ends trimmed, which removes an entire class of .strip() calls from your parsing code. The full set of text-matching options, including case-insensitive comparison and the differences between the libraries, is covered in Selecting Elements by Visible Text in Python.

6. Run the same selectors inside Scrapy

Scrapy wraps both languages in one object. response.css() and response.xpath() return a SelectorList, and every element of that list is itself a selector you can query again — which is exactly the container-then-field pattern from step 4, expressed more compactly. The two extraction verbs are .get(), which returns the first match as a string or None, and .getall(), which returns every match as a list of strings.

import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]
    custom_settings = {
        "USER_AGENT": (
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
            "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
        ),
        "DOWNLOAD_DELAY": 0.5,
    }

    def parse(self, response: scrapy.http.Response):
        for card in response.css("article.product_pod"):
            yield {
                "title": card.css("h3 a::attr(title)").get(default=""),
                "price": card.css("p.price_color::text").get(default="").strip(),
                "stock": card.xpath(
                    "normalize-space(.//p[contains(@class, 'instock')])"
                ).get(),
                "cover": response.urljoin(card.css("img::attr(src)").get(default="")),
            }

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

Three things in that spider are worth isolating. First, card.css(...) and card.xpath(...) are relative automatically for CSS, but not for XPath: card.xpath("//p") still searches the whole document, so the leading . in .//p is as mandatory here as it is in raw lxml. Second, ::text and ::attr(name) are Scrapy extensions to CSS, not standard CSS — they exist precisely so you can stay in one language for the common cases instead of switching to XPath just to read an attribute. Third, get(default="") collapses the empty-list problem into a single expression, which is why Scrapy spiders rarely contain the index guards you need with bare lxml.

.getall() is the right verb whenever a field is legitimately multi-valued — categories, tags, image URLs, a list of authors. Reaching for .get() on a multi-valued field is a quiet data-loss bug that no exception will ever report.

# One value, or None if the node is missing.
card.css("h3 a::attr(title)").get()

# Every value, as a list. Empty list if nothing matched.
card.css("ul.tags li::text").getall()

# re_first applies a regex to the matched strings without a second parse.
card.css("p.instock::text").re_first(r"(\d+) available")

.re() and .re_first() run a regular expression over the extracted strings, not over the raw HTML, which is the safe way to combine the two techniques described in Extracting Data with Regular Expressions. If you find yourself writing a regex against response.text instead, you have skipped the parser and inherited every fragility that comes with it.

Outside a spider, the same object is available standalone, which makes it useful for testing a selector against saved markup:

from scrapy.selector import Selector

sel = Selector(text="<div class='card'><h3>Nod on the Tune</h3><p>£24.00</p></div>")
print(sel.css("h3::text").get())
print(sel.xpath("normalize-space(//p)").get())

7. Register the selectors you reuse

lxml parses an XPath string every time you call .xpath() with it. etree.XPath compiles once and can then be applied to any tree, which also gives you a natural place to keep the expression and its documentation together.

from lxml import etree, html

PRICE = etree.XPath("normalize-space(.//p[@class='price_color'])")
TITLE = etree.XPath("string(./h3/a/@title)")

tree = html.fromstring(
    """
    <article class="product_pod">
      <h3><a title="Nod on the Tune">Nod on the Tune</a></h3>
      <p class="price_color">£24.00</p>
    </article>
    """
)
card = tree.cssselect("article.product_pod")[0]
print(TITLE(card), PRICE(card))

Compiled expressions are also the cleanest way to hold a per-site selector table in one module, so that a redesign is a one-file change rather than a search through the codebase.

XPath Functions Worth Knowing

Most XPath written by scrapers uses about six functions. Learning them removes the temptation to fall back on positional selectors.

Two syntactic details cause more confusion than the functions themselves.

The union operator | joins two node sets into one result, in document order. It is the way to write "the price is in one of two possible places" without running two separate queries and merging in Python:

from lxml import html

tree = html.fromstring(
    """
    <div>
      <span class="sale-price">£19.99</span>
      <p class="legacy_price">£24.00</p>
    </div>
    """
)
prices = tree.xpath('//span[@class="sale-price"] | //p[@class="legacy_price"]')
print([p.text for p in prices])

The union is evaluated against the document once, so a two-branch expression costs roughly what a one-branch expression costs. Use it when a site is mid-migration and both the old and the new markup appear on different pages of the same crawl.

Predicate order matters. (//tr)[2] and //tr[2] are different expressions. The first collects every tr in the document and takes the second one overall. The second takes, for every parent, the tr that is second among its children — which on a page with three tables returns three nodes. Wrapping the path in parentheses before indexing is what makes "the second table on the page" expressible at all, and it is a capability CSS has no equivalent for.

last() deserves a similar warning: //tr[last()] returns the last row of each table, not the last row on the page. (//tr)[last()] returns exactly one node.

Finally, contains() on the class attribute is a common trap. //div[contains(@class, "price")] also matches class="price-was-struck" and class="unpriced". The correct token test pads both sides with spaces, which is exactly what cssselect generates when you write div.price:

tree.xpath(
    '//div[contains(concat(" ", normalize-space(@class), " "), " price ")]'
)

If that expression looks tedious, that is a good argument for writing div.price in CSS and letting the translator produce it — the trade-off is examined in XPath vs CSS Selectors: Which Should You Use?.

A Debugging Workflow for a Selector That Returns Nothing

An empty result has a small number of causes, and working through them in a fixed order is faster than guessing. The order below moves from cheapest check to most expensive.

1. Confirm you are looking at the same document the browser is. DevTools shows the rendered DOM after JavaScript, style injection and the browser's own markup repair. Your parser sees the bytes on the wire. The one-line test is to save the response and open it:

import pathlib

import requests

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-GB,en;q=0.9",
}

response = requests.get("https://books.toscrape.com/", headers=HEADERS, timeout=20)
path = pathlib.Path("debug/last_response.html")
path.parent.mkdir(exist_ok=True)
path.write_text(response.text, encoding="utf-8")
print(response.status_code, len(response.text), response.headers.get("content-type"))

Open debug/last_response.html in a browser with JavaScript disabled, or search it with grep. If the string you expect is not in the file, no selector will ever find it and the problem belongs to the fetch layer, not the parse layer.

2. Search for the value, not the container. Rather than testing your selector, grep the saved file for a literal you know is on the page — a price, a product code, a heading. That tells you three things at once: whether the value is present, what element actually holds it, and whether the surrounding markup matches what DevTools showed you.

3. Widen the selector one step at a time. If article.product_pod h3 a returns nothing, test article.product_pod h3, then article.product_pod, then article. The first level that returns results is where the assumption broke. This bisection takes four print(len(...)) calls and replaces an hour of staring at markup.

4. Check for the invisible tag. Browsers insert tbody into every table that lacks one; lxml does not. A selector copied from Chrome's "Copy selector" menu almost always contains a tbody that does not exist in your tree. Namespaced documents are the mirror image of this problem — XHTML served as application/xhtml+xml puts every element in a namespace, so //div matches nothing until you use //*[local-name()="div"].

5. Assert on the field, then let the run fail loudly. The worst outcome is not a crash; it is a scraper that writes 40,000 rows with an empty price column for a week. Every extraction function should state what it requires:

def parse_card(card) -> dict[str, str]:
    title = card.xpath("string(./h3/a/@title)").strip()
    price = card.xpath("normalize-space(.//p[@class='price_color'])")
    if not title or not price:
        raise ValueError(f"selector miss: title={title!r} price={price!r}")
    return {"title": title, "price": price}

Raising per record is too aggressive for a large crawl, where a handful of malformed pages is normal. The scalable version counts: track the fill rate of each field over the run, and alert when a field drops below a threshold you set from a known-good baseline. The full method — including snapshot-testing selectors against saved HTML so a break shows up in CI rather than in your dataset — is set out in Writing Selectors That Survive a Redesign.

6. Only now reach for a browser. If the value genuinely is not in the HTML, the page builds it in the client. The cheaper fix is usually to find the JSON endpoint the page calls, as described in Reverse-Engineering Private APIs; rendering the page is the fallback, not the first move.

Performance and Scaling Considerations

Selector cost is rarely the bottleneck in a scraper — the network is — but it becomes visible in two situations: very large documents, and very high page throughput.

• Compile expressions you reuse. lxml lets you compile an XPath once with etree.XPath("...") and call it against many trees. On a crawl of tens of thousands of pages this removes the repeated parse of the expression itself.
• Prefer a scoped search over a global one. // scans the entire document. When you already hold a container node, .// limits the walk to that subtree, which on a page with thousands of nodes is measurably cheaper and, more importantly, correct.
• Avoid text() when you mean the whole value. //p[@class="price"]/text() returns a list of text nodes and silently drops anything nested inside a child tag. string(.) collects everything.
• Parse once per response. If a page feeds five extraction functions, pass the tree, not the HTML string. Parsing a 200 KB page with lxml is fast but not free, and at high concurrency — see Asynchronous Scraping with Asyncio and HTTPX — parsing is CPU work that competes with everything else in the event loop.
• Consider lxml over html.parser for volume. The parser choice affects both speed and the shape of the tree you get for broken markup; the measured differences are covered in BeautifulSoup vs lxml: Which Parser Is Faster.
• Beware predicates that stringify every node. //*[contains(., "Total")] computes the string value of every element in the document, and because an ancestor's string value contains all of its descendants' text, it also matches html and body. Always constrain the node test — //td[contains(., "Total")] — so the engine only builds the strings you asked about.
• Release the tree when you are done with it. lxml elements keep a reference to the whole document, so holding a single matched node in a long-lived list keeps the entire parsed page in memory. Extract the strings you need and drop the element; on a long-running crawler this is the difference between a flat memory profile and a slow climb.
• Profile before optimising a selector at all. In a scraper that fetches pages, a CPU profile almost always shows parsing above querying and both far below waiting on sockets. The optimisation that moves throughput is concurrency at the request layer; selector micro-tuning is worth doing only after a profile names it.

Common Errors and Fixes

IndexError: list index out of range — you indexed [0] on an empty result. Selectors return lists, and a missing element is a normal outcome on a real site, not an exception. Guard every access:

def first_text(node, expression: str, default: str = "") -> str:
    matches = node.xpath(expression)
    if not matches:
        return default
    match = matches[0]
    return match.strip() if isinstance(match, str) else match.text_content().strip()

XPathEvalError: Invalid expression — usually an unbalanced bracket, or a quote inside a quoted literal (//a[@title="Bob's book"] parsed with the same quote character). Use XPath variables ($title) and pass the value as a keyword argument instead of building strings.

A selector works in the browser console but returns nothing in Python — three usual causes. First, the browser executed JavaScript and you did not: the element does not exist in the raw HTML at all, which means you need the rendering path in Using Playwright for Modern Web Automation or the API behind the page. Second, the browser's parser repaired broken markup differently from lxml, so tbody exists in DevTools but not in your tree — never hard-code tbody in a selector. Third, the server returned a different page to your client, typically a consent wall or a block page, which you can confirm by printing len(response.text) and the first 200 characters.

Everything matches, but every row is identical — a missing leading dot on a relative XPath, as described in step 4.

Namespace errors on XML feeds — XHTML and XML documents carry namespaces, so //item matches nothing. Either pass a namespaces= map to .xpath(), or match on the local name: //*[local-name()="item"]. Feed handling is covered further in Parsing JSON and XML Responses.

Frequently Asked Questions

When is it worth compiling an expression with etree.XPath? Whenever the same expression runs against many documents — which is every crawl. Compiling moves expression parsing out of the per-page loop and gives you one module where every selector for a site lives. For a handful of calls on one page it makes no measurable difference.

How do I select an element by its visible text? Use XPath: //button[normalize-space(text())="Next page"] for an exact match, or //td[contains(., "In stock")] for a partial one. Prefer normalize-space() over raw text() so that template whitespace does not break the match.

Why does my selector break every few weeks? It is almost certainly anchored on presentation rather than meaning — a utility class, a hashed class name, or an nth-child position. Re-anchor on data-*, itemprop, id, or visible label text, and add a check that alerts when a field comes back empty so you find out before the dataset is polluted.

Should I write one long selector or several short ones? Several short ones, scoped to a container. A long chain fails as a unit and tells you nothing about which link broke; a container plus per-field selectors lets you log exactly which field went missing, which is what makes a scraper maintainable at scale.

What is the difference between .get() and .getall() in Scrapy?.get() returns the first match as a string, or None when nothing matched, and accepts a default= argument. .getall() returns every match as a list of strings and returns an empty list when nothing matched. Use .getall() for anything legitimately multi-valued, because .get() on a multi-valued field discards data without raising.

Related

• The Complete Guide to Python Web Scraping — the parent guide to this section
• XPath vs CSS Selectors: Which Should You Use? — the head-to-head comparison in detail
• Writing Selectors That Survive a Redesign — the maintenance method, fixtures and alerting
• Selecting Elements by Visible Text in Python — text predicates in depth
• Handling Pagination and Infinite Scroll — walking from one listing page to the next