Selecting Elements by Visible Text in Python

When a page offers no usable attribute, the words on screen are the anchor, and this page goes through every way to match them — the deep dive behind the text-anchor step in Selecting Elements with XPath and CSS Selectors.

text() returns the loose text nodes only and silently drops anything wrapped in a child tag; string() and normalize-space() flatten the whole subtree.

The short version: use XPath, and use normalize-space(.) rather than text(). text() returns the individual text nodes that are direct children of an element, so it drops anything wrapped in a nested tag and keeps the template's raw whitespace. string(.) flattens the whole subtree into one string, and normalize-space(.) does that and then trims and collapses the whitespace, which is what almost every text comparison actually wants.

How XPath Sees Text

An element's content is not a single string in the document model. <p>In stock <span>(22)</span> today</p> holds three children: the text node "In stock ", the span element, and the text node " today". The (22) belongs to the span, not to the paragraph. Three XPath constructs give you three different views of that:

from lxml import html

node = html.fromstring("<p>In stock  <span>(22)</span>  today</p>")

print(node.xpath("//p/text()"))              # ['In stock  ', '  today']
print(node.xpath("//p//text()"))             # ['In stock  ', '(22)', '  today']
print(node.xpath("string(//p)"))             # 'In stock  (22)  today'
print(node.xpath("normalize-space(//p)"))    # 'In stock (22) today'
print(node.text_content())                   # 'In stock  (22)  today'

//p/text() selects direct child text nodes only. //p//text() selects every descendant text node, but still as separate strings, so you would have to join them yourself. string() performs the join for you, in document order. normalize-space() applies string() and then strips leading and trailing whitespace and collapses every internal run of whitespace to one space.

The practical consequence: a predicate written as [text()="Add to basket"] compares against one text node at a time. If the button is <button>Add to <b>basket</b></button>, no single text node equals the whole label and the predicate fails. [normalize-space()="Add to basket"] compares against the flattened string value of the element and matches.

normalize-space() with no argument defaults to the context node, so inside a predicate normalize-space() and normalize-space(.) mean the same thing. Both are shorter and safer than normalize-space(text()), which normalises only the first text node.

Exact and Partial Matching

Two predicates cover almost everything. Equality gives you an exact match on the whole flattened label; contains() gives you a substring test.

from lxml import html

PAGE = """
<div class="panel">
  <button type="submit">  Add to basket </button>
  <a class="cta" href="/checkout">Proceed to checkout</a>
  <p class="availability">In stock (22 available)</p>
  <p class="availability">Out of stock</p>
</div>
"""
tree = html.fromstring(PAGE)

# Exact match on the whole visible label.
print(tree.xpath('//button[normalize-space()="Add to basket"]'))

# Substring match anywhere in the element's flattened text.
print(tree.xpath('//p[contains(normalize-space(), "In stock")]/text()'))

# Prefix match, useful for hrefs and codes.
print(tree.xpath('//a[starts-with(@href, "/check")]/@href'))

# Exclude a variant you do not want.
print(
    tree.xpath(
        '//p[contains(., "stock") and not(contains(., "Out of"))]/text()'
    )
)

Prefer equality when the label is a fixed string you control the expectation for, because contains() is greedy in a way that bites: contains(., "stock") matches both In stock (22 available) and Out of stock. The not() guard above is the usual repair, and it is a sign that an exact match on a different node would be better.

contains() takes the string value of its first argument, so contains(., "…") already sees through nested tags. Writing contains(text(), "…") narrows it to the first text node and reintroduces the problem normalize-space() was solving.

One more function is worth remembering: starts-with(). XPath 1.0, which is what lxml implements, has no ends-with(). The idiomatic substitute is a substring() comparison, though in practice a Python-side check on the extracted string is clearer than an XPath acrobatic.

Case-Insensitive Matching

XPath 1.0 has no lower-case() either — that arrived in XPath 2.0, which lxml does not support. The portable technique is translate(), which maps characters one-for-one from a source alphabet to a target alphabet.

Whitespace first, case second, comparison last — the order matters because translate cannot fix the padding that normalize-space removes.

from lxml import html

UPPER = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
LOWER = "abcdefghijklmnopqrstuvwxyz"

PAGE = """
<ul>
  <li><button>  ADD TO BASKET </button></li>
  <li><button>Add to Basket</button></li>
  <li><button>Remove</button></li>
</ul>
"""
tree = html.fromstring(PAGE)

expression = (
    f'//button[contains(translate(normalize-space(.), "{UPPER}", "{LOWER}"),'
    f' "add to basket")]'
)
print([b.text_content().strip() for b in tree.xpath(expression)])

Order matters. normalize-space() runs first because translate() cannot fix leading padding — it maps characters, it does not trim. Lower-casing second means the literal you compare against must also be lower-case, which is easy to get wrong when the expression is assembled from a variable.

translate() is ASCII-only as written. Accented characters pass through unchanged, so Café lower-cases to café but CAFÉ stays CAFÉ unless you extend both alphabets with the accented pairs you care about. For anything beyond Latin-1 this becomes unmanageable, and the better answer is to select a slightly wider set of candidate nodes with a cheap predicate and finish the comparison in Python with str.casefold(). Encoding problems that make these comparisons fail for a different reason are covered in Fixing Common Unicode Errors in Python Scraping.

translate() has a second, unrelated use worth knowing: deleting characters. When the target string is shorter than the source, the surplus source characters are removed. translate(., "£,", "") strips currency symbols and thousands separators inside the expression, which occasionally saves a cleanup pass.

Matching Across Nested Inline Tags

Marketing markup wraps fragments of a label in <b>, <em>, <span> or a screen-reader-only element. Two rules keep matches working.

First, always compare against the element's string value, not a text node — that is the normalize-space() habit above. Second, be aware that string() includes text from elements the user cannot see. A button rendered as <button>Add to basket<span class="sr-only"> — Nod on the Tune</span></button> has a visible label of "Add to basket" and a string value that includes the product name. An exact match fails; contains() succeeds. This is the one situation where contains() is the correct choice rather than the lazy one.

from lxml import html

node = html.fromstring(
    '<button>Add to basket<span class="sr-only"> — Nod on the Tune</span></button>'
)

print(node.xpath('//button[normalize-space()="Add to basket"]'))          # []
print(node.xpath('//button[starts-with(normalize-space(), "Add to")]'))   # [<Element button>]

# Compare only the text the button owns directly, ignoring the nested span.
print(node.xpath('//button[normalize-space(text())="Add to basket"]'))    # [<Element button>]

The third form is the precise tool: normalize-space(text()) normalises the first direct child text node and ignores descendants entirely. It is exactly wrong for <button>Add to <b>basket</b></button> and exactly right for the screen-reader case, which is why you have to look at the markup rather than pick a favourite.

Elements hidden with display: none are also in the string value, because the parser knows nothing about CSS. A price hidden behind a "show price" toggle is in your match whether you want it or not, and no static parser can tell you it was invisible.

What the Other Libraries Do

Every library exposes text matching differently, and the differences are not cosmetic.

Only BeautifulSoup's string argument compares a single text node; the other three compare the flattened text of the element, which is why they see through nested inline tags.

BeautifulSoup's string= argument (and the older text=) filters on NavigableString objects — individual text nodes — which is why it behaves unlike XPath:

from bs4 import BeautifulSoup

soup = BeautifulSoup(
    "<button>Add to <b>basket</b></button><button>Remove</button>",
    "lxml",
)

# Matches nothing: no single text node equals the full label.
print(soup.find_all("button", string="Add to basket"))

# Matches: 'Remove' is a lone text node.
print(soup.find_all("button", string="Remove"))

# soupsieve's CSS pseudo-class compares the element's full text.
print(soup.select("button:-soup-contains('Add to basket')"))

# Or filter in Python, which is the clearest option in BeautifulSoup.
print([b for b in soup.find_all("button") if "Add to basket" in b.get_text(" ", strip=True)])

The separator argument in that last line is not optional. get_text(strip=True) strips each string individually and then concatenates them with nothing in between, producing "Add tobasket" for the nested button. get_text(" ", strip=True) inserts a space between fragments and gives "Add to basket", which is the string a user would read.

find_all(string=...) also returns the strings themselves rather than the elements when no tag name is given, which surprises people who expected a list of tags. Where a tag name is given, BeautifulSoup requires the element to have exactly one child that is a string for the filter to apply — hence the empty result above. The :-soup-contains() pseudo-class is the closest equivalent to XPath's contains(.), but it is a soupsieve extension and is not valid CSS anywhere else. The rest of the library's API is covered in Parsing HTML with BeautifulSoup.

For pages where the text only exists after JavaScript runs, Playwright's get_by_text matches on the rendered accessible text and auto-waits for it to appear:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page(
        user_agent=(
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
            "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
        )
    )
    page.goto("https://books.toscrape.com/", wait_until="domcontentloaded")
    page.get_by_text("Travel", exact=True).click()
    print(page.get_by_role("heading", level=1).inner_text())
    browser.close()

get_by_text defaults to a case-insensitive, whitespace-normalised substring match, and exact=True switches it to a case-sensitive full match that still normalises whitespace. It also ignores elements that are not rendered, which is the one genuine advantage over XPath — a static parser cannot know that a node is hidden. get_by_role("button", name="Add to basket") is usually better still, because it matches the accessible name that assistive technology exposes. The wider browser workflow is in Using Playwright for Modern Web Automation.

Edge Cases and Caveats

Non-breaking spaces. Templates emit   (U+00A0), which XPath's normalize-space() does not treat as whitespace — it only collapses space, tab, carriage return and line feed, so the normalised result still contains a literal \xa0. In stock joined by a non-breaking space therefore never equals In stock joined by an ordinary one. Build the predicate as normalize-space(translate(., NBSP, " ")), where NBSP is a Python constant holding "\u00a0", or clean the extracted value with value.replace("\xa0", " ") before comparing.
Invisible formatting characters. Soft hyphens (U+00AD) and zero-width spaces (U+200B) are common in CMS output and break exact matches while looking identical on screen. Print repr() of the extracted string when a match inexplicably fails.
Typographic punctuation. Don't with a right single quotation mark (U+2019) is not Don't with an ASCII apostrophe. Match on a substring that excludes the punctuation rather than fighting it.
Localisation. Any text anchor is a locale anchor. If the site serves translated labels by Accept-Language or geography, keep a per-locale label map instead of hard-coding one language, and pass the label as an XPath variable.
Quoting. A label containing an apostrophe breaks a single-quoted XPath literal. Pass it as a variable — tree.xpath("//th[normalize-space()=$label]", label=value) — rather than interpolating.
Performance on large documents. A predicate such as //*[contains(., "x")] evaluates the string value of every element in the document, and because ancestors contain their descendants' text it also matches html and body. Always constrain the node test (//button[...], //td[...]) and scope to a container where possible; the reasoning behind scoped selectors is in Writing Selectors That Survive a Redesign.

Frequently Asked Questions

Why does my text()="Next" selector fail when the button clearly says Next? Almost always whitespace or nesting. The template renders the label with surrounding newlines, so the text node is "\n Next\n", or the word is wrapped in a nested tag so no single text node holds it. Use //button[normalize-space()="Next"], which compares against the trimmed, flattened string value of the element.

Can CSS selectors match text in Python? Standard CSS cannot, but soupsieve adds a :-soup-contains() pseudo-class that BeautifulSoup exposes through .select(). It is library-specific, so it will not work in lxml, in a browser console, or in Scrapy's CSS support. For anything portable, use XPath.

How do I match text case-insensitively without XPath 2.0? Wrap the value in translate() with an upper-case source alphabet and a lower-case target alphabet, comparing against a lower-case literal. It handles ASCII cleanly; for accented or non-Latin scripts, select candidate nodes with a looser predicate and finish the comparison in Python with str.casefold().

Is matching on text more or less stable than matching on a class name? More stable than a utility or hashed class, less stable than a data-testid or itemprop. Visible copy changes deliberately and infrequently, whereas generated class names change on unrelated deploys. Use a partial match on a distinctive fragment rather than the full sentence to absorb small copy edits.

Selecting Elements with XPath and CSS Selectors — the parent topic and the wider selector method
XPath vs CSS Selectors: Which Should You Use? — why text matching is one of the two reasons to pick XPath
Extracting Data with Regular Expressions — pulling a value out of the text once you have selected the node