Writing Selectors That Survive a Redesign

Selectors are the part of a scraper that rots, and this page turns the principles in Selecting Elements with XPath and CSS Selectors into a method you can apply field by field.

The method has four moves. Rank the available anchors and always take the most contract-like one. Select the record container once and make every field expression relative to it. Assert that each field produced a value, and track the fill rate so a partial break is visible. Finally, freeze a copy of the page as a test fixture so a selector regression fails in continuous integration rather than in your dataset three weeks later.

Why Selectors Break

A redesign does not change a page uniformly. It changes the parts that carry presentation and leaves the parts that carry meaning, because the meaning is what the application's own code, tests and structured data depend on. That asymmetry is the whole opportunity.

A modern build pipeline generates class names. Utility-first CSS produces class="mt-4 flex items-center text-sm", where every token describes appearance; adjusting a margin rewrites the selector you anchored on. CSS-in-JS produces hashed names like css-1x7yz9 that change whenever the surrounding source file changes, sometimes on every deploy. Meanwhile id="product-detail", data-testid="add-to-cart" and itemprop="price" survive, because deleting them breaks the site's own test suite or its rich results in search.

The second failure mode is structural rather than nominal. A selector like div > div:nth-child(3) > span encodes a position, and positions move the moment a designer inserts a wrapper. Anything positional is a countdown timer.

The third is subtler: the selector still matches, but it matches the wrong node. A field-level expression written with // instead of .// looks correct in testing on a single-record page and silently returns record one for every row on a listing page. This class of bug produces plausible data, which is far worse than producing none.

Rank Your Anchors Before You Write Anything

Open the markup and list every hook the target element offers, then take the highest-ranked one available. The ranking is stable across sites:

Ranks 1 to 5 are worth building on. Rank 6 is acceptable with a test around it. Ranks 7 and 8 should appear in your codebase only with a comment admitting that no better anchor exists.

A practical trick when a site offers nothing: check whether the same values appear in a JSON-LD block. Many commerce templates emit both the visible markup and a <script type="application/ld+json"> payload from the same data, and the JSON keys are far more stable than the markup around them.

Scope Every Field to a Record Container

The nesting shown above is the single highest-value habit in this article. Select the container once, then run short relative expressions inside it. The container selector is the only one that has to be globally correct; the field expressions only have to be correct within a card, which makes them shorter and therefore less coupled to layout.

from lxml import html

FIXTURE = """
<div id="listing">
  <article class="product_pod" data-sku="a897fe39b1053632">
    <h3><a href="/catalogue/nod/index.html" title="Nod on the Tune">Nod on the Tune</a></h3>
    <p class="price_color">£24.00</p>
    <p class="instock availability">In stock (22 available)</p>
  </article>
  <article class="product_pod" data-sku="90fa61229261140a">
    <h3><a href="/catalogue/hush/index.html" title="Hush Now">Hush Now</a></h3>
    <p class="price_color">£13.99</p>
    <p class="instock availability">In stock (5 available)</p>
  </article>
</div>
"""


def parse_records(markup: str) -> list[dict[str, str]]:
    tree = html.fromstring(markup)
    records: list[dict[str, str]] = []
    for card in tree.cssselect("article[data-sku]"):
        records.append(
            {
                "sku": card.get("data-sku", ""),
                "title": card.xpath("string(./h3/a/@title)"),
                "url": card.xpath("string(./h3/a/@href)"),
                "price": card.xpath("normalize-space(.//p[@class='price_color'])"),
                "stock": card.xpath("normalize-space(.//p[contains(@class,'instock')])"),
            }
        )
    return records


for record in parse_records(FIXTURE):
    print(record)

Note article[data-sku] as the container: it uses a rank-1 anchor and additionally guarantees that every card carries a stable primary key, which is what lets you deduplicate and diff runs later. Note also that every field expression starts with . — string(./h3/a/@title) rather than string(//h3/a/@title). Drop that dot and both records report the first title.

string() and normalize-space() are doing real work here too. string(./h3/a/@title) returns "" rather than raising when the attribute is missing, and normalize-space() removes the template whitespace that would otherwise leave '\n £24.00\n ' in your database.

Anchor on Text When Attributes Run Out

Specification tables, definition lists and key-value panels usually have no per-field attributes at all — every cell looks identical to a selector. The label is the anchor. XPath can match it and then step sideways to the value, which is the canonical case for the axes CSS lacks.

from lxml import html

SPEC = """
<table class="table">
  <tr><th>UPC</th><td>a897fe39b1053632</td></tr>
  <tr><th>Product Type</th><td>Books</td></tr>
  <tr><th>Availability</th><td>In stock (22 available)</td></tr>
</table>
"""

EXPRESSION = (
    "normalize-space(//th[normalize-space(text())=$label]/following-sibling::td[1])"
)


def spec_value(markup: str, label: str) -> str:
    return html.fromstring(markup).xpath(EXPRESSION, label=label)


for label in ("UPC", "Availability", "Missing Row"):
    print(f"{label:16} {spec_value(SPEC, label)!r}")

Two details make this durable. normalize-space() around the label means the match survives whatever indentation the templating engine emits. Passing label=$label as an XPath variable rather than formatting it into the string means a label containing an apostrophe cannot break the expression, in the same way parameterised SQL cannot be broken by a quote in a value. The mechanics of text predicates, including partial and case-insensitive matching, are covered in Selecting Elements by Visible Text in Python.

Text anchors have one obvious weakness: localisation. If the same template renders Disponibilité for French visitors, the selector must key off something else — usually a lang-independent attribute, or a per-locale label map you pass in.

Assert on Extraction and Alert on Empty Fields

A broken selector rarely raises. It returns an empty list, your code writes "", and the pipeline reports success. The fix is to make emptiness a measured signal.

At the record level, validate. At the run level, count. Both are cheap:

from collections import Counter

REQUIRED = ("sku", "title", "price")


def validate(records: list[dict[str, str]], threshold: float = 0.95) -> dict[str, float]:
    filled: Counter[str] = Counter()
    for record in records:
        for field, value in record.items():
            if value.strip():
                filled[field] += 1

    total = len(records) or 1
    rates = {field: filled[field] / total for field in records[0]} if records else {}
    for field, rate in rates.items():
        if field in REQUIRED and rate < threshold:
            raise SystemExit(f"fill rate for {field} is {rate:.0%}, expected >= {threshold:.0%}")
    return rates

Raising per record is too strict for a large crawl — real sites have out-of-stock items, missing images and the occasional malformed page. Raising on the aggregate is right: a required field at 97 per cent is normal variation, and the same field at 2 per cent is a broken selector. Baseline the threshold from a known-good run rather than guessing, and treat a sudden change in either direction as suspicious. Wiring these counts into a dashboard and a pager is the subject of Monitoring and Alerting for Scrapers, and the broader failure taxonomy is in Detecting Silent Scraper Failures.

Snapshot-Test Selectors Against a Saved Fixture

Fill rates tell you something broke after the run. A fixture test tells you before it. Save one representative page to disk, commit it, and assert your parser's output against it — the test costs no network and runs in milliseconds.

# tests/test_selectors.py
from pathlib import Path

import pytest

from myscraper.parsers import parse_records

FIXTURE = Path(__file__).parent / "fixtures" / "listing_page.html"


@pytest.fixture(scope="module")
def records() -> list[dict[str, str]]:
    return parse_records(FIXTURE.read_text(encoding="utf-8"))


def test_container_selector_finds_every_card(records: list[dict[str, str]]) -> None:
    assert len(records) == 20, "product container selector matched the wrong count"


@pytest.mark.parametrize("field", ["sku", "title", "url", "price", "stock"])
def test_every_field_is_populated(records: list[dict[str, str]], field: str) -> None:
    missing = [r["sku"] for r in records if not r[field].strip()]
    assert not missing, f"{field} empty for {len(missing)} record(s): {missing[:3]}"


def test_records_are_distinct(records: list[dict[str, str]]) -> None:
    titles = {r["title"] for r in records}
    assert len(titles) == len(records), "records repeat — a field selector is unscoped"

The third test is the one people forget, and it is the one that catches the missing-dot bug. If a field expression escapes its container, every record gets record one's title and the set of distinct titles collapses to one. That assertion turns an invisible data-quality disaster into a red build.

Refresh the fixture deliberately, not automatically. A scheduled job that re-downloads the page and fails the build when the saved copy no longer matches the live one gives you an early warning of a redesign, days before the data goes wrong; running that on a timer is straightforward with Scheduling Scrapers with Cron and GitHub Actions. Keep fixtures small — one listing page and one detail page per site is usually enough — and strip anything personal before committing them.

Edge Cases and Caveats

• A/B tests serve two markups. Some visitors get the new template, some the old. A selector that works from your laptop can fail from a datacentre IP. Write the container selector as a union of both forms (//article[@data-sku] | //li[@data-product-id]) until the rollout finishes.
• Consent walls and block pages parse fine. They are valid HTML with zero matching cards, so container count drops to zero rather than raising. Assert on the container count, not just on field presence.
• Fixtures drift into fiction. A fixture saved two years ago tests your parser against a site that no longer exists. Re-record on every intentional selector change and note the capture date in the filename.
• nth-child is occasionally the only option. Some tables genuinely carry no labels. Where you must use position, add a guard that checks a neighbouring cell's text still matches the expected header before trusting the column index.
• Scrapy shifts where the assertions live. In a spider, per-field validation belongs in an item pipeline or a schema model rather than in parse(), so that one bad page drops one item instead of killing the crawl. See Web Scraping with Scrapy for where those hooks sit.
• Encoding changes look like selector breaks. A page that switches from UTF-8 to a declared-but-wrong charset produces mojibake that fails an exact text match. Check the response encoding before rewriting the selector.

Frequently Asked Questions

How often should I expect to rewrite selectors for an actively developed site? For a site under continuous deployment, plan on a container or field selector needing attention every few months, and much more often if you anchored on utility or hashed classes. Anchoring on data-*, id and itemprop typically pushes that to once a year or less, because those attributes only change during a genuine rebuild.

Is it worth committing large HTML fixtures to the repository? Commit one small representative page per template, not a crawl. A single listing page is usually 100 to 300 KB, which is trivial for version control and gives you a deterministic, offline test that runs in milliseconds. Store anything larger with a fetch script that downloads it on demand instead.

What threshold should I set for a field fill-rate alert? Derive it from a known-good run rather than picking a round number. Take the observed rate over a full successful crawl and set the alert a few points below it, so normal variation from out-of-stock or incomplete records does not page anyone while a genuine break still trips immediately.

Should the container selector use CSS or XPath? Use CSS when the container has a class or data attribute, which is the usual case, and reserve XPath for containers you can only identify by their contents — for example the card that contains a particular badge. The full trade-off is set out in XPath vs CSS Selectors: Which Should You Use?.

Related

• Selecting Elements with XPath and CSS Selectors — the parent topic and the selector fundamentals
• Parsing HTML with BeautifulSoup — turning matched nodes into clean fields
• Extracting JSON-LD and Structured Data — the most redesign-proof anchor of all