Extracting Data with Regular Expressions

Part of The Complete Guide to Python Web Scraping, this guide covers the narrow but genuinely useful role of regular expressions in a scraping pipeline: pulling flat tokens out of text that a parser has already isolated, or out of text that was never structured markup in the first place. The scope excludes navigating HTML, which is what Parsing HTML with BeautifulSoup is for.

The reputation regex has in scraping is deserved but misattributed. It is a poor tool for hierarchy — HTML is a recursive language and regular expressions provably cannot count nesting depth — and an excellent tool for lexical patterns. A price, an ISO date, a session token embedded in a script, a numeric id inside a URL: all of these are flat token problems, and a compiled pattern solves them in microseconds with three lines of code.

When to Use Regex and When Not To

The decision is not "regex versus BeautifulSoup" as a general preference. It is a question you ask per field, and the answer usually changes within a single page.

There is a second axis worth naming: how the value is delimited. Regex is at its best when the target is bounded by characters that cannot occur inside it — a quote, a comma, a currency symbol, a line break. It is at its worst when the boundary is defined by something a human eye resolves visually, such as "the number in the third column" or "the text in the box on the right". Those are structural statements dressed up as lexical ones, and they belong to a selector, not a pattern. If you find yourself counting > characters to reach a field, the pattern has already become a parser and a worse one.

The most productive pattern in practice is a two-stage pipeline: use the parser to reduce the page to the smallest region containing your value, then use one small regex inside that region. That combination is robust, because the parser absorbs markup churn and the regex only ever sees a short, predictable string.

Prerequisites

Python 3.10 or newer. The re module is in the standard library, so nothing is strictly required, but the examples below use a parser for the two-stage pattern and regex for the timeout protection discussed later.

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install "requests>=2.31" "beautifulsoup4>=4.12" "lxml>=5.1" "regex>=2024.5.15"

The third-party regex package is a drop-in superset of re that adds a timeout= argument to matching calls, plus atomic groups and possessive quantifiers. On untrusted input those features are the difference between a slow request and a hung worker. See the environment guide, Setting Up Your Python Scraping Environment, if the virtual environment is not set up yet.

Step-by-Step: Building Patterns That Hold Up

1. Choose the right re function

The four functions differ in what they return and how much memory they use, and picking the wrong one is a common source of subtle bugs — re.match in particular anchors at position zero, which is almost never what a scraper wants.

import re

TEXT = "SKU: AB-1023 | SKU: CD-9981 | SKU: EF-4410"
PATTERN = r"SKU: ([A-Z]{2}-\d{4})"

print(re.search(PATTERN, TEXT).group(1))        # first match only -> AB-1023
print(re.findall(PATTERN, TEXT))                # every match as a list of strings
print([m.span() for m in re.finditer(PATTERN, TEXT)])  # lazy, with positions
print(re.match(PATTERN, TEXT))                  # None: anchored at index 0

findall materialises every match at once, which is fine for a page and wasteful for a 200 MB log file. finditer yields match objects one at a time and gives you .span(), .group() and .groupdict() per match, so it is the better default whenever the input is large or you need positions.

2. Make quantifiers lazy by default

.* consumes as much as possible and then backtracks, so on a line containing two of the same tag it captures everything between the first opener and the last closer. Appending ? makes the quantifier stop at the first opportunity.

import re

LINE = "<b>one</b> and <b>two</b>"

print(re.search(r"<b>(.*)</b>", LINE).group(1))    # one</b> and <b>two
print(re.search(r"<b>(.*?)</b>", LINE).group(1))   # one
print(re.findall(r"<b>([^<]*)</b>", LINE))         # ['one', 'two']

The third form is better still. A negated character class, [^<]*, cannot cross a tag boundary at all, so it never has to backtrack — it is both faster and impossible to over-match. As a rule, replace .*? with a negated class whenever you know which character terminates the field.

3. Name your groups

Numeric group indices break the moment someone adds a parenthesised alternation earlier in the pattern. (?P<name>…) makes the extraction self-describing and lets you build a dict in one call.

import re

ROW = re.compile(
    r"(?P<sku>[A-Z]{2}-\d{4})\s*\|\s*"
    r"(?P<currency>[£$€])(?P<price>\d+\.\d{2})\s*\|\s*"
    r"(?P<updated>\d{4}-\d{2}-\d{2})"
)

LINES = [
    "AB-1023 | £51.77 | 2026-07-14",
    "CD-9981 | $8.50 | 2026-07-15",
    "bad line with no data",
]

for line in LINES:
    match = ROW.search(line)
    print(match.groupdict() if match else f"no match: {line!r}")

groupdict() returns exactly the record you want, keyed by name. Note the use of a non-capturing structure throughout: any group you do not name should be written (?:…) so it never appears in findall output or shifts an index.

4. Compile once, at module scope

re caches the last 512 compiled patterns, so calling re.search(pattern, text) in a loop is not catastrophic — but the cache lookup still hashes the pattern string on every call, and the cache is cleared whenever it fills. Compiling at import time removes both costs and makes the patterns reviewable in one place.

import re
from typing import Final

PRICE: Final = re.compile(r"(?<![\d.])(\d{1,3}(?:,\d{3})*(?:\.\d{2})?)(?![\d.])")
EMAIL: Final = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")
NEXT_PAGE: Final = re.compile(r'href="([^"]*page-(\d+)\.html)"')


def prices_in(text: str) -> list[str]:
    return PRICE.findall(text)


print(prices_in("Was 1,299.00, now 999.99 (save 299.01)"))
print(EMAIL.findall("mail support@example.com or sales@example.org"))

The lookarounds in PRICE do real work: (?<![\d.]) prevents matching the tail of a longer number, and (?![\d.]) prevents stopping halfway through one. Without them, 1,299.00 yields three separate spurious matches.

5. Combine the parser and the pattern

This is the pattern to reach for most often. The parser narrows the document to one node; the regex extracts the token from that node's text. Neither tool is asked to do the other's job.

import re
import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-GB,en;q=0.9",
}
AVAILABLE = re.compile(r"\((\d+)\s+available\)")
AMOUNT = re.compile(r"([\d.]+)")


def stock_and_price(url: str) -> dict[str, int | float | None]:
    response = requests.get(url, headers=HEADERS, timeout=(5, 20))
    response.raise_for_status()
    soup = BeautifulSoup(response.content, "lxml")

    stock_node = soup.select_one("p.instock.availability")
    price_node = soup.select_one("p.price_color")

    stock_match = AVAILABLE.search(stock_node.get_text(" ", strip=True)) if stock_node else None
    price_match = AMOUNT.search(price_node.get_text(strip=True)) if price_node else None

    return {
        "in_stock": int(stock_match.group(1)) if stock_match else None,
        "price": float(price_match.group(1)) if price_match else None,
    }


print(stock_and_price("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"))

6. Slice out embedded JSON, then parse it properly

Many sites ship their entire page state as a JSON object inside a <script> tag. Extracting individual fields from that with regex is fragile; extracting the whole object with one regex and then handing it to json.loads is robust, because after the slice you are working with a real parser again.

import json
import re

HTML = """
<script>window.__STATE__ = {"product": {"sku": "AB-1023", "price": 51.77,
"tags": ["fiction", "poetry"]}};</script>
"""
STATE = re.compile(r"window\.__STATE__\s*=\s*(\{.*?\});", re.DOTALL)


def page_state(html: str) -> dict[str, object]:
    match = STATE.search(html)
    if match is None:
        raise ValueError("page state block not found — the site template changed")
    return json.loads(match.group(1))


state = page_state(HTML)
print(state["product"]["sku"], state["product"]["tags"])

re.DOTALL lets . match newlines, which is required because minified state blocks are frequently wrapped. Working with the resulting structure is covered in Parsing JSON and XML Responses, and the specific case of application/ld+json blocks in Extracting JSON-LD and Structured Data.

7. Normalise before you match

Patterns are written against ASCII assumptions and web text is not ASCII. A price rendered with a non-breaking space (£ 51.77), a hyphen that is actually an en dash, or a Turkish dotless ı will silently fail a pattern that looks correct. Normalise first, then match.

import re
import unicodedata

DASHES = dict.fromkeys(map(ord, "‐‑‒–—−"), "-")
RANGE = re.compile(r"(\d+\.\d{2})\s*-\s*(\d+\.\d{2})")


def normalise(raw: str) -> str:
    """NFKC folds NBSP and typographic variants; the table folds dash lookalikes."""
    return unicodedata.normalize("NFKC", raw).translate(DASHES)


raw = "Price range: 19.99– 29.99"
print(RANGE.search(raw))                     # None
print(RANGE.search(normalise(raw)).groups()) # ('19.99', '29.99')

The wider family of decoding failures — mojibake, double-encoded UTF-8, surrogate escapes — is diagnosed in Fixing Common Unicode Errors in Python Scraping.

8. Fail loudly when a pattern stops matching

A pattern that matches nothing returns None or an empty list, and a pipeline that treats that as "no data on this page" will happily write empty rows for a month after a template change. Wrap the extraction so a required field that vanishes is an error.

import re


class ExtractionError(RuntimeError):
    """Raised when a required pattern no longer matches the page."""


def required(pattern: re.Pattern[str], text: str, field: str) -> str:
    match = pattern.search(text)
    if match is None:
        raise ExtractionError(f"{field}: pattern {pattern.pattern!r} matched nothing")
    return match.group(1)


SKU = re.compile(r"\bSKU:\s*([A-Z]{2}-\d{4})\b")
print(required(SKU, "Item SKU: AB-1023 in stock", "sku"))

Escalating that error into an alert rather than a log line is the subject of Detecting Silent Scraper Failures.

9. Pin the patterns down with fixtures

A regular expression is code, and the cheapest way to keep it honest is a table of inputs with expected outputs, including the ones that must not match. Saved fixtures also give you a regression suite for free: when a site changes, you add the new page's text as a case and see immediately which patterns still hold.

import re

MONEY = re.compile(r"(?<![\d.])(?:£|\$|€)\s?(\d{1,3}(?:,\d{3})*(?:\.\d{2})?)(?![\d.])")

CASES: list[tuple[str, str | None]] = [
    ("Price: £51.77", "51.77"),
    ("Now $1,299.00 only", "1,299.00"),
    ("€8.50 including VAT", "8.50"),
    ("Order 12345 shipped", None),      # a bare number is not a price
    ("Rated 4.5 out of 5", None),       # no currency symbol
]


def run_cases() -> None:
    for text, expected in CASES:
        match = MONEY.search(text)
        actual = match.group(1) if match else None
        status = "ok " if actual == expected else "FAIL"
        print(f"{status} {text!r} -> {actual!r} (expected {expected!r})")


run_cases()

The negative cases carry most of the value. Over-matching is far more damaging than under-matching in a scraping pipeline, because an under-match raises an alert while an over-match writes plausible-looking wrong data into the dataset and nobody notices for weeks.

Performance and Scaling Considerations

A compiled pattern runs in microseconds; the wrong pattern can run forever. Matching a well-written pattern against a 250 KB page costs roughly 0.5–3 ms — an order of magnitude less than parsing the same page with lxml. That is why the two-stage approach is fast: the parser is called once, and the patterns are effectively free.

Catastrophic backtracking is the real risk. Nested quantifiers over overlapping character classes — (\w+\s*)+$, (a+)+b, (.*,)* — produce exponential behaviour on inputs that nearly match. A 30-character adversarial string against (a+)+b takes longer than the age of the universe to reject. Python's re has no timeout, so the worker simply stops responding, holding its connection and its slot in the pool.

import regex  # third-party; re has no timeout parameter

BAD = regex.compile(r"(\w+\s*)+$")
try:
    BAD.match("x " * 40 + "!", timeout=0.25)
except TimeoutError:
    print("pattern abandoned after 250 ms — rewrite it, do not raise the timeout")

The structural fixes are: replace nested quantifiers with a single one, use negated character classes instead of ., anchor with \A and \Z where the whole string is meant, and cap repetition explicitly ({1,64} rather than +) when the field has a known maximum length.

findall on a large body allocates the whole result set. Extracting every URL from a 100 MB sitemap with findall builds a list of several hundred thousand strings before you touch any of them. finditer keeps memory flat. The same applies to re.sub with a function replacement, which streams.

Compile at import, not in the loop. Ten thousand re.search(r"...", text) calls spend measurable time on cache lookups; ten thousand PATTERN.search(text) calls do not. On a crawl extracting six fields per page across 50,000 pages, that is 300,000 avoidable lookups.

Prefer one pass over many. When you need several tokens from the same text, a single alternation with named groups — (?P<sku>…)|(?P<price>…)|(?P<date>…) — visits the string once instead of three times. This matters when the text is large; for a short node's text it is noise, and readability should win.

Common Errors and Fixes

re.error: nothing to repeat at position 0. The pattern begins with a quantifier applied to nothing, usually because a literal *, + or ? was not escaped. Escape the literal or use re.escape() when the pattern is built from user or site input.

import re

term = "50% off (limited)"
pattern = re.compile(rf"\b{re.escape(term)}\b")
print(bool(pattern.search("Sale: 50% off (limited) today")))

re.error: bad escape \d at position 1. The pattern was written as a normal string, so Python consumed the backslash before re saw it. Always write patterns as raw strings.

import re

wrong = "\d{4}-\d{2}-\d{2}"          # SyntaxWarning, then re.error on some escapes
right = r"\d{4}-\d{2}-\d{2}"
print(re.search(right, "released 2026-08-01").group(0))

AttributeError: 'NoneType' object has no attribute 'group'.search returned None and you chained straight onto it. This is the same failure shape as chaining onto a missing element in a parser: assign first, test, then use.

import re

match = re.search(r"id=(\d+)", "no identifier here")
value = match.group(1) if match else None
print(value)

TypeError: cannot use a string pattern on a bytes-like object. You applied a str pattern to response.content. Either decode the body first or compile a bytes pattern with a rb"" literal — but do not mix them, because a bytes pattern has no Unicode semantics for \w.

import re

body = b"<title>Books to Scrape</title>"
print(re.search(rb"<title>([^<]+)</title>", body).group(1).decode("utf-8"))

The pattern matches in a tester but not in Python. Almost always one of three differences: the online tester defaulted to multiline or global flags you did not set; the page text contains a non-breaking space where you typed a normal one; or the source has newlines inside the region you assumed was one line. Print repr() of the actual input before rewriting the pattern — it makes all three visible immediately.

findall returns tuples instead of strings. The pattern contains more than one capturing group, so each match is a tuple of groups. Convert the groups you do not need into non-capturing (?:…) form, or use finditer and select by name.

Frequently Asked Questions

Is it ever acceptable to parse HTML with regular expressions? For a flat, non-nested pattern in a known region — extracting content from a specific meta tag, or a numeric id from a URL — a regex is fine and often clearer than a parser call. It becomes unacceptable the moment correctness depends on nesting, because a regular language cannot match balanced delimiters at arbitrary depth.

How do I match across multiple lines? Use re.DOTALL so . includes newlines, and re.MULTILINE only if you want ^ and $ to mean line boundaries rather than string boundaries. They are independent flags that people frequently confuse; DOTALL changes what . matches, MULTILINE changes what the anchors mean.

Why is my scraper suddenly using all its CPU on one page? Almost certainly catastrophic backtracking triggered by an unusual page. Find the pattern with nested quantifiers, rewrite it with negated character classes or bounded repetition, and add a timeout using the third-party regex module so a future variant degrades into an error rather than a hang.

Should I use regex to clean scraped text? For whitespace collapsing and stripping known junk tokens, yes — re.sub(r"\s+", " ", text) is the standard idiom. For anything involving currency, dates, units or deduplication, use a purpose-built normaliser instead; those problems have far more edge cases than a pattern can express, as covered in Normalizing Prices, Dates and Units.

Can regex extract data that JavaScript renders? Only if the data exists somewhere in the response you fetched. It very often does — as a JSON state blob in a <script> tag — in which case a slice-and-parse regex is the fastest route to it. If it genuinely arrives from a later XHR call, fetch that endpoint directly instead of rendering the page.

Related

• The Complete Guide to Python Web Scraping — the parent guide this page belongs to.
• Fixing Common Unicode Errors in Python Scraping — decoding problems that break patterns silently.
• Parsing HTML with BeautifulSoup — the structural half of the two-stage pipeline.
• Understanding HTTP Requests and Responses — getting the text these patterns run against.
• Cleaning and Validating Scraped Data — what to do with the tokens once you have them.