Cleaning and Validating Scraped Data

Extraction gives you strings; storage wants values, and this guide — part of Data Extraction Patterns and Working with APIs — covers the stage in between. The scope is deliberately narrow: what happens to a record after a selector or a JSON path has produced it and before a writer commits it to disk or a database.

That gap is where most scraped datasets quietly rot. A selector that returns "£1,299.00" is working perfectly; the problem is that the value is a presentation artefact, and comparing it to 1299.0 from another source produces nothing useful. Worse, nobody notices. The scraper reports 20,000 items collected, the database has 20,000 rows, and six weeks later somebody discovers that a third of the prices are 0.00 because a currency symbol changed and the parser silently returned the empty string.

Why Raw Fields Are Never Clean

Every value that arrives from a page is a string, even when the page renders it as a number, and it carries whatever the front end needed for display. That includes leading and trailing whitespace, non-breaking spaces (U+00A0) used to stop a price wrapping, HTML entities that a text extractor did not decode, thousands separators, currency symbols, unit suffixes, marketing prefixes like from and only, zero-width joiners inside emoji-adjacent text, and soft hyphens.

JSON sources are only marginally better. An API that returns "price": "19.99" has still handed you a string, and JSON-LD blocks stringify numbers constantly — a habit worth remembering when you work through Parsing JSON and XML API Responses. The type in the payload tells you what the backend's serialiser did, not what the value means.

Four families of field cause almost all of the work. Money carries a symbol or code, a thousands separator and a decimal separator whose roles swap between locales. Dates arrive in a local ordering with no time zone. Free text carries entities and invisible characters. Sizes and weights mix unit systems within the same catalogue. Each of these has a dedicated treatment in Normalizing Prices, Dates and Units.

When to Use a Separate Cleaning Stage

Not every scraper needs a formal stage. The decision is about how the output is consumed and how long it lives.

The rule that matters: the moment more than one code path writes to the same table, the cleaning logic has to live outside all of them. Two spiders that each parse prices "their own way" will disagree within a month, and the disagreement will be invisible because both produce valid-looking decimals.

Prerequisites

Python 3.10 or newer. The normalising half needs nothing beyond the standard library; the validating half is worth doing with a schema library rather than hand-rolled if statements.

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install "pydantic>=2.7" python-dateutil

html, unicodedata, decimal and re all ship with Python. pydantic provides the typed contract, and python-dateutil handles the date formats that datetime.strptime cannot guess.

Step-by-Step: From Raw Strings to a Validated Row

1. Look at what actually arrives

Before writing a single parser, dump twenty raw records with repr() rather than print(). repr() shows the escape sequences — '\xa0', '​', '\n\t' — that a plain print hides, and it will usually change your mind about which fields need work.

import json

def dump_raw_sample(rows: list[dict[str, str]], path: str, limit: int = 20) -> None:
    """Write a reviewable sample with every invisible character made visible."""
    with open(path, "w", encoding="utf-8") as handle:
        for row in rows[:limit]:
            visible = {key: repr(value) for key, value in row.items()}
            handle.write(json.dumps(visible, ensure_ascii=False) + "\n")


sample = [
    {"title": "A Light in the\xa0Attic ", "price": " £51.77", "stock": "In stock (22 available)"},
    {"title": "Tipping the Velvet", "price": "£53.74", "stock": "In stock (20 available)"},
]
dump_raw_sample(sample, "raw_sample.jsonl")

The first record above looks fine on screen. Its title actually ends with a space and contains a non-breaking space, so title == "A Light in the Attic" is False and a join on title silently produces nothing.

2. Normalise text before anything else

Text normalisation is the cheapest step and it has to run first, because every later parser assumes ordinary spaces and decoded characters. Three operations cover almost everything: unescape HTML entities, apply Unicode NFKC compatibility folding, then collapse runs of whitespace.

import html
import unicodedata


def clean_text(value: str) -> str:
    """Unescape entities, fold compatibility forms, drop invisibles, collapse space."""
    text = html.unescape(value)
    text = unicodedata.normalize("NFKC", text)
    text = "".join(c for c in text if unicodedata.category(c) != "Cf")
    return " ".join(text.split())


print(repr(clean_text("Ada Lovelace  \n")))   # 'Ada Lovelace'
print(repr(clean_text("Café\tdu Monde")))   # 'Café du Monde'
print(repr(clean_text("soft\u00adhyphen")))         # 'softhyphen'

NFKC is the important part and the part people skip. It maps the non-breaking space to a regular space, decomposes ligatures such as fi to fi, converts full-width Latin characters to ASCII, and turns superscript digits into ordinary ones. " ".join(text.split()) then handles the rest, because str.split() with no argument splits on any run of whitespace including tabs and newlines.

The category(c) != "Cf" filter closes the remaining gap. Zero-width space, zero-width joiner, soft hyphen and the byte-order mark all belong to Unicode's format category, and none of them are whitespace as far as str.split() is concerned — so without that line a title containing a soft hyphen compares unequal to the same title without one, and no amount of stripping reveals why. If the strings arriving from the network are mangled before you even get here, the problem is decoding rather than normalising — see Fixing Common Unicode Errors in Python Scraping.

3. Coerce the awkward types

Normalising and validating are different jobs and belong in different functions. A normaliser takes a string and returns a value or raises; it never decides whether the record is acceptable. Keeping the two separate means you can unit-test the parsers with a table of literals and no database in sight.

import re
from decimal import Decimal, InvalidOperation

CURRENCY_SYMBOLS = {"£": "GBP", "$": "USD", "€": "EUR", "¥": "JPY"}
_NUMBER_RE = re.compile(r"\d[\d\s.,']*\d|\d")


def parse_price(raw: str) -> tuple[Decimal, str | None]:
    """Return an exact amount and an ISO currency code from a displayed price."""
    text = clean_text(raw)
    currency = next((code for sym, code in CURRENCY_SYMBOLS.items() if sym in text), None)
    match = _NUMBER_RE.search(text)
    if match is None:
        raise ValueError(f"no numeric part in price {raw!r}")

    digits = re.sub(r"[\s']", "", match.group(0))
    if "," in digits and "." in digits:
        # Whichever separator comes last is the decimal separator.
        thousands = "," if digits.rfind(".") > digits.rfind(",") else "."
        digits = digits.replace(thousands, "").replace(",", ".")
    elif digits.count(",") == 1 and len(digits.split(",")[1]) != 3:
        digits = digits.replace(",", ".")
    else:
        digits = digits.replace(",", "")

    try:
        return Decimal(digits), currency
    except InvalidOperation as exc:
        raise ValueError(f"cannot parse price {raw!r}") from exc


print(parse_price(" from £1.299,00 "))   # (Decimal('1299.00'), 'GBP')
print(parse_price("$1,234.56"))          # (Decimal('1234.56'), 'USD')

Two details are load-bearing. Decimal rather than float, because 0.1 + 0.2 is not 0.3 in binary floating point and money that has been through a float round-trip cannot be reconciled against an invoice. And the separator logic is a heuristic, not a truth: "1,234" is genuinely ambiguous, and the code above resolves it as one thousand two hundred and thirty-four because three trailing digits after a single separator is overwhelmingly a thousands group.

4. Validate as a contract, not as a cleanup

Validation answers one question: is this row allowed to exist? It does not repair anything. Anything a validator would have to fix belongs one step earlier, in the normaliser. Expressed as a schema, the contract is short and self-documenting.

from decimal import Decimal
from typing import Annotated

from pydantic import BaseModel, Field, HttpUrl


class BookRow(BaseModel):
    title: Annotated[str, Field(min_length=1, max_length=300)]
    price: Annotated[Decimal, Field(gt=0, lt=100_000, decimal_places=2)]
    currency: Annotated[str, Field(pattern=r"^[A-Z]{3}$")]
    stock: Annotated[int, Field(ge=0)]
    url: HttpUrl


row = BookRow(
    title="A Light in the Attic",
    price=Decimal("51.77"),
    currency="GBP",
    stock=22,
    url="https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
)
print(row.model_dump(mode="json"))

Each constraint encodes a failure you have actually seen. min_length=1 catches the selector that matched an empty node. gt=0 catches the "Call for price" placeholder that parsed to zero. lt=100_000 catches the row where a phone number landed in the price column. decimal_places=2 catches a currency conversion that produced fifteen digits of noise. The full treatment of models, validators and modes is in Validating Scraped Data with Pydantic.

5. Fail the run rather than write junk

A validation stage that logs a warning and carries on is worse than no validation, because it creates the appearance of a check. Decide in advance what proportion of rejects is normal for the target, then abort the run when the rate exceeds it. A site redesign shows up as a rejection spike long before it shows up in a dashboard.

from pydantic import ValidationError


class RejectRateExceeded(RuntimeError):
    pass


def validate_batch(
    rows: list[dict[str, object]], max_reject_rate: float = 0.05
) -> tuple[list[BookRow], list[dict[str, object]]]:
    accepted: list[BookRow] = []
    rejected: list[dict[str, object]] = []

    for raw in rows:
        try:
            accepted.append(BookRow.model_validate(raw))
        except ValidationError as exc:
            rejected.append({
                "url": raw.get("url"),
                "errors": [
                    {
                        "field": ".".join(str(p) for p in err["loc"]) or "(row)",
                        "type": err["type"],
                        "message": err["msg"],
                        "input": repr(err.get("input"))[:120],
                    }
                    for err in exc.errors()
                ],
            })

    if rows and len(rejected) / len(rows) > max_reject_rate:
        raise RejectRateExceeded(
            f"{len(rejected)}/{len(rows)} rows rejected, above {max_reject_rate:.0%}"
        )
    return accepted, rejected

The err["type"] field is the machine-readable code — string_too_short, decimal_parsing, greater_than — and it is what you group by when you want to know which single failure mode is responsible for a spike. Grouping by the human-readable message works too, but the codes are stable across library versions and the messages are not. Wiring the reject rate into an alert is covered in Detecting Silent Scraper Failures.

6. Record the reason instead of dropping the row

A dropped row leaves no evidence. A rejected row written to a side file leaves the input, the field, the error code and the source URL, which is everything needed to fix the parser and replay the batch without re-crawling. Write it as JSON Lines so it appends cheaply and stays readable.

import json
from datetime import datetime, timezone


def write_rejects(rejected: list[dict[str, object]], path: str, run_id: str) -> None:
    """Append rejected rows so the batch can be diagnosed and replayed."""
    stamp = datetime.now(timezone.utc).isoformat()
    with open(path, "a", encoding="utf-8") as handle:
        for entry in rejected:
            record = {"run_id": run_id, "rejected_at": stamp, **entry}
            handle.write(json.dumps(record, ensure_ascii=False, default=str) + "\n")

Keeping the raw input in the reject entry matters more than it sounds. Once a parser is fixed, the rejects file is a replay queue: read it back, re-run the normaliser, and only the genuinely broken source pages remain. Without the raw input you have to crawl the site again, by which time the page may have changed.

7. Put it in the pipeline, not the spider

In Scrapy this is a set of item pipelines with distinct priorities, and the separation is the point: normalising never drops an item, validating is the only stage allowed to, and storage assumes everything reaching it is already correct.

from itemadapter import ItemAdapter
from pydantic import ValidationError
from scrapy.exceptions import DropItem


class NormalizePipeline:
    """Coerce every field. Never raises, never drops."""

    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        for field in ("title", "author"):
            if adapter.get(field):
                adapter[field] = clean_text(adapter[field])
        if adapter.get("price"):
            amount, currency = parse_price(adapter["price"])
            adapter["price"] = amount
            adapter["currency"] = currency or "GBP"
        return item


class ValidatePipeline:
    """The only stage allowed to reject an item."""

    def open_spider(self, spider):
        self.rejects = open("rejects.jsonl", "a", encoding="utf-8")

    def close_spider(self, spider):
        self.rejects.close()

    def process_item(self, item, spider):
        raw = ItemAdapter(item).asdict()
        try:
            return BookRow.model_validate(raw).model_dump()
        except ValidationError as exc:
            self.rejects.write(json.dumps({
                "spider": spider.name,
                "url": raw.get("url"),
                "errors": exc.errors(include_url=False),
            }, ensure_ascii=False, default=str) + "\n")
            raise DropItem(f"{exc.error_count()} validation errors")

Register them with explicit priorities in settings.py, leaving gaps so a stage can be inserted later without renumbering:

ITEM_PIPELINES = {
    "myproject.pipelines.NormalizePipeline": 100,
    "myproject.pipelines.ValidatePipeline": 200,
    "myproject.pipelines.PostgresPipeline": 300,
}

DropItem increments item_dropped_count in the crawl stats, which is the number to assert on in CI. The mechanics of writing, ordering and testing these classes are covered in Writing Scrapy Item Pipelines.

8. The same stage in an async writer

Outside Scrapy the shape is identical: a clean coroutine and a validate call between the fetch and the writer. Because validation is CPU-bound and fast, it belongs inline rather than in a thread pool — a Pydantic model validates a small row in single-digit microseconds, far below the cost of the network hop that produced it.

import asyncio

import httpx

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept": "application/json",
}


async def fetch_and_validate(client: httpx.AsyncClient, url: str) -> BookRow | None:
    response = await client.get(url, headers=HEADERS, timeout=20)
    response.raise_for_status()
    payload = response.json()
    try:
        return BookRow.model_validate({
            "title": clean_text(payload["title"]),
            "price": parse_price(payload["price"])[0],
            "currency": "GBP",
            "stock": int(payload.get("stock", 0)),
            "url": url,
        })
    except (ValidationError, ValueError, KeyError):
        return None


async def main(urls: list[str]) -> list[BookRow]:
    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(*(fetch_and_validate(client, u) for u in urls))
    return [row for row in results if row is not None]


rows = asyncio.run(main(["https://httpbin.org/json"]))
print(len(rows))

The concurrency controls that belong around this loop are covered in Asynchronous Scraping with Asyncio and HTTPX.

9. Pin the parsers with table-driven tests

Normalisers are pure functions from a string to a value, which makes them the easiest part of a scraper to test and the part most likely to be left untested. A table of inputs and expected outputs runs in milliseconds, needs no network, and turns every production surprise into a permanent regression test: when a new price format breaks a run, the fix is one row in the table plus one line in the parser.

from decimal import Decimal

import pytest

PRICE_CASES = [
    ("£12.99", Decimal("12.99"), "GBP"),
    (" from £1.299,00 ", Decimal("1299.00"), "GBP"),
    ("$1,234.56", Decimal("1234.56"), "USD"),
    ("€ 19", Decimal("19"), "EUR"),
    ("9.99", Decimal("9.99"), None),
]


@pytest.mark.parametrize("raw,amount,currency", PRICE_CASES)
def test_parse_price(raw: str, amount: Decimal, currency: str | None) -> None:
    assert parse_price(raw) == (amount, currency)


@pytest.mark.parametrize("raw", ["Call for price", "", "TBC", "—"])
def test_parse_price_rejects(raw: str) -> None:
    with pytest.raises(ValueError):
        parse_price(raw)


def test_clean_text_folds_invisible_characters() -> None:
    assert clean_text("Ada Lovelace  \n") == "Ada Lovelace"
    assert clean_text("​Title Here ") == "Title Here"

The rejection cases matter as much as the success cases. A parser that quietly returns Decimal("0") for "Call for price" passes any test that only checks the happy path, and the bug reaches production as a column of zeros. Asserting that bad input raises is what keeps the "fail loudly" property from eroding as the parser accumulates special cases.

Keep the fixtures alongside the tests as saved HTML or JSON files rather than live URLs. A test that fetches a page fails when the site is down, fails differently when the site changes, and cannot run in CI without network access. Save one representative page per format the parser handles, and add a new fixture whenever a rejection turns out to be a format you should have supported.

Performance and Scaling Considerations

• Compile every regular expression once at module level. A re.compile inside a per-row function re-checks the pattern cache on every call; hoisting it out is a measurable win once you are past a few hundred thousand rows.
• Validate in the pipeline, not after the crawl. Holding 200,000 dicts in memory to validate them at the end costs hundreds of megabytes and delays the first failure signal until the run is over. Streaming validation keeps memory flat and surfaces a broken selector in the first minute.
• Pydantic v2 validation is roughly 10–50 µs per small row because the core is compiled Rust. At 50 µs, one million rows costs under a minute of CPU — irrelevant next to the network time that produced them. Do not skip validation for speed.
• Decimal arithmetic is around an order of magnitude slower than float. That still leaves it far below parsing and I/O costs, and correctness on money is not negotiable. Convert to float only at the point of display.
• Deduplicate after validation, not before. Comparing unnormalised strings produces false negatives, so every duplicate you miss is a row you store twice. The ordering matters more than the algorithm.
• Batch the writer. Validation produces rows one at a time; the database wants them in groups of a few hundred. Buffer inside the storage pipeline as described in Storing and Exporting Scraped Data.

Common Errors and Fixes

decimal.InvalidOperation on a price that looks fine — the string still contains a non-breaking space or a narrow no-break space (U+202F), both common in European price formatting. Decimal(" 12.99") fails on the leading whitespace character. Run clean_text first so NFKC folds those to ordinary spaces, then strip.

ValueError: time data '03/04/2026' does not match format '%m/%d/%Y' — the source is day-first and the parser is month-first, or vice versa. Never guess per row; determine the site's ordering once from a date you can verify (any day above 12 gives it away) and pass that as a fixed setting.

Prices silently equal to 0.00 — a regex matched nothing and the code used a default instead of raising. Make the normaliser raise ValueError on no match and let validation reject the row. A missing value must never become a plausible one.

ValidationError: Input should be a valid integer on "In stock (22 available)" — the field is being validated before it is normalised. Extract the digits in the normalising stage; a validator's job is to reject that string, not to mine it.

Rows vanish with no log line — a bare except Exception: continue inside the parse loop. Catch the specific exceptions you expect, record the row, and let anything unexpected propagate so the run fails loudly.

Duplicate key violations on insert — the natural key was built from an unnormalised string, so "Acme Kettle " and "acme kettle" produced two rows that the database later collapsed. Build keys from normalised values only; the techniques are in Deduplicating Records with Fuzzy Matching.

Frequently Asked Questions

Should cleaning happen in the spider or in a pipeline? In a pipeline. A spider's job is to locate values in a document, and mixing parsing rules into selector code means every spider carries its own copy. A shared pipeline gives one place to fix a currency format and one place to test it, which matters as soon as a second spider writes to the same table.

What is the difference between normalising and validating? Normalising transforms a value into its canonical form and never decides whether the record is acceptable. Validating checks the canonical value against a contract and either passes it through or rejects the whole row. Keeping them apart means a validator never has repair logic hiding inside it, and a normaliser never silently swallows a bad record.

Should a bad row abort the whole run or just be skipped? Skip the individual row, record why, and abort the run only when the rejection rate crosses a threshold you set in advance. One malformed product page out of ten thousand is normal; four hundred of them means the site changed and the rest of the batch should not be trusted either.

How do I stop two spiders from cleaning the same field differently? Put the normalisers in one module that every spider imports, and forbid parsing inside spider code by convention and by review. The failure mode is slow and invisible: two implementations agree on the common cases and diverge on the rare ones, so the table ends up with a mix of 1299.00 and 1.299, and nothing in the data says which spider produced which.

Where should the rejected rows go? A JSON Lines file next to the run output, or a rejects table with the same retention as the main data. It needs the raw input, the field, the error code, the source URL and the run identifier. With those five things a fixed parser can replay the file; without the raw input you have to crawl the site again.

Related

• Data Extraction Patterns and Working with APIs — the parent guide covering where scraped records come from
• Validating Scraped Data with Pydantic — models, field validators, strict mode and a rejects file
• Normalizing Prices, Dates and Units — runnable parsers for the three hardest field types
• Deduplicating Records with Fuzzy Matching — exact keys, blocking and similarity thresholds
• Flattening Nested JSON with pandas — getting nested payloads into flat rows before you clean them
• Saving Scraped Data to PostgreSQL — the typed destination these constraints are protecting