Validating Scraped Data with Pydantic

A scraper produces dicts of strings, and this page — part of Cleaning and Validating Scraped Data — shows how to put a typed contract between those dicts and whatever stores them.

Define one model per record type with explicit field types and constraints, call Model.model_validate(raw_dict) on every item, and treat the resulting ValidationError as data rather than as a crash. Put string surgery in mode="before" validators, put business rules in mode="after" validators and model_validator, and append every failure to a JSON Lines rejects file that carries the original input so a fixed parser can replay it.

Why a Model Beats Hand-Written Checks

The alternative is a function full of if not value: return None branches, and it fails for three specific reasons. It checks each field in isolation, so a broken row reports one problem, you fix it, re-run, and discover the next — Pydantic reports every failing field in a single ValidationError. It gives no machine-readable error identity, so you cannot count how many rows failed for the same reason. And it inevitably mixes repair into checking: the if that strips a currency symbol lives next to the if that rejects a negative price, and six months later nobody can say which fields are guaranteed clean downstream.

A model also becomes the documentation. price: Annotated[Decimal, Field(gt=0, decimal_places=2)] states the contract in one line that a reader and a type checker both understand, and it is the same declaration the database column should be built from.

Version note: everything here is Pydantic v2 (2.7 or newer). The v1 API used @validator, parse_obj and Config classes; v2 renamed these to @field_validator, model_validate and model_config = ConfigDict(...), and rewrote the core in Rust, which is why per-row validation costs tens of microseconds rather than milliseconds.

How Validation Runs, Step by Step

Knowing the order the hooks fire in is what stops most confusion, because a validator that works on a string will fail silently if it runs after coercion.

A mode="before" validator receives the raw input exactly as the scraper produced it — usually str, possibly None. This is the only place where string surgery makes sense: stripping a currency symbol, pulling digits out of "In stock (22 available)", unescaping entities. Return whatever the declared type can accept; it does not have to be the final type.

Then pydantic-core applies the declared type. "51.77" becomes Decimal("51.77"), "22" becomes 22. If this step cannot proceed, it raises with a code such as decimal_parsing or int_parsing and the after validators never run.

A mode="after" validator receives the value already converted, so value.quantize(...) on a Decimal is safe there and would raise AttributeError in a before validator. Finally @model_validator(mode="after") sees the whole populated model, which is the only place a cross-field rule can live — a sale price that exceeds a list price says the two columns were swapped, and no single-field check can see that.

Because these hooks are ordinary functions, they must raise ValueError (or AssertionError) to signal failure. Raising anything else propagates out of model_validate instead of becoming a validation entry, which is almost never what you want inside a pipeline.

A Runnable Model for Scraped Rows

The model below covers the four things scraped rows actually need: cleaning in before validators, real type constraints, a cross-field rule, and extra="forbid" so a renamed source field is a loud error rather than a silently missing column.

import html
import re
import unicodedata
from decimal import Decimal
from typing import Annotated

from pydantic import (
    BaseModel,
    ConfigDict,
    Field,
    HttpUrl,
    ValidationError,
    field_validator,
    model_validator,
)


def tidy(value: str) -> str:
    return " ".join(unicodedata.normalize("NFKC", html.unescape(value)).split())


class BookItem(BaseModel):
    model_config = ConfigDict(str_strip_whitespace=True, extra="forbid")

    title: Annotated[str, Field(min_length=1, max_length=300)]
    price: Annotated[Decimal, Field(gt=0, lt=100_000, decimal_places=2)]
    list_price: Decimal | None = None
    currency: Annotated[str, Field(pattern=r"^[A-Z]{3}$")] = "GBP"
    stock: Annotated[int, Field(ge=0)]
    url: HttpUrl

    @field_validator("title", mode="before")
    @classmethod
    def clean_title(cls, value: object) -> object:
        return tidy(value) if isinstance(value, str) else value

    @field_validator("price", "list_price", mode="before")
    @classmethod
    def strip_currency(cls, value: object) -> object:
        if not isinstance(value, str):
            return value
        digits = re.sub(r"[^\d.,]", "", tidy(value)).replace(",", "")
        return digits or value

    @field_validator("stock", mode="before")
    @classmethod
    def digits_only(cls, value: object) -> object:
        if not isinstance(value, str):
            return value
        match = re.search(r"\d+", value)
        return match.group(0) if match else value

    @model_validator(mode="after")
    def sale_price_not_above_list(self) -> "BookItem":
        if self.list_price is not None and self.price > self.list_price:
            raise ValueError("price is above list_price; the two fields were swapped")
        return self

Note return digits or value in strip_currency. When the input is "Call for price" the substitution leaves an empty string, and returning that would produce a confusing "input should be a valid decimal" error against "". Returning the original string instead means the error report shows what the page actually contained, which is the whole point of keeping the raw input.

Collecting Failures Instead of Crashing

ValidationError.errors() returns one dict per failing field with a loc tuple, a stable type code, a human message and the offending input. Flatten those into a reject entry and the file becomes both a diagnosis and a replay queue.

import json


def to_row(raw: dict[str, object]) -> dict[str, object]:
    return BookItem.model_validate(raw).model_dump(mode="json")


def reject_entry(raw: dict[str, object], exc: ValidationError) -> dict[str, object]:
    return {
        "url": raw.get("url"),
        "errors": [
            {
                "field": ".".join(str(part) for part in err["loc"]) or "(row)",
                "type": err["type"],
                "message": err["msg"],
                "input": repr(err.get("input"))[:120],
            }
            for err in exc.errors(include_url=False)
        ],
        "raw": {key: repr(value)[:200] for key, value in raw.items()},
    }


BATCH = [
    {"title": "A Light in the Attic ", "price": "£51.77",
     "stock": "In stock (22 available)",
     "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"},
    {"title": "Soumission", "price": "£50.10", "list_price": "£40.00", "stock": "20",
     "url": "https://books.toscrape.com/catalogue/soumission_998/index.html"},
    {"title": "", "price": "Call for price", "currency": "pounds",
     "stock": "out of stock", "url": "not-a-url"},
]

accepted, rejected = [], []
for raw in BATCH:
    try:
        accepted.append(to_row(raw))
    except ValidationError as exc:
        rejected.append(reject_entry(raw, exc))

print(f"{len(accepted)} accepted, {len(rejected)} rejected")
for entry in rejected:
    print(json.dumps(entry["errors"]))

Running it accepts the first row (title cleaned to A Light in the Attic, price to Decimal("51.77"), stock to 22), rejects the second with a single value_error from the model validator whose loc is empty — hence the or "(row)" fallback — and rejects the third with five separate entries: string_too_short, decimal_parsing, string_pattern_mismatch, int_parsing and url_parsing. Getting all five from one call, rather than one per re-run, is the practical difference from hand-written checks.

exc.errors(include_url=False) drops the documentation link Pydantic attaches to each error. Left in, it adds roughly 60 bytes per error to every line of the rejects file, which is noticeable across a few hundred thousand failures.

Strict Versus Lenient Coercion

By default Pydantic runs in lax mode and performs conversions it considers lossless. That is usually what a scraper wants, because everything arrives as a string, but it is worth knowing exactly where the line sits.

In lax mode "22" and 22.0 both become the integer 22, and "true" becomes True. 22.7 is rejected with int_from_float because the conversion would lose information, and "in stock" is rejected with bool_parsing because it is not a recognised boolean literal. Lax is permissive about representation, not about meaning.

ConfigDict(strict=True) turns all of that off: "22" into an int field raises int_type, and only an actual int is accepted. That is the right setting when the source is a JSON API that already types its fields, because a string arriving where a number was promised means the endpoint changed and you want to hear about it. For HTML-derived rows, strict mode on the whole model is unusable — instead apply it per field with Annotated[int, Field(strict=True)] on the handful of fields that come from a typed source.

Pydantic also offers a middle option: model_validate(raw, strict=True) sets the mode for one call, so the same model can validate leniently during a crawl and strictly in a test that asserts the normalising stage did its job.

Wiring It Into a Scrapy Pipeline

In a Scrapy project the model belongs in its own module and the pipeline stays thin. Counting rejects by error code in the crawl stats is the part worth adding, because it turns "the run had 400 failures" into "the run had 400 decimal_parsing failures", which points straight at one parser.

import json

from itemadapter import ItemAdapter
from pydantic import ValidationError
from scrapy.exceptions import DropItem

from myproject.models import BookItem, reject_entry


class PydanticValidationPipeline:
    def open_spider(self, spider):
        self.handle = open("rejects.jsonl", "a", encoding="utf-8")

    def close_spider(self, spider):
        self.handle.close()

    def process_item(self, item, spider):
        raw = ItemAdapter(item).asdict()
        try:
            return BookItem.model_validate(raw).model_dump(mode="json")
        except ValidationError as exc:
            for err in exc.errors(include_url=False):
                spider.crawler.stats.inc_value(f"validation/{err['type']}")
            self.handle.write(
                json.dumps({"spider": spider.name, **reject_entry(raw, exc)},
                           ensure_ascii=False, default=str) + "\n"
            )
            raise DropItem(f"{exc.error_count()} validation errors")

Register it with a priority above the normalising stage and below the storage stage. The pipeline mechanics themselves — ordering, open_spider, testing — are covered in Writing Scrapy Item Pipelines.

Those validation/<code> counters end up in the stats dictionary Scrapy prints when the crawl closes, which makes them trivially assertable. A run that finishes with validation/decimal_parsing: 412 out of 20,000 items is telling you one specific thing: the price selector matched something that is not a price on roughly two per cent of pages. A run that finishes with validation/url_parsing: 19,998 is telling you the site moved to relative links. Neither message is available from a log line that only says an item was dropped.

Testing the Model Instead of the Crawl

Because the model is a plain class, its contract can be tested without touching the network — and the tests worth writing are the negative ones, which assert that specific bad input produces a specific error code.

import pytest
from pydantic import ValidationError

VALID = {
    "title": "A Light in the Attic",
    "price": "51.77",
    "stock": "22",
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
}


def test_valid_row_round_trips() -> None:
    item = BookItem.model_validate(VALID)
    assert item.stock == 22
    assert str(item.price) == "51.77"


@pytest.mark.parametrize(
    "field,value,code",
    [
        ("title", "", "string_too_short"),
        ("price", "0", "greater_than"),
        ("price", "Call for price", "decimal_parsing"),
        ("stock", "unavailable", "int_parsing"),
        ("url", "/relative/path", "url_parsing"),
    ],
)
def test_bad_field_is_rejected(field: str, value: str, code: str) -> None:
    with pytest.raises(ValidationError) as caught:
        BookItem.model_validate({**VALID, field: value})
    assert caught.value.errors()[0]["type"] == code

Asserting on err["type"] rather than on the message is what makes these tests survive a library upgrade: Pydantic treats the codes as part of its public interface and rewords the human-readable messages freely between releases.

Edge Cases and Caveats

• extra="forbid" breaks when the site adds a field. That is intentional, but it means a harmless new key aborts rows. Use extra="ignore" while a source is still settling and switch to forbid once the shape is known.
• HttpUrl is not a str. model_dump() returns a Url object; use model_dump(mode="json") when the row is heading for json.dumps or a database driver that only speaks primitives.
• Decimal fields serialise as strings under mode="json". That preserves exactness, but a consumer expecting a number needs field_serializer or an explicit cast at the boundary.
• Validators do not run on assignment by default. item.price = "abc" after construction succeeds silently unless the model sets ConfigDict(validate_assignment=True).
• A digit-mining before validator eats the sign. re.search(r"\d+", "-1") returns "1", so Field(ge=0) can never fire on a string input that reached it through that hook. Match -?\d+ when negatives are meaningful, or accept that the constraint only guards non-string input.
• A before validator receives None for missing optional fields. Guard with isinstance(value, str) rather than assuming a string, or the first None raises TypeError out of the model instead of producing a validation entry.
• Field(pattern=...) uses Rust regex, not Python re. Lookahead and backreferences are unsupported and raise at class-definition time. Move those patterns into an after validator that uses re.
• Time zones. A bare datetime field accepts naive values, so half your rows can end up without an offset. Use AwareDatetime when every timestamp must carry one.

Frequently Asked Questions

Should I use Pydantic models or Scrapy Items? Use both, or use Pydantic alone. A Scrapy Item declares field names but enforces nothing about their values, so it cannot tell you that a price is negative. Keeping the Item as the spider's output shape and validating with a Pydantic model in the pipeline gives you field-name checking during parsing and value checking before storage.

How much does validation slow a crawl down? Not measurably. Pydantic v2 validates a small row in roughly 10 to 50 microseconds because its core is compiled Rust, so a million rows costs well under a minute of CPU. Any single HTTP request in the crawl costs more than validating the entire batch it produced.

What is the difference between a field validator and a model validator? A @field_validator sees one field and runs either before or after that field's type coercion. A @model_validator(mode="after") runs once the whole model is populated and can compare fields to each other, which is the only way to express rules such as a discount price never exceeding a list price.

How do I validate a list of records in one call? Wrap the model in a TypeAdapter(list[BookItem]) and call validate_python on the list. The resulting ValidationError reports failures with an index as the first element of each loc tuple, so you still know which record failed — though for long crawls, validating item by item surfaces the first failure sooner.

Related

• Cleaning and Validating Scraped Data — the parent guide and where this stage sits
• Normalizing Prices, Dates and Units — the parsers that feed these before validators
• Deduplicating Records with Fuzzy Matching — what to run once every row is valid
• Saving Scraped Data to PostgreSQL — turning model constraints into column constraints