Python Web Scraping
Reading layout

The Complete Guide to Python Web Scraping

Web scraping is the practice of turning a page that was designed for a human into a table a program can use. The mechanics are simple enough to fit in ten lines of Python; the difficulty is everywhere else — in the request that gets a 403, the parser that silently returns an empty list, the selector that stops matching after a deploy, the pagination loop that never terminates, and the session that logs itself out halfway through a run. This path covers the whole chain, in the order you will actually hit it, and each stage links to a page that goes deeper.

It is written for developers who can write a Python function and read a traceback but have not yet operated a scraper against a site that did not want to be scraped. If you already have that experience and want throughput, correctness at scale, or defence evasion, the sibling paths on Scaling Python Web Scrapers and Advanced Scraping Techniques and Anti-Bot Evasion pick up where this one stops.

Web scraping pipeline Five sequential stages: Fetch with requests, Parse with BeautifulSoup, Extract with selectors, Validate with Pydantic, and Store to a database or file. FetchrequestsParseBeautifulSoupExtractselectorsValidatePydanticStoreSQLite · CSV
The scraping pipeline: each stage feeds clean data into the next.

A scrape is a pipeline, and the single most useful debugging habit is knowing which stage of the pipeline a symptom belongs to. An AttributeError: 'NoneType' object has no attribute 'text' is almost never a parsing bug — it is a selector that matched nothing, which is usually a fetch that returned a different page than you expected. Print the status code and the first 500 characters of the body before you touch the selector.

Six stages of a scraping run and their failure modes Fetch, parse, select, paginate, authenticate and store, laid out in two rows of three. Each card names the library involved and the error you see when that stage is the one that broke. Where a run breaks, stage by stage1. Fetchrequests.Session, headers403, 429, read timeout2. ParseBeautifulSoup, lxmlmojibake, empty body3. SelectCSS and XPath queriesNoneType has no .text4. Paginateoffsets, cursors, scrollsilent infinite loop5. Authenticatecookies, CSRF, tokensredirect back to login6. Storevalidate, then writeduplicate or null rows
Each stage of a run has its own characteristic failure. Knowing which stage a bug belongs to is most of the debugging work.

Setting Up an Environment You Can Reproduce

Before any extraction logic, isolate the project. Scraping pulls in libraries with C extensions (lxml), browser binaries (Playwright), and packages that pin conflicting versions of urllib3. A global install turns those into a permanent, undiagnosable mess; a virtual environment turns them into a directory you can delete.

The environment also decides which Python syntax you can use. Everything on this site is written for Python 3.10 or newer, which is what allows built-in generics such as list[dict[str, str]] and the str | None union form to appear directly in signatures without importing typing. On 3.9 those raise at import time, not at call time, so the failure looks confusingly like a syntax problem rather than a version problem.

python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
pip install "requests==2.32.3" "beautifulsoup4==4.12.3" "lxml==5.3.0"
pip freeze > requirements.txt
python -c "import sys, requests, bs4; print(sys.version.split()[0], requests.__version__, bs4.__version__)"

The last line is the one worth keeping. It prints the three numbers that explain most "it works on my machine" reports, and pasting its output into a bug report saves a round trip. The full walkthrough, including per-project interpreter selection and what to do when lxml fails to build a wheel, is in Setting Up Your Python Scraping Environment.

Understanding What the Server Actually Sees

A scraper is an HTTP client, and it succeeds or fails on how closely its requests resemble the ones a browser sends. The status code is the first signal: 200 is success, 301/302 mean you were redirected — possibly to a login page that still returns 200 afterwards — 403 means the server identified you and declined, 404 means the URL is wrong, and 429 means you were fine until you were too fast.

Headers matter more than most beginners expect. A bare requests.get(url) sends User-Agent: python-requests/2.32.3, which is a signed confession. Sending a realistic User-Agent, an Accept header that matches what you want back, and an Accept-Language costs nothing and removes the cheapest reason to block you. Equally important is a timeout: without one, requests will wait forever on a server that accepts your connection and then says nothing, and a crawl can hang overnight on a single URL.

import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
}

def inspect(url: str) -> None:
    response = requests.get(url, headers=HEADERS, timeout=(5, 20))
    print("status:", response.status_code)
    print("final url:", response.url)
    print("content-type:", response.headers.get("Content-Type"))
    print("encoding:", response.encoding, "apparent:", response.apparent_encoding)
    print("body starts:", response.text[:120].replace("\n", " "))

inspect("https://books.toscrape.com/catalogue/page-1.html")

The timeout=(5, 20) tuple sets the connect and read timeouts separately, which is what you want: a server that refuses to connect should fail fast, while one that streams a large page slowly should be given room. The full request-response cycle, including redirects, compression, and what each status code implies for a retry policy, is covered in Understanding HTTP Requests and Responses.

Choosing the Right Fetching Tool

Before writing any parsing code, answer one question: is the data you want present in the raw response body, or does it only appear after JavaScript runs? The answer decides everything downstream, and getting it wrong costs either a wasted afternoon of selector debugging or an unnecessary browser dependency.

Decision tree for picking a fetching tool The first question asks whether the values appear in the raw HTML response. If yes, use requests with BeautifulSoup. If no, the second question asks whether a JSON endpoint supplies them; if yes replay that call, otherwise drive a real browser. Value in theraw HTML?A JSON callreturns it?yesnoyesnorequests + BeautifulSoupfastest, no browser neededreplay the JSON callclean fields, stable shapePlaywright or Seleniumslowest, use as a last resort
Two checks decide the tool: view the raw response first, and only reach for a browser when neither the HTML nor a JSON endpoint carries the values.

The test takes thirty seconds. Fetch the page with requests, search the response text for a value you can see on screen, and see whether it is there. If it is, an HTTP client and a parser are all you need, and you should not reach for a browser. If it is not, open the DevTools Network panel and look for a JSON response containing that value — most modern pages hydrate from an endpoint you can call directly, which is faster and far more stable than driving a browser. Only when neither is true is a headless browser the right answer.

import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,*/*;q=0.8",
}

def value_is_in_raw_html(url: str, needle: str) -> bool:
    """Decide whether a browser is needed, before writing any selectors."""
    response = requests.get(url, headers=HEADERS, timeout=10)
    response.raise_for_status()
    return needle.lower() in response.text.lower()

print(value_is_in_raw_html("https://books.toscrape.com/", "A Light in the Attic"))

That prints True for a server-rendered catalogue, which tells you the whole of this path applies. When it prints False, the techniques in Data Extraction Patterns and APIs — reading the JSON the page itself fetches — are usually a better next step than rendering.

Parsing HTML into Something You Can Query

Once you have the markup, it has to become a tree. BeautifulSoup is the standard entry point: it accepts broken HTML, normalises it, and gives you a small, memorable API over the result. The choice that actually matters is the parser backend you hand it. html.parser ships with Python and needs no build step; lxml is written in C, parses several times faster, and is worth the dependency the moment you are processing more than a handful of pages.

The defensive habit to build early is never chaining directly off a search. soup.select_one("h1").text raises when the page changed, when you were served a consent interstitial, or when the item is simply missing on this particular row. Assign first, check for None, and decide explicitly whether a missing field is a skipped record or a None in the output.

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
}

def parse_books(html: str) -> list[dict[str, str]]:
    soup = BeautifulSoup(html, "lxml")
    rows: list[dict[str, str]] = []
    for card in soup.select("article.product_pod"):
        title = card.select_one("h3 a")
        price = card.select_one("p.price_color")
        if title is None or price is None:
            continue                      # a malformed card, not a crash
        rows.append({
            "title": title.get("title", "").strip(),
            "price": price.get_text(strip=True),
        })
    return rows

page = requests.get("https://books.toscrape.com/", headers=HEADERS, timeout=10)
page.raise_for_status()
for row in parse_books(page.text)[:5]:
    print(row)

Note title.get("title", "") rather than title["title"]: attribute access on a Tag raises KeyError for a missing attribute, which is a crash in the middle of a long run. The parser comparison, encoding handling, and navigation methods are covered in Parsing HTML with BeautifulSoup.

Writing Selectors That Do Not Break Next Month

Selectors are where scrapers actually die. A fetch failure is loud and obvious; a selector that stops matching produces an empty list and a run that "succeeded" with zero rows. The difference between a scraper you maintain for a year and one you rewrite every month is almost entirely a question of what you anchored on.

Anchor on things the application itself depends on: id attributes, data-testid hooks its own test suite uses, itemprop and schema.org attributes its SEO depends on, and visible label text a redesign is unlikely to reword. Avoid anything generated — hashed class names such as .css-1x7yz9, deep nth-child chains, and utility classes like .mt-4 that describe styling rather than meaning. Python gives you two selector languages over the same tree: CSS, which is concise and reads well, and XPath, which can also walk upwards and backwards through the document. The choice is covered in Selecting Elements with XPath and CSS Selectors.

import requests
from lxml import html as lxml_html

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
}

def stock_for_labelled_row(url: str, label: str) -> str | None:
    """Find a table row by its visible label, then read the cell beside it."""
    response = requests.get(url, headers=HEADERS, timeout=10)
    response.raise_for_status()
    tree = lxml_html.fromstring(response.text)
    cells = tree.xpath(f'//th[normalize-space()="{label}"]/following-sibling::td[1]/text()')
    return cells[0].strip() if cells else None

url = "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
print(stock_for_labelled_row(url, "Availability"))

That expression anchors on the word "Availability" and then steps sideways, so it keeps working when the table is restyled, re-ordered, or wrapped in new containers. normalize-space() collapses the whitespace that template engines scatter through markup, which is the single most common reason a text-matching XPath silently fails.

Walking Through Every Page of Results

A listing is almost never one page. There are three shapes to recognise. Numbered pagination exposes the page in the URL, so you increment until a request 404s or a "next" link disappears. Offset pagination uses ?offset=50&limit=50 and needs the same loop with a different arithmetic. Cursor pagination returns an opaque token you must send back on the following request, which means you cannot parallelise it — each call depends on the previous one.

Infinite scroll is not a fourth shape; it is one of the three, driven by JavaScript. The scroll event fires a request to an endpoint that uses offsets or cursors, and calling that endpoint directly is dramatically faster than simulating scrolling in a browser. Whatever the shape, every loop needs a hard stop: a maximum page count, a check that the new page actually produced new records, and a guard against a server that happily returns page 999 as a copy of page 1.

import time
import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
}

def crawl_catalogue(max_pages: int = 5) -> list[str]:
    titles: list[str] = []
    seen: set[str] = set()
    for page in range(1, max_pages + 1):
        url = f"https://books.toscrape.com/catalogue/page-{page}.html"
        response = requests.get(url, headers=HEADERS, timeout=10)
        if response.status_code == 404:
            break                                    # ran off the end of the list
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "lxml")
        found = [a.get("title", "") for a in soup.select("article.product_pod h3 a")]
        fresh = [t for t in found if t not in seen]
        if not fresh:
            break                                    # server is repeating itself
        seen.update(fresh)
        titles.extend(fresh)
        time.sleep(1.0)                              # be a polite client
    return titles

print(len(crawl_catalogue()), "titles collected")

The fresh check is the guard that matters. Plenty of sites clamp an out-of-range page number back to the first page and return 200, and without that check the loop collects the same twenty records forever. Cursor handling and scroll-driven endpoints are covered in Handling Pagination and Infinite Scroll.

Keeping State Across Requests with Sessions

requests.get() opens a fresh TCP connection, performs a fresh TLS handshake, and throws away every cookie the server set. A requests.Session keeps all three. That is worth real time — reusing a connection removes the handshake from every request after the first — but the important part is the cookie jar, because anything involving login, a shopping basket, a regional preference, or a consent banner is carried in cookies.

Sessions also give you one place to set headers and mount an adapter for connection pooling and retries. Set the User-Agent on the session rather than per call and it applies everywhere, including to redirects you did not plan for.

import requests

def session_demo() -> None:
    with requests.Session() as session:
        session.headers.update({
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                          "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
            "Accept": "application/json",
        })
        session.get("https://httpbin.org/cookies/set/region/eu-west", timeout=10)
        echoed = session.get("https://httpbin.org/cookies", timeout=10)
        print("server sees:", echoed.json())
        print("local jar:", session.cookies.get_dict())

session_demo()

Both lines print the same region cookie, which is the whole point: the jar travelled with the session across two separate requests. Persisting that jar to disk so a long-running job survives a restart, and the difference between session cookies and Set-Cookie headers with an explicit expiry, are covered in Managing Cookies and Sessions.

Getting Past a Login Form

Authenticated pages are the point at which most scrapers stop being read-only fetches and start being conversations. The pattern is almost always the same: GET the login page first, because it plants a CSRF token in a hidden input and a session cookie in the jar; extract that token; then POST the credentials together with the token through the same session. Skipping the first GET is the classic mistake, and it produces a 403 or a silent redirect back to the form.

Verifying success needs care. A failed login usually returns 200 with the form re-rendered, so raise_for_status() tells you nothing. Check for something that only exists when authenticated — a logout link, an account name, a redirect away from /login.

import requests
from bs4 import BeautifulSoup

LOGIN = "https://quotes.toscrape.com/login"
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
}

def log_in(username: str, password: str) -> requests.Session:
    session = requests.Session()
    session.headers.update(HEADERS)

    form_page = session.get(LOGIN, timeout=10)
    form_page.raise_for_status()
    soup = BeautifulSoup(form_page.text, "lxml")
    token_input = soup.select_one('input[name="csrf_token"]')
    if token_input is None:
        raise RuntimeError("no csrf_token on the login form; the page changed")

    payload = {
        "csrf_token": token_input.get("value", ""),
        "username": username,
        "password": password,
    }
    result = session.post(LOGIN, data=payload, headers={"Referer": LOGIN}, timeout=10)
    result.raise_for_status()
    if "Logout" not in result.text:
        raise RuntimeError("login rejected; credentials or token wrong")
    return session

authed = log_in("anything", "anything")
print("logged in:", "Logout" in authed.get("https://quotes.toscrape.com/", timeout=10).text)

The Referer header on the POST is not decoration — a meaningful minority of frameworks reject a form submission whose Referer does not match the form's own origin. Bearer tokens, OAuth flows, and multi-step forms are covered in Handling Forms and Authentication.

Extracting Values That Are Not in Tags

Some values never become elements. A product identifier lives inside an inline <script>, a publication date sits in a sentence, a phone number is embedded in free text. This is the one place where a regular expression is the right tool — not for parsing HTML structure, which it cannot do reliably, but for pulling a known pattern out of a string you have already isolated with a parser.

The rule that keeps this sane: use the DOM to narrow down to the smallest string that contains what you want, then apply the regex to that string. Compile patterns once at module level, prefer non-greedy quantifiers so a match does not run to the end of the document, and validate the result before storing it.

import re
import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
}
PRICE = re.compile(r"£\s?(\d+(?:\.\d{2})?)")
UPC = re.compile(r"\b([0-9a-f]{16})\b")

def numbers_from_page(url: str) -> dict[str, str | None]:
    response = requests.get(url, headers=HEADERS, timeout=10)
    response.raise_for_status()
    body = response.text
    price = PRICE.search(body)
    upc = UPC.search(body)
    return {
        "price": price.group(1) if price else None,
        "upc": upc.group(1) if upc else None,
    }

print(numbers_from_page(
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
))

Both patterns are anchored on the shape of the value rather than on surrounding markup, so they survive a restyle. Compiled-pattern performance, the encoding traps that make £ arrive as £, and when to abandon regex for a parser are covered in Extracting Data with Regular Expressions.

Validating and Storing What You Collected

The last stage is the one most tutorials skip, and it is where silent data corruption enters a project. Scraped fields arrive as strings — "£51.77", "In stock (22 available)", "3 May 2026" — and if you write them straight to a file you have moved the problem downstream rather than solved it. Cast at the boundary, reject records that fail, and count the rejects so a schema change shows up as a number instead of as bad data.

Pydantic makes that boundary explicit: declare the types you expect, let it coerce and complain, and keep a tally. A run that suddenly rejects 90% of its records has told you the site changed, which is exactly the alert you want.

import csv
import re
from pydantic import BaseModel, ValidationError, field_validator

class Book(BaseModel):
    title: str
    price: float
    in_stock: int

    @field_validator("price", mode="before")
    @classmethod
    def strip_currency(cls, value: str | float) -> float:
        if isinstance(value, str):
            return float(re.sub(r"[^\d.]", "", value))
        return value

def clean(raw: list[dict[str, str]]) -> tuple[list[Book], int]:
    good: list[Book] = []
    bad = 0
    for row in raw:
        try:
            good.append(Book(**row))
        except ValidationError:
            bad += 1
    return good, bad

def write_csv(books: list[Book], path: str) -> None:
    with open(path, "w", newline="", encoding="utf-8") as handle:
        writer = csv.DictWriter(handle, fieldnames=["title", "price", "in_stock"])
        writer.writeheader()
        writer.writerows(book.model_dump() for book in books)

rows = [
    {"title": "A Light in the Attic", "price": "£51.77", "in_stock": "22"},
    {"title": "Broken Record", "price": "n/a", "in_stock": "3"},
]
books, rejected = clean(rows)
write_csv(books, "books.csv")
print(f"kept {len(books)}, rejected {rejected}")

Databases, columnar formats, and incremental writes that survive a crash mid-run belong to the next path — see Storing and Exporting Scraped Data.

Common Pitfalls

Almost every scraper that fails in production fails for one of a small number of reasons, and none of them are exotic. The list below is ordered roughly by how often it is the actual cause when a working scraper stops working.

  • Debugging the selector when the fetch was wrong. An empty result list usually means the response was a consent wall, a login redirect, or a 403 body served with a 200 status. Print response.status_code, response.url and the first few hundred characters of response.text before touching the selector.
  • Chaining off a search that can return None. soup.select_one(...).text is a crash waiting for the first row with a missing field. Assign, test for None, then decide.
  • Omitting timeout. requests has no default timeout. One unresponsive server will hang a crawl indefinitely, and the process will look busy rather than stuck.
  • Trusting response.text encoding. When a server sends no charset, requests falls back to ISO-8859-1 and every non-ASCII character arrives mangled. Compare response.encoding with response.apparent_encoding and set it explicitly when they disagree.
  • Pagination loops with no termination guard. A server that clamps out-of-range pages back to page one will keep a naive loop running forever. Stop when a page yields no records you have not already seen.
  • Parsing HTML structure with regular expressions. Nested tags, optional attributes, and unquoted values defeat any pattern you can write. Use a parser for structure and a regex only inside a string you have already isolated.
  • Scraping at full speed from the first run. Add a delay, cap concurrency, and cache responses while you develop, so that iterating on a selector does not mean re-requesting the same page fifty times.
  • Writing records straight from the parser to disk. Untyped strings that pass through unchecked become someone else's problem three systems downstream. Validate at the boundary and count the rejections, so a schema change on the target surfaces as a rising reject rate rather than as quietly wrong numbers in a report.

Frequently Asked Questions

Do I need Scrapy, or are requests and BeautifulSoup enough? For a few hundred pages from one site, requests plus BeautifulSoup is simpler and easier to debug. Scrapy earns its complexity when you need link-following across a domain, built-in retry and throttling policy, item pipelines, and a job you will run on a schedule for months. The decision is dissected in Scrapy vs BeautifulSoup: Which to Use.

How do I scrape a site that renders everything with JavaScript? First check whether you actually need to. Open the Network panel, filter to Fetch/XHR, and look for a JSON response containing the values you see on screen — replaying that request is faster and far more stable than rendering. If the data genuinely only exists after scripts run, use a headless browser as described in Using Playwright for Modern Web Automation.

Why does my scraper get a 403 when the page loads fine in my browser? The usual causes, in order of likelihood: a missing or obviously automated User-Agent, a missing Referer on a request that came from a page, a datacenter IP with a poor reputation, or a TLS fingerprint that does not match any real browser. Fix the headers first because it is free, then read TLS and JA3 Fingerprint Evasion if the block persists on the very first request.

How fast can I scrape without causing problems? Slower than you think, and the number depends on the target rather than on your hardware. One request per second against a single origin is a reasonable starting point; watch for 429 and 503 responses and back off exponentially when they appear. Caching responses during development removes most of the load you would otherwise generate.

Should I use CSS selectors or XPath? Use CSS while it expresses what you mean, and switch to XPath when you need to match on text content or walk upwards to a parent or backwards to a preceding sibling — neither of which CSS can do. Most production scrapers mix the two, and the trade-offs are laid out in XPath vs CSS Selectors: Which Should You Use?.

What is the single most common cause of a scraper breaking silently? A selector that no longer matches. It raises nothing, returns an empty list, and the run reports success with zero rows. Assert on a minimum expected record count at the end of every run so that "found nothing" is treated as a failure rather than as an unusually quiet day.

Back to home