About Python Web Scraping

Python Web Scraping is a free, open-access technical reference for extracting data from the web with Python, organised into the four learning paths reachable from the home page — fundamentals, anti-bot work, scaling, and reading data sources directly rather than parsing markup.

Four learning paths, from first request to production-scale extraction.

This page exists so that you can judge how much weight to put on anything else here. In short: every Python and shell block on this site was executed before the page was published, the library versions used are recorded below, external claims that depend on someone else's infrastructure are dated rather than stated as fact, and mistakes are corrected in place with the dateModified field bumped. The site is written and maintained by one engineer, publicly, with the working code kept on GitHub.

What This Site Covers

The material is split into four paths, and each one assumes only the path before it:

The Complete Guide to Python Web Scraping — environment setup, the HTTP request-response cycle, parsing HTML, writing selectors, pagination, forms and authentication, and sessions. Start here if pip install requests is new to you.
Advanced Scraping Techniques and Anti-Bot Evasion — headless browsers with Playwright and Selenium, proxy rotation, TLS and JA3 fingerprints, browser fingerprint surfaces, and what actually happens when a challenge page appears.
Scaling Python Web Scrapers — Scrapy, asyncio concurrency, distributed crawling, caching, monitoring, storage, and deployment.
Data Extraction Patterns and APIs — JSON-LD, private JSON APIs, GraphQL, XML feeds, and the data-cleaning layer that turns raw responses into records you can trust.

Every page names its parent in the opening sentence and ends with links to its neighbours, so you can walk the tree in either direction without going back to a menu.

Who Writes This

The site is written by a single working software engineer rather than a content team, which is deliberate: the failure modes described here — a 403 that only appears from a datacenter IP, a selector that survived four redesigns, a Celery worker that silently stopped acknowledging tasks — come from building and operating scrapers, not from summarising other articles. Where a page describes a measurement, that measurement was taken on a stated machine with a stated library version. Where a page describes a trade-off, it names the number that decides it.

That also means the site has one person's blind spots. If a page contradicts your experience, the correction process below is the fastest way to fix it for everyone.

How a Page Gets Verified

Nothing is drafted from memory alone. Each page passes through four stages, and a page that fails at stage two goes back to stage one rather than shipping with a caveat.

Nothing is published until every code block in it has been executed in a clean virtual environment and the library versions used have been recorded.

Draft. The outline is built from primary sources — library documentation, the library's own source code and changelog, the relevant RFC for protocol behaviour, and the browser's own DevTools output for anything about request shape. Secondary write-ups are not used as a source of fact.

Execute. Every ```python and ```bash block on the page is run top to bottom in a freshly created virtual environment, in the order it appears, against the target named in the code. Examples target deliberately stable, scrape-friendly endpoints — https://books.toscrape.com/, https://quotes.toscrape.com/, https://httpbin.org/ — or clearly generic example.com URLs, so that the block still runs for you a year from now and so that no third party absorbs traffic from readers copy-pasting a snippet. A block that needs a paid proxy, a live account, or a site that blocks automated access is marked as such in the surrounding prose instead of being presented as runnable.

Review. Version numbers are read back out of the environment rather than typed from memory, every internal link is resolved, and each factual claim is checked against the primary source that motivated it.

Publish. The page goes out with a datePublished and a dateModified, and is linked from its parent so that it is reachable by navigation rather than only by search.

The Versions Everything Is Tested Against

Scraping libraries move quickly, and a snippet that is correct for urllib3 1.x can raise on 2.x. Examples on this site are written against Python 3.10 or newer — typed signatures such as list[dict[str, str]] and the X | None union syntax are used throughout — and against the following baseline. You can reproduce it exactly:

python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install \
  "requests==2.32.3" \
  "httpx==0.27.2" \
  "beautifulsoup4==4.12.3" \
  "lxml==5.3.0" \
  "Scrapy==2.11.2" \
  "playwright==1.47.0" \
  "pydantic==2.9.2"
python -c "import requests, bs4, lxml; print(requests.__version__, bs4.__version__, lxml.__version__)"

The final line is the check that matters: if the versions it prints differ from the ones a page quotes, treat any behavioural detail on that page as needing a re-test. Where a page depends on a specific version boundary — the allowed_methods rename in urllib3 2.0, or Scrapy's move to asyncio as the default reactor — the boundary is called out inline rather than left for you to discover through a traceback.

What Is Verified and What Is Only Observed

Some statements can be executed. Others describe how a third party's system behaved on a particular day, and no amount of testing makes those permanent. The site labels the two differently, and it is worth knowing which you are reading.

Anything that can be run is run. Anything that depends on a third party's current behaviour is dated and marked as an observation rather than a guarantee.

Anything in the left column is reproducible from this page. Anything in the right column is a dated observation: a defence provider can change its challenge logic in an afternoon, a proxy vendor can re-price overnight, and a site can ship a redesign that invalidates a selector in an example. When a page says "as of August 2026, Cloudflare's managed challenge does X", that date is doing real work — it is telling you how much to trust the sentence.

How Pages Are Kept Current

Pages are revisited on two triggers rather than on a fixed calendar. The first is a dependency event: when a library used in an example publishes a major release, every page that imports it is re-run against the new version, and the page is either updated or annotated with the version boundary. The second is a reader report — a broken snippet, a changed API, a link that now 404s.

Every substantive change updates dateModified in the page's frontmatter. Cosmetic edits do not, so a recent modification date on a page means the technical content actually changed. Nothing is silently rewritten to look as though it had always said the new thing; where a recommendation reverses, the page says what changed and why.

Corrections and Contact

If something here is wrong, the useful report contains three things: the page URL, the exact command or snippet you ran, and the full traceback or the response status and body you got back. That is usually enough to reproduce the problem without a round trip.

Reports, code, and the working examples all live in the same place:

github.com/python-web-scraping-com

Open an issue there for a factual error, a version drift, a broken example, or a topic you expected to find and did not. Corrections are applied to the page itself rather than appended as an erratum, because a reader arriving from search should get the corrected version without having to scroll for a footnote.

Scope: A Technical Resource, Not Legal Advice

This site explains how scraping works at a technical level: what a request looks like on the wire, why a parser returns None, how a queue distributes work. It does not give legal advice, and no page here should be read as a statement that a particular scrape is lawful.

Whether you may collect a given dataset depends on the jurisdiction, the site's terms of service, the nature of the data — personal data in particular is regulated independently of how it was obtained — and often on contract terms you agreed to when you created an account. Those questions have real answers, but they are answers a qualified lawyer gives after looking at your specific case, not answers a technical reference can give in general. The engineering guidance throughout the site assumes you have already established that you are allowed to collect the data, and it consistently favours the polite option: honour robots.txt, cap concurrency, cache aggressively so you request each URL once, prefer a published API over an undocumented one, and never collect personal information you do not need.

Frequently Asked Questions

Are the code examples on this site actually runnable? Yes. Every Python and shell block is executed in a clean virtual environment before the page is published, in the order the blocks appear, and blocks are written complete rather than as fragments with ellipses. The exception is any snippet that requires a paid service or a live account, which is flagged in the surrounding prose as not directly runnable.

Which Python version should I use to follow along? Python 3.10 or newer. Examples use the modern typing syntax — built-in generics such as list[dict[str, str]] and the X | None union form — which raises a TypeError on 3.9 and earlier. Anything that depends on a newer feature than 3.10 says so at the point of use.

How do I report a mistake or a broken example? Open an issue at github.com/python-web-scraping-com with the page URL, the snippet you ran, and the traceback or HTTP status you received. Corrections are made directly on the page and the dateModified field is updated so you can tell that the technical content changed.

Does this site tell me whether scraping a particular site is legal? No. It is a technical reference and deliberately stays out of legal questions, which depend on your jurisdiction, the target's terms of service, and the kind of data involved. The engineering advice assumes you have already resolved that question, and it consistently recommends the least intrusive approach that gets the data.

Why do the examples use books.toscrape.com and httpbin.org instead of real sites? Because those endpoints exist to be scraped, they are stable enough that a snippet still works years later, and pointing thousands of readers at a live commercial site would generate exactly the kind of unwanted load this site tells you to avoid. The techniques transfer unchanged to whatever target you are working on.

The Complete Guide to Python Web Scraping — the fundamentals path, from first request to stored records
Advanced Scraping Techniques and Anti-Bot Evasion — browsers, proxies, and fingerprints
Scaling Python Web Scrapers — concurrency, queues, caching, and deployment
Data Extraction Patterns and APIs — JSON-LD, private APIs, and GraphQL
Setting Up Your Python Scraping Environment — reproduce the environment the examples are tested in