Using Playwright for Modern Web Automation

Playwright is the browser-automation option in Advanced Scraping Techniques and Anti-Bot Evasion built for a world of single-page applications, where the useful data arrives over a socket rather than in the initial HTML. This guide covers the context model that makes isolation cheap, the auto-waiting locators that remove most flakiness, request routing that cuts bandwidth in half, and the concurrency limits you will actually hit. As always, this is reference material for collecting data you are permitted to collect — respect robots.txt, honour published rate limits, and read the terms of service of anything you automate.

Playwright drives a real browser over a socket; each context is an isolated profile.

When to Use Playwright

Playwright and a plain HTTP client solve different problems, and Playwright and Selenium solve the same problem differently. The table below is the short version of the decision.

Signal	Choose	Reason
The data is in the HTML the server returns	`requests` + parser	No renderer needed; orders of magnitude cheaper
A single JSON endpoint returns everything	HTTP client	See Reverse-Engineering Private APIs
Blocked before any JavaScript runs	Impersonating client	Handshake-level problem — TLS and JA3 Fingerprint Evasion
App renders client-side, many short-lived sessions	Playwright	Contexts are cheap, so each identity is cheap
You need response bodies as well as the DOM	Playwright	`page.on("response")` gives both in one pass
Heavy async concurrency in one process	Playwright	Native `asyncio` API, one connection per browser
Legacy Safari, old Firefox, or an existing grid	Selenium	Wider driver and infrastructure support

The architectural difference that matters for scraping is the unit of isolation. In Selenium, a clean profile means a new browser process. In Playwright, a BrowserContext is a full incognito profile — its own cookie jar, storage, cache, permissions, geolocation, user agent and proxy — created inside an already-running browser in about fifteen milliseconds. That single design choice is what makes "one identity per target account" or "one identity per proxy exit" affordable at scale.

The second difference is auto-waiting. Playwright's locators re-resolve the selector and re-check actionability on every attempt, so a click issued while a React tree is still reconciling waits for the settled state instead of throwing a stale-element error. Most of the retry scaffolding a Selenium scraper accumulates simply disappears. Where raw numbers matter, Playwright vs Selenium Performance Benchmarks compares startup, navigation and interaction costs directly.

Prerequisites

Python 3.10 or newer. Playwright ships its own browser builds, so nothing is taken from the system Chrome.

pip install "playwright>=1.44" "lxml>=5.2"
playwright install chromium

On a bare Linux server, install the shared libraries the bundled browser expects. Playwright can do this for you:

playwright install-deps chromium

Verify the install resolves a browser before writing scraping logic:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    print("chromium", browser.version)
    browser.close()

playwright install chromium downloads roughly 170 MB into ~/.cache/ms-playwright. In a container, run it during the image build, not at container start, or every cold start pays for the download.

Step-by-Step: Building a Playwright Scraper

1. Separate the browser, the context and the page

The lifecycle you choose here determines both your throughput and your isolation. Launch one browser per worker process, create one context per identity, and one page per unit of work.

A fresh browser context buys you a clean cookie jar for a fraction of the cost of a fresh browser, which is why contexts are the right unit of isolation.

import asyncio
from playwright.async_api import async_playwright

USER_AGENT = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
)

async def fetch_titles(urls: list[str]) -> list[str]:
    """Fetch several pages, each in its own isolated context."""
    titles: list[str] = []
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        try:
            for url in urls:
                context = await browser.new_context(
                    user_agent=USER_AGENT,
                    viewport={"width": 1440, "height": 900},
                    locale="en-GB",
                    timezone_id="Europe/London",
                    extra_http_headers={"Accept-Language": "en-GB,en;q=0.9"},
                )
                page = await context.new_page()
                await page.goto(url, wait_until="domcontentloaded", timeout=30000)
                titles.append(await page.title())
                await context.close()
        finally:
            await browser.close()
    return titles

if __name__ == "__main__":
    print(asyncio.run(fetch_titles([
        "https://books.toscrape.com/",
        "https://books.toscrape.com/catalogue/page-2.html",
    ])))

Setting locale and timezone_id alongside the user agent keeps the identity internally consistent — a detail that matters more than any single spoofed value, as Browser Fingerprint and Stealth Configuration explains. wait_until="domcontentloaded" is the right default for scraping; networkidle waits for a 500 ms gap in network activity, which never arrives on pages with polling or live chat widgets.

2. Use locators, not one-shot queries

page.query_selector resolves once and hands you a stale handle. A Locator is a lazy description of an element that re-resolves every time you use it, and every action it exposes runs the actionability checks first.

One call to click hides four checks that Playwright re-runs on every animation frame until they all pass or the timeout expires.

import asyncio
from playwright.async_api import Page, async_playwright

async def read_first_price(page: Page) -> str:
    """Return the first product price, waiting for it to become readable."""
    price = page.locator("article.product_pod p.price_color").first
    await price.wait_for(state="visible", timeout=15000)
    return (await price.inner_text()).strip()

async def main() -> None:
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto("https://books.toscrape.com/", wait_until="domcontentloaded")
        print(await read_first_price(page))
        await browser.close()

if __name__ == "__main__":
    asyncio.run(main())

The four checks Playwright runs before dispatching an interaction — attached, visible, stable, and receiving events — are the reason page.click() rarely needs a retry wrapper. The "receiving events" check is the interesting one: Playwright hit-tests the click point and refuses to click if another element owns it, which turns the classic "cookie banner ate my click" bug into an explicit timeout naming the offending element rather than a silent no-op.

For lists, prefer count-based waits over sleeping. await page.locator("article.product_pod").count() is a single call, and expect-style polling on that count is the async equivalent of the stable-scroll loop. The specific mechanics of scroll-triggered loading get their own treatment in Handling Infinite Scroll with Playwright.

3. Block the bytes you will never parse

On a commercial page, images, fonts, video and third-party analytics routinely account for 70-85% of transferred bytes and a large share of layout work. Routing lets you refuse them before they leave the browser.

import asyncio
from playwright.async_api import Route, async_playwright

BLOCKED_TYPES = {"image", "media", "font", "stylesheet"}
BLOCKED_HOSTS = ("googletagmanager.com", "google-analytics.com", "doubleclick.net")

async def block_noise(route: Route) -> None:
    """Abort requests that never contribute to extracted data."""
    request = route.request
    if request.resource_type in BLOCKED_TYPES:
        await route.abort()
    elif any(host in request.url for host in BLOCKED_HOSTS):
        await route.abort()
    else:
        await route.continue_()

async def main() -> None:
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        context = await browser.new_context()
        await context.route("**/*", block_noise)
        page = await context.new_page()
        await page.goto("https://books.toscrape.com/", wait_until="domcontentloaded")
        print(len(await page.content()), "bytes of HTML")
        await browser.close()

if __name__ == "__main__":
    asyncio.run(main())

Register the route on the context, not the page, so every page it spawns inherits it. Be careful with stylesheet: blocking CSS removes layout, which means elements that depend on computed styles may never register as visible, and any visibility-based wait will time out. If you use visibility waits, block images and fonts but leave stylesheets alone.

4. Capture the API responses behind the render

A single-page application fetches its data as JSON and then paints it. Reading the JSON directly gives you full numeric precision, fields the UI hides, and pagination cursors, all without writing a selector.

import asyncio
from playwright.async_api import Response, async_playwright

async def collect_api_payloads(url: str, marker: str) -> list[dict]:
    """Load a page and return every JSON response whose URL contains `marker`."""
    payloads: list[dict] = []

    async def on_response(response: Response) -> None:
        if marker not in response.url:
            return
        if "application/json" not in (response.headers.get("content-type") or ""):
            return
        try:
            payloads.append(await response.json())
        except Exception:
            pass

    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        page.on("response", on_response)
        await page.goto(url, wait_until="domcontentloaded", timeout=30000)
        await page.wait_for_load_state("networkidle", timeout=15000)
        await browser.close()
    return payloads

if __name__ == "__main__":
    print(len(asyncio.run(collect_api_payloads("https://httpbin.org/json", "/json"))))

Filtering on both the URL marker and the content type keeps telemetry beacons out of your buffer. If you need to wait for a specific call rather than collect everything, async with page.expect_response(lambda r: marker in r.url) as info: gives you a handle that resolves the moment the matching response lands, which is far more precise than any timeout. Once you know the endpoint, its parameters and its auth requirements, consider dropping the browser entirely — the techniques in Parsing JSON and XML Responses apply directly to the captured payloads.

5. Give each context its own exit address

Playwright accepts proxy settings per browser and per context, and the per-context form is what makes IP rotation practical: one browser process can host ten contexts on ten different exits without a restart.

import asyncio
from playwright.async_api import async_playwright

async def fetch_via_proxy(server: str, username: str, password: str) -> str:
    """Fetch an IP echo endpoint through a context-scoped proxy."""
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        context = await browser.new_context(
            proxy={"server": server, "username": username, "password": password},
            extra_http_headers={"Accept-Language": "en-US,en;q=0.9"},
        )
        page = await context.new_page()
        await page.goto("https://httpbin.org/ip", wait_until="domcontentloaded")
        body = await page.inner_text("body")
        await browser.close()
        return body

Keep the whole identity coherent: an exit node in Frankfurt paired with timezone_id="America/New_York" is a contradiction any fingerprinting script can read in one line. The pool mechanics, health checks and cooldown policy live in Rotating Proxies and Managing IP Blocks.

6. Reuse authentication with storage state

context.storage_state() serializes cookies and localStorage into a JSON structure you can save and replay. Logging in once per day instead of once per run removes the most suspicious traffic pattern a scraper generates.

import asyncio
from pathlib import Path
from playwright.async_api import async_playwright

STATE_FILE = Path("storage_state.json")

async def save_state(url: str) -> None:
    """Visit a site once and persist its cookies and local storage."""
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        context = await browser.new_context()
        page = await context.new_page()
        await page.goto(url, wait_until="domcontentloaded")
        await context.storage_state(path=str(STATE_FILE))
        await browser.close()

async def reuse_state(url: str) -> str:
    """Start a context from saved state and return the page title."""
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        context = await browser.new_context(
            storage_state=str(STATE_FILE) if STATE_FILE.exists() else None
        )
        page = await context.new_page()
        await page.goto(url, wait_until="domcontentloaded")
        title = await page.title()
        await browser.close()
        return title

if __name__ == "__main__":
    asyncio.run(save_state("https://books.toscrape.com/"))
    print(asyncio.run(reuse_state("https://books.toscrape.com/")))

Treat the state file as a credential: it grants whatever access the session had. Keep it out of version control and encrypt it at rest if it represents a logged-in account.

7. Synchronise actions with the request they trigger

The most reliable way to know a filter, a sort or a "next page" click has finished is to wait for the network call it caused, not for a timer. expect_response opens a window, runs your action inside it, and resolves as soon as the matching response arrives.

import asyncio
from playwright.async_api import Page, async_playwright

async def click_and_capture(page: Page, selector: str, marker: str) -> dict:
    """Click an element and return the JSON body of the request it triggers."""
    async with page.expect_response(
        lambda r: marker in r.url and r.status == 200, timeout=20000
    ) as info:
        await page.locator(selector).click()
    response = await info.value
    return await response.json()

async def main() -> None:
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto("https://httpbin.org/", wait_until="domcontentloaded")
        print(await page.title())
        await browser.close()

if __name__ == "__main__":
    asyncio.run(main())

The same shape works for expect_navigation, expect_download and expect_popup. Content inside an <iframe> needs page.frame_locator("iframe#checkout").locator("#total"), which chains through the frame boundary without the explicit context switching Selenium requires; a shadow root needs nothing special at all, because Playwright's selector engine pierces open shadow DOM by default.

Performance and Scaling Considerations

Playwright's concurrency ceiling is memory, not CPU or the event loop. A Chromium process starts around 120 MB and a content-heavy context adds 40-90 MB on top, so an 8 GB worker comfortably runs one browser with roughly twelve to twenty concurrent contexts before swap pressure destroys tail latency. Because everything is asyncio, an unbounded asyncio.gather over ten thousand URLs will happily open ten thousand contexts and kill the machine; a semaphore is mandatory.

import asyncio
from playwright.async_api import Browser, async_playwright

async def scrape_one(browser: Browser, url: str, gate: asyncio.Semaphore) -> str:
    """Scrape one URL in its own context, bounded by a semaphore."""
    async with gate:
        context = await browser.new_context()
        try:
            page = await context.new_page()
            await page.goto(url, wait_until="domcontentloaded", timeout=30000)
            return await page.title()
        finally:
            await context.close()

async def scrape_all(urls: list[str], concurrency: int = 8) -> list[str]:
    """Scrape many URLs through one browser with bounded parallelism."""
    gate = asyncio.Semaphore(concurrency)
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        try:
            return list(await asyncio.gather(*(scrape_one(browser, u, gate) for u in urls)))
        finally:
            await browser.close()

The same bounded-fan-out pattern applies to non-browser work; Limiting Concurrency with Semaphores covers the tuning, and Asynchronous Scraping with Asyncio and HTTPX covers the case where you can skip the browser altogether.

Four further levers matter in production. Always close contexts in a finally block — a leaked context keeps its cache and cookie jar resident for the browser's lifetime, and the leak is invisible until the process is killed by the OOM killer several hours in. Prefer domcontentloaded over networkidle, which is by far the most common cause of a scraper that "works but is slow". Take one page.content() snapshot and parse it with lxml rather than issuing dozens of inner_text calls across the socket. And pin the Playwright version in your lockfile: the Python package and the downloaded browser build are versioned together, and a mismatched pair fails at launch rather than degrading gracefully. When the fleet moves to a scheduler or a container platform, the image-size and cold-start notes in Deploying Scrapers to the Cloud apply.

Choose the browser engine deliberately as well. Chromium starts fastest and has the best DevTools coverage, Firefox occasionally renders a site Chromium refuses to, and WebKit is the lightest of the three in memory but the most divergent in behaviour. Unless you have a specific reason, standardise on Chromium and install only that engine — playwright install chromium rather than the full set keeps container images several hundred megabytes smaller.

Finally, remember that a browser multiplies your request volume. One "page" can be sixty sub-requests; blocking images and analytics is not just a speed optimisation, it is a large reduction in the load you place on someone else's infrastructure.

Common Errors and Fixes

playwright._impl._errors.TimeoutError: Timeout 30000ms exceeded waiting for networkidle. The page keeps a socket, poller or analytics beacon alive, so the 500 ms quiet window never occurs. Switch to wait_until="domcontentloaded" and then wait on a concrete signal — await page.locator("article.product_pod").first.wait_for(state="visible") — which is both faster and more honest about what you actually need.

Error: Executable doesn't exist at .../chrome-linux/chrome. The Python package is installed but the browser binaries are not. Run playwright install chromium. Inside Docker, do this in the image build layer and set PLAYWRIGHT_BROWSERS_PATH=/ms-playwright so the download is shared rather than repeated per user.

Error: strict mode violation: locator resolved to 3 elements. Playwright refuses ambiguous locators by design. Narrow the selector, or state your intent explicitly with .first, .nth(1), or a chained .filter(has_text="…"). Suppressing the error with a broader selector is how silent data corruption starts.

net::ERR_TUNNEL_CONNECTION_FAILED on a proxied context. The proxy rejected the CONNECT request — usually wrong credentials, or credentials embedded in the server URL instead of the separate username and password fields. Playwright expects them split out; verify the exit independently with a plain HTTP client before blaming the browser.

Target page, context or browser has been closed. An await on a page outlived the async with block that owned the browser, or a finally closed the context while a pending task still referenced it. Make ownership explicit: whoever opens the context closes it, and no coroutine holding a page reference may outlive that scope.

BrowserType.launch: Host system is missing dependencies. A minimal base image lacks the shared libraries Chromium needs. Run playwright install-deps chromium, or start from an image that already bundles them.

Memory climbs steadily across a long run. Contexts are not being closed, or you are creating a browser per URL and relying on garbage collection. Audit every new_context for a matching close, and log len(browser.contexts) periodically — it should return to zero between batches.

It looks like you are using Playwright Sync API inside the asyncio loop. The synchronous and asynchronous APIs cannot coexist in one thread. Pick one per process; if a library you depend on already runs an event loop, you must use async_playwright.

Locator actions silently target the wrong element after a re-render. A locator built from an index — .nth(2) — is positional, and a re-order changes what it means. Prefer a locator anchored on stable text or a data attribute, and reserve index-based selection for lists you have just counted in the same tick.

Downloads never complete. Playwright discards downloads unless the context accepts them. Pass accept_downloads=True to new_context and use expect_download to obtain the handle, then call save_as with an explicit path.

Frequently Asked Questions

Should I use the sync or the async Playwright API for scraping? Use the async API for anything that fetches more than a handful of pages, because bounded asyncio fan-out over one browser is where Playwright's efficiency comes from. The sync API is convenient for exploration and one-off scripts, but it cannot be called from inside a running event loop, so mixing the two in one process raises an immediate error.

What is the real difference between a new context and a new page? A context is an isolated profile with its own cookies, storage, cache, permissions and proxy; a page is a tab that shares everything with its context. If two units of work must not see each other's session, they need separate contexts. If they are steps in one session, separate pages are cheaper.

Does Playwright avoid detection better than Selenium? Not inherently. Both drive a real browser, and both leak automation signals unless you patch them. Playwright's advantage is that per-context user agent, locale, timezone and proxy settings make a self-consistent identity easy to assemble; the actual patching of navigator.webdriver, canvas and WebGL is a separate job covered in undetected-chromedriver vs playwright-stealth.

How do I scrape a site that only renders after a login? Log in once interactively, save context.storage_state() to a file, and start every subsequent context from that state. Refresh it on a schedule that matches the session lifetime rather than on every run, and handle the case where the state has expired by detecting the login page and re-authenticating.

Can I run Playwright inside a serverless function? Yes, but the browser download makes images large and cold starts slow, so it fits scheduled batch work far better than request-response workloads. Bake the browser into the image, keep the function warm if latency matters, and check the platform's memory ceiling against the 250-400 MB a loaded Chromium actually uses.

Advanced Scraping Techniques and Anti-Bot Evasion — the parent section for this topic
Handling Infinite Scroll with Playwright — stop conditions for scroll-triggered loading
Playwright vs Selenium Performance Benchmarks — measured startup, navigation and interaction costs
Bypassing Cloudflare and Akamai Protections — what an edge challenge does to a browser session
Scraping Mobile App APIs — the same data without any browser at all