Handling Infinite Scroll with Playwright

Infinite scroll is the one browser-automation pattern where a script that looks correct still returns a fraction of the data, which is why it deserves its own treatment alongside Using Playwright for Modern Web Automation.

Each pass scrolls, waits for the row count to grow, then harvests immediately — because a virtualised list destroys the rows you skipped.

The short version: do not scroll to the bottom in a loop and parse the DOM at the end. Scroll one viewport at a time, wait for the rendered row count to exceed the count you recorded before the scroll, and extract on every pass into a de-duplicating structure keyed by something stable. Guard the loop with three independent limits so a feed that never terminates cannot hang the job. And before you write any of it, spend three minutes in the network panel — most infinite-scroll widgets are a thin wrapper over a paginated JSON endpoint that you can call directly for a fraction of the cost.

Why the Naive Scroll Loop Loses Rows

The loop almost everyone writes first scrolls to document.body.scrollHeight, sleeps, compares the new height with the old one, and breaks when they match. It fails in three distinct ways.

The first is a false negative on height. Plenty of feeds render a fixed-height loading placeholder or a sticky footer, so scrollHeight is identical immediately before and immediately after a batch arrives. The loop sees no change, decides the feed has ended, and exits after two passes on a list of forty thousand items.

The second is a false positive. Lazy-loaded images resize their containers as they decode, an advertisement slot expands, or a "back to top" affordance appears. scrollHeight grows without a single new record being added, so the loop keeps going, burning wall-clock time on passes that yield nothing.

The third is the expensive one: jumping straight to the bottom of the document skips intermediate scroll positions entirely. Feeds that use an IntersectionObserver on a sentinel element near the bottom will usually still fire, but feeds that load per viewport — observing several sentinels spread through the list — only load the batches whose sentinels actually crossed the viewport. A single jump to the bottom passes through those regions in one frame, and the observer callbacks that would have requested the middle batches never run. You end up with the first batch, the last batch, and a hole between them.

The fix for all three is to stop measuring pixels and start measuring the thing you actually care about: how many item nodes are currently in the DOM. page.locator(...).count() is cheap, unambiguous, and immune to layout noise.

Waiting on a Count Change Instead of Network Idle

The instinct after abandoning scrollHeight is to wait for networkidle. On an infinite feed that is usually the wrong tool. Playwright's networkidle resolves when there have been no network connections for 500 ms, and a modern feed page rarely goes quiet: analytics beacons, a websocket for live counts, image prefetching, and a long-polling notification channel all keep the connection count above zero. You get a 30-second timeout instead of a 400 ms wait.

Waiting on a count change is both faster and more precise. Playwright's assertion library has a built-in retrying matcher for exactly this, and expect(locator).to_have_count(n, timeout=...) polls until the count matches or the timeout expires. Because it retries rather than sampling once, it absorbs the gap between the fetch resolving and the framework committing the new nodes to the DOM.

There is a subtlety worth naming. to_have_count waits for an exact count, and a feed that appends 20 items at a time will pass through no intermediate value — but a feed that appends items one at a time as they stream in will briefly sit at every value between the old and new count. In that case, assert on a lower bound instead by waiting for the nth item to be attached: page.locator(sel).nth(previous_count).wait_for(state="attached", timeout=...) resolves the moment item number previous_count exists, whatever the final batch size turns out to be. That is the formulation used in the script below because it works for both append styles.

Harvesting Virtualised Lists as You Go

Nest three unrelated limits around the scroll pass. Any one of them can fire, so no single broken assumption can leave the loop spinning.

A virtualised (or "windowed") list renders only the rows near the viewport and recycles the DOM nodes as you move. React Virtuoso, TanStack Virtual, cdk-virtual-scroll-viewport and every large in-house table implementation do this, because a browser slows to a crawl somewhere past a few thousand live rows.

The consequence for scraping is absolute: after you scroll past a row, that row's node is gone. Not hidden, not display: none — removed and its element reused for a row further down. If your script scrolls to the end and then calls page.content() once, you will get the last screenful and nothing else, no matter how long the feed was.

Two signals tell you a list is virtualised before you waste a run. First, the item count stops growing and hovers around a constant — 30, 60, 100 — even as the scrollbar keeps shrinking. Second, the container has a large fixed height or a spacer element whose height changes while the number of children does not. Either sign means you must extract inside the loop.

Extract into a dictionary keyed on something the site itself treats as an identity: a data-id attribute, the product slug in the row's href, an SKU. Do not key on the row's index or its rendered text, because a recycled node can legitimately reappear and index positions shift as the window slides. A dictionary keyed on a stable identifier also gives you the loop's stop signal for free — if a full pass adds zero new keys, the feed is either exhausted or stuck, and both mean the same thing to your script.

A Scroll Loop That Cannot Run Forever

The script below scrolls a real, stable target and applies everything above: viewport-sized scrolling, a bounded wait for the next item to attach, harvesting on every pass, and three independent guards — a wall-clock deadline, a maximum pass count, and a consecutive no-growth streak. Any one of them ends the loop, so a broken assumption about one guard cannot leave the process spinning.

import asyncio
import time
from urllib.parse import urljoin

from playwright.async_api import async_playwright, TimeoutError as PWTimeout

START_URL = "https://books.toscrape.com/catalogue/page-1.html"
ITEM_SELECTOR = "article.product_pod"
USER_AGENT = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
)


async def harvest_pass(page) -> dict[str, dict[str, str]]:
    """Read every item currently in the DOM, keyed by its stable link."""
    rows = await page.eval_on_selector_all(
        ITEM_SELECTOR,
        """(nodes) => nodes.map((n) => ({
            href: n.querySelector('h3 a')?.getAttribute('href') ?? '',
            title: n.querySelector('h3 a')?.getAttribute('title') ?? '',
            price: n.querySelector('.price_color')?.textContent?.trim() ?? '',
        }))""",
    )
    return {r["href"]: r for r in rows if r["href"]}


async def scroll_and_collect(
    url: str,
    max_passes: int = 60,
    deadline_seconds: float = 180.0,
    stall_limit: int = 2,
) -> list[dict[str, str]]:
    """Scroll a feed, harvesting on every pass, under three independent guards."""
    collected: dict[str, dict[str, str]] = {}
    started = time.monotonic()

    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            user_agent=USER_AGENT,
            viewport={"width": 1366, "height": 900},
            extra_http_headers={"Accept-Language": "en-GB,en;q=0.9"},
        )
        page = await context.new_page()
        await page.goto(url, wait_until="domcontentloaded", timeout=30_000)
        await page.wait_for_selector(ITEM_SELECTOR, timeout=15_000)

        stalls = 0
        for _ in range(max_passes):
            if time.monotonic() - started > deadline_seconds:
                break

            before = len(collected)
            collected.update(await harvest_pass(page))
            if len(collected) == before:
                stalls += 1
                if stalls >= stall_limit:
                    break
            else:
                stalls = 0

            node_count = await page.locator(ITEM_SELECTOR).count()
            await page.mouse.wheel(0, 900)
            try:
                await page.locator(ITEM_SELECTOR).nth(node_count).wait_for(
                    state="attached", timeout=4_000
                )
            except PWTimeout:
                # No new node attached: either the end of the feed, or a
                # virtualised list that recycled instead of growing.
                pass

        await context.close()
        await browser.close()

    return [
        {**row, "url": urljoin(url, row["href"])}
        for row in collected.values()
    ]


if __name__ == "__main__":
    items = asyncio.run(scroll_and_collect(START_URL))
    print(f"collected {len(items)} items")
    for item in items[:3]:
        print(item["title"], item["price"], item["url"])

Two details matter more than they look. page.mouse.wheel(0, 900) dispatches a real wheel event just under one viewport high, which triggers scroll listeners that a direct window.scrollTo assignment sometimes does not. And the TimeoutError on the attach wait is deliberately swallowed rather than raised: the absence of a new node is information the loop consumes through the stall counter, not an error condition. Raising there would abandon everything harvested so far.

Skipping the Browser: Call the Feed Endpoint

The scroll loop and the endpoint return the same 1,000 rows; only one of them needs a rendering engine and fifty round-trips of waiting.

Every infinite scroll is a paginated request in disguise. The widget watches a sentinel, fires a fetch with a cursor or an offset, and appends whatever comes back. If you can issue that request yourself, you replace fifty browser scroll passes and roughly three minutes of waiting with twenty HTTP calls that finish in seconds and need no rendering engine at all.

Finding it takes about three minutes. Open the network panel, filter to Fetch/XHR, clear it, and scroll once. Exactly one request typically appears; its query string or JSON body carries the pagination parameter. The common shapes are an integer page or offset, a limit/per_page size, or an opaque cursor/after token echoed from the previous response. The systematic version of this hunt is covered in finding hidden API endpoints in network traffic.

Once you have it, replay it with an ordinary HTTP client, sending the same Accept, Referer and any custom API-version header the page sent — those are frequently required, and their absence is the usual reason a hand-built request returns 403 where the browser gets 200. If the endpoint requires a token minted by page JavaScript, a hybrid works well: let Playwright load the page once, capture the request with page.on("request") to read the headers, then hand those headers to the fast client for the remaining pages.

Keep the browser path for the cases where it genuinely earns its cost: endpoints signed per request with a value derived in obfuscated JavaScript, feeds behind a challenge that only clears after a real render, or one-off jobs where three minutes of scrolling is cheaper than thirty minutes of reverse engineering. The wider comparison of when a browser pays for itself lives in Playwright vs Selenium performance benchmarks.

Edge Cases and Caveats

The scroll container is not the window. Many feeds scroll an inner div with overflow-y: auto. page.mouse.wheel only affects the element under the pointer, so move the mouse over the list first with page.mouse.move(x, y) using coordinates from locator.bounding_box(), or call locator.scroll_into_view_if_needed() on the last row instead.
Bidirectional virtualisation. Some lists recycle nodes above the viewport as well as below. If you ever need to revisit earlier rows, you cannot — harvest on the way down or not at all.
A stall is not always the end. A stalled pass can mean a failed fetch behind a silent retry. Before treating two stalls as completion, check for a visible error or "load more" control and click it; a truthful stop condition beats an early exit.
Duplicate identifiers across sort orders. If the feed re-sorts while you scroll — "recommended" ordering that shuffles on each fetch — the same item can arrive at two positions and genuinely new items can be pushed past your stop point. Pin the sort order through a query parameter where one exists.
Memory grows with the harvest. A dictionary of 200,000 rows in a long-running browser process competes with the renderer for RAM. Flush to disk or a queue every few thousand keys and keep only the identifier set in memory.
Scroll speed as a signal. Fifty programmatic wheel events at machine speed with no pauses is a behavioural pattern anti-bot systems score. Vary the wheel delta and add short randomised pauses when the target is defended, and align the session with the rest of your browser fingerprint and stealth configuration.

Frequently Asked Questions

Why does my scroll loop stop after two passes when the feed clearly has more? Almost always because it compares document.body.scrollHeight between passes and the height did not change — a fixed-height loading placeholder or a sticky footer masks the growth. Compare the number of rendered item nodes instead, which changes only when real content arrives.

Should I wait for networkidle after each scroll? Usually not. Feed pages keep background connections open for analytics, websockets and prefetching, so networkidle frequently never resolves and you pay the full timeout on every pass. Wait for the next item element to attach, which is both faster and directly tied to the data you want.

How do I know whether a list is virtualised? Watch the item count while you scroll. If it climbs and then plateaus at a constant like 60 while the scrollbar keeps shrinking, nodes are being recycled. The other tell is a container with a large fixed height or a spacer element whose height changes without the child count changing.

Is it always better to call the API instead of scrolling? When the endpoint is reachable with ordinary headers, yes — it is roughly an order of magnitude faster and needs no browser. Stay with the browser when requests are signed by obfuscated page JavaScript, when a challenge must be cleared by a real render, or when the reverse-engineering time exceeds the run time for a one-off job.

Using Playwright for Modern Web Automation — the parent guide covering contexts, auto-waiting and network interception
Handling Pagination and Infinite Scroll — pagination shapes and cursor handling without a browser
Reverse-Engineering Private APIs — turning the feed request into a documented client
Scrapy vs Playwright for Single-Page Apps — when to render and when to fetch at scale