Handling Pagination and Infinite Scroll

Part of The Complete Guide to Python Web Scraping, this guide covers the traversal layer: how to get from the first twenty results to all four thousand without missing rows, duplicating rows, or looping forever. The scope is the loop itself — identifying the pagination mechanism, driving it, knowing when to stop, and keeping the collected set consistent while the source shifts underneath you.

Pagination is where scrapers quietly go wrong. A single-page scraper either works or throws. A paginated scraper can run to completion, exit zero, and be missing 30% of the data because a duplicate check silently discarded rows, or a stop condition fired early on a page that happened to be empty. Every design decision below exists to make that outcome loud instead of quiet.

When to Use Each Traversal Strategy

Identify the mechanism before writing any loop. Open the browser's network panel, filter to Fetch/XHR, and click through to page two — what appears there decides everything that follows.

The order of preference is fixed: hidden JSON endpoint first, HTML fragment endpoint second, offset or cursor URLs third, and a headless browser last. Each step down costs an order of magnitude in resources and adds a class of failure. Finding the endpoint is worth the twenty minutes it takes; the method is in Finding Hidden API Endpoints in Network Traffic.

The measured gap is not marginal. Collecting 500 items through a JSON endpoint is roughly ten requests, about a megabyte of transfer and a few seconds. The same 500 items via a headless browser is a full page render plus twenty-five scroll-and-wait cycles, tens of megabytes of assets, several hundred megabytes of resident memory and well over a minute. Browser automation is a legitimate tool when the data genuinely only exists post-render — see Handling Infinite Scroll with Playwright — but it should be the conclusion of an investigation, not its starting point.

Prerequisites

Python 3.10 or newer with the environment from Setting Up Your Python Scraping Environment. The HTTP examples need a client and a parser; the browser example needs Playwright and its bundled Chromium.

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install "requests>=2.31" "beautifulsoup4>=4.12" "lxml>=5.1"
pip install "playwright>=1.44" && playwright install chromium

playwright install chromium downloads roughly 150 MB of browser and is only needed for the last step; skip it if the target exposes a JSON endpoint. Understanding of status codes and headers is assumed — the reference is Understanding HTTP Requests and Responses.

Step-by-Step: Traversing a Complete Result Set

1. Drive an offset loop with a real stop condition

Never hardcode a page limit and never rely on a single signal. Combine three: a non-200 status, an empty item list, and the absence of a "next" link. A page that returns 200 with zero cards is the normal end-of-set behaviour on many sites; a page that returns 200 with the same cards as the previous page is a site that clamps out-of-range page numbers, which is what turns a naive loop infinite.

import random
import time
import requests
from bs4 import BeautifulSoup

BASE = "https://books.toscrape.com/catalogue/page-{n}.html"
HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
}


def crawl_pages(max_pages: int = 60) -> list[dict[str, str]]:
    """Walk numbered pages until a stop signal, guarding against a clamped page param."""
    seen_ids: set[str] = set()
    records: list[dict[str, str]] = []
    stalls = 0

    with requests.Session() as session:
        session.headers.update(HEADERS)
        for page in range(1, max_pages + 1):
            response = session.get(BASE.format(n=page), timeout=(5, 20))
            if response.status_code == 404:
                break
            response.raise_for_status()

            soup = BeautifulSoup(response.content, "lxml")
            cards = soup.select("article.product_pod")
            if not cards:
                break

            fresh = 0
            for card in cards:
                link = card.select_one("h3 > a")
                price = card.select_one("p.price_color")
                key = (link.get("href") or "") if link else ""
                if not key or key in seen_ids:
                    continue
                seen_ids.add(key)
                fresh += 1
                records.append(
                    {
                        "id": key,
                        "title": (link.get("title") or "").strip(),
                        "price": price.get_text(strip=True) if price else "",
                    }
                )

            stalls = stalls + 1 if fresh == 0 else 0
            if stalls >= 2:
                break
            if soup.select_one("li.next > a") is None:
                break

            time.sleep(random.uniform(1.0, 2.5))

    return records


rows = crawl_pages()
print(f"{len(rows)} unique records across the catalogue")

The seen_ids set does double duty: it deduplicates, and it produces the fresh count that drives the stall detector. That is the whole termination guarantee — a loop that cannot add new records twice in a row is done, regardless of what the site claims.

2. Follow cursor tokens rather than computing them

Cursor pagination hands you an opaque token and expects it back. Never try to decode or synthesise one: it may encode a sort key, a timestamp and a shard, and a fabricated cursor either errors or silently returns the wrong window. Loop while the response says there is more, and guard against a server that echoes the same cursor forever.

import time
import requests

API = "https://api.example.com/v1/products"
HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) Chrome/136.0.0.0 Safari/537.36",
    "Accept": "application/json",
}


def crawl_cursor(limit: int = 100, max_requests: int = 200) -> list[dict[str, object]]:
    """Follow next-page tokens until the API says there are none left."""
    params: dict[str, str | int] = {"limit": limit}
    collected: list[dict[str, object]] = []
    previous_cursor: str | None = None

    with requests.Session() as session:
        session.headers.update(HEADERS)
        for _ in range(max_requests):
            response = session.get(API, params=params, timeout=(5, 30))
            response.raise_for_status()
            payload = response.json()

            batch = payload.get("results") or []
            collected.extend(batch)
            if not batch:
                break

            cursor = payload.get("next_cursor")
            if not cursor or cursor == previous_cursor:
                break
            previous_cursor = cursor
            params["cursor"] = cursor
            time.sleep(0.5)

    return collected

The cursor == previous_cursor check is the cursor equivalent of the stall counter. The GraphQL variant of this loop, with pageInfo.hasNextPage and endCursor, is covered in Handling GraphQL Pagination and Cursors.

3. Prefer the endpoint the page itself calls

When the network panel shows an XHR returning JSON, that endpoint is your pagination interface. It is smaller, faster, versioned more slowly than the HTML, and usually returns fields the rendered page does not show. Copy the request as cURL from the browser, strip it to the headers that are actually required, and replay it.

import requests

ENDPOINT = "https://httpbin.org/anything/api/items"
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/136.0.0.0 Safari/537.36",
    "Accept": "application/json",
    "X-Requested-With": "XMLHttpRequest",
    "Referer": "https://httpbin.org/",
}


def probe(offset: int, limit: int = 50) -> dict[str, object]:
    """Replay the page's own XHR with an explicit offset window."""
    response = requests.get(
        ENDPOINT,
        params={"offset": offset, "limit": limit, "sort": "created_desc"},
        headers=HEADERS,
        timeout=(5, 20),
    )
    response.raise_for_status()
    return response.json()


print(probe(0)["args"])

Two headers matter more than the rest: X-Requested-With, which many frameworks require for XHR routes, and Referer, which is often checked to ensure the call originated from a page rather than directly. The full technique is in Reverse-Engineering Private APIs.

4. Sort by something stable before paginating

This is the subtlest failure in the whole subject. If a feed is sorted by "most recent" and new items arrive while you are on page 3, every subsequent item shifts one position forward — so an item that was on page 4 moves to page 5 and you skip it entirely. Nothing errors. You end up with a dataset that is missing rows at every page boundary.

Two fixes, in order of preference. Ask the API to sort by an immutable key — an id, a creation timestamp — because a stable sort makes offsets meaningful. If the sort cannot be changed, switch from offsets to keyset pagination: request items with id < last_seen_id, which is immune to insertions.

import requests

HEADERS = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) Chrome/136.0.0.0 Safari/537.36",
           "Accept": "application/json"}


def keyset_pages(session: requests.Session, url: str, page_size: int = 100):
    """Yield batches using the last seen id as the boundary instead of an offset."""
    last_id: str | None = None
    while True:
        params: dict[str, str | int] = {"limit": page_size, "order": "id_desc"}
        if last_id is not None:
            params["before_id"] = last_id
        response = session.get(url, params=params, headers=HEADERS, timeout=(5, 30))
        response.raise_for_status()
        batch = response.json().get("results") or []
        if not batch:
            return
        yield batch
        last_id = batch[-1]["id"]

5. Scroll only when there is no endpoint

If the data really is assembled client-side, drive a browser — but drive it on the right signal. Waiting a fixed number of seconds after each scroll is both slow and unreliable; wait on the item count changing, with a bounded number of consecutive no-change rounds before giving up.

from playwright.sync_api import sync_playwright

ITEM = "article.product_pod"


def scroll_collect(url: str, max_rounds: int = 40, patience: int = 3) -> int:
    """Scroll until the item count stops growing for `patience` consecutive rounds."""
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(
            user_agent=("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                        "(KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"),
            viewport={"width": 1280, "height": 900},
        )
        page.goto(url, wait_until="domcontentloaded", timeout=30_000)

        count = page.locator(ITEM).count()
        stalls = 0
        for _ in range(max_rounds):
            page.mouse.wheel(0, 4000)
            page.wait_for_timeout(600)
            new_count = page.locator(ITEM).count()
            if new_count == count:
                stalls += 1
                if stalls >= patience:
                    break
            else:
                stalls = 0
                count = new_count

        browser.close()
        return count


print(scroll_collect("https://books.toscrape.com/"))

page.mouse.wheel produces a real wheel event, unlike window.scrollTo, which some virtualised feeds ignore because they listen for wheel and touch input specifically. Note also that virtualised lists remove off-screen items from the DOM, so on those the item count stops growing while data keeps loading — extract during scrolling rather than at the end.

6. Persist progress so a failure resumes instead of restarting

A crawl that takes forty minutes will eventually be interrupted at minute thirty-eight. Write each batch out as it completes and record the cursor or page number alongside it, so the next run starts where the last one stopped.

import json
from pathlib import Path

STATE = Path("data/state.json")
OUT = Path("data/out/items.jsonl")


def save_batch(rows: list[dict[str, object]], cursor: str | None) -> None:
    """Append records and checkpoint the position in one step."""
    OUT.parent.mkdir(parents=True, exist_ok=True)
    with OUT.open("a", encoding="utf-8") as handle:
        for row in rows:
            handle.write(json.dumps(row, ensure_ascii=False) + "\n")
    STATE.write_text(json.dumps({"cursor": cursor, "written": len(rows)}), encoding="utf-8")


def load_cursor() -> str | None:
    if not STATE.exists():
        return None
    return json.loads(STATE.read_text(encoding="utf-8")).get("cursor")

Appending JSON Lines rather than rewriting a JSON array means an interrupted write costs one row, not the whole file. Turning this into a proper incremental crawl — only re-fetching what changed — is the subject of Caching and Incremental Crawling.

Performance and Scaling Considerations

Pagination is a sequential dependency, which is why it is slow. You cannot request page N+1 until page N tells you the cursor. At 300 ms per request plus a 1.5 second politeness delay, 200 pages take about six minutes of almost entirely idle time. Offset pagination is the exception: page numbers are computable ahead of time, so you can request pages 1–20 concurrently. Cap that concurrency per domain with a semaphore, as in Limiting Concurrency with Semaphores.

Raise the page size before you raise the concurrency. An API that accepts limit=200 instead of limit=20 cuts your request count by ten with no politeness cost at all, because it is one round trip rather than ten. Test the ceiling — most APIs clamp silently rather than erroring, so compare the returned length to what you asked for.

Deduplicate on identity, not on content. A set of URLs or ids costs about 80 bytes per entry, so a million-item crawl holds roughly 80 MB in memory — acceptable, but not free. Above a few million, switch to a probabilistic structure; the trade-off between memory and a small false-positive rate is quantified in Deduplicating URLs with Bloom Filters.

Do not accumulate results in a list. A 50,000-record crawl holding dicts in memory is 200–500 MB before you have written anything. Stream each batch to disk or a database as it arrives, which also gives you the crash resumption above. Storage options and their trade-offs are in Storing and Exporting Scraped Data.

Browser sessions leak across a long scroll. A Chromium page that has scrolled through five thousand items holds every image, every detached node and every listener the site created. Memory climbs steadily and eventually the tab is killed by the OS. Restart the browser context every few hundred items and resume from a recorded position rather than trying to do one long run.

Politeness sets the real ceiling. One request every 1–3 seconds per domain is a defensible default when a site publishes no policy, and it means a 500-page crawl takes 15–25 minutes no matter how fast your code is. Design for that: run overnight, checkpoint often, and make the crawl resumable rather than fast. Additional defensive measures are covered in How to Scrape a Static Website Without Getting Blocked.

Common Errors and Fixes

The loop never terminates. The site clamps out-of-range page numbers to the last valid page and keeps returning it with a 200. Detect it by counting new ids per page rather than trusting the status, and stop after two consecutive pages with zero new records.

fresh = len(ids_on_page - seen_ids)
seen_ids |= ids_on_page
stalls = stalls + 1 if fresh == 0 else 0
if stalls >= 2:
    break

requests.exceptions.HTTPError: 429 Too Many Requests partway through. You crossed the rate limit, usually because a fixed delay was too short for a site whose limit is per-minute rather than per-request. Honour Retry-After, back off exponentially, and treat a 429 as a signal to slow the whole crawl rather than to retry one page.

import time

if response.status_code == 429:
    wait = float(response.headers.get("Retry-After", "60"))
    time.sleep(min(wait, 300))
    delay = min(delay * 2, 30)      # slow the whole loop, not just this request

Records are missing at every page boundary. The result set is sorted by a mutable key and shifted while you paginated. Switch to keyset pagination or request a stable sort order; no amount of retrying fixes a moving window.

playwright._impl._errors.TimeoutError: Timeout 30000ms exceeded. The wait condition never became true. Usually the selector is wrong, or wait_until="networkidle" was used on a page with a persistent WebSocket or analytics beacon, which never goes idle. Wait for a specific element instead.

page.goto(url, wait_until="domcontentloaded", timeout=30_000)
page.wait_for_selector("article.product_pod", state="attached", timeout=15_000)

Duplicate rows in the final dataset. Either the dedupe key is not stable — a URL with a tracking parameter differs per page load — or items genuinely appear on two pages because of a shifting sort. Normalise the key by stripping query parameters before hashing it.

from urllib.parse import urlsplit, urlunsplit

def canonical(url: str) -> str:
    parts = urlsplit(url)
    return urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))

The item count grows during scrolling, then the extracted list is short. The feed is virtualised and removes off-screen nodes. Extract after each scroll round into a set keyed by id, rather than reading the DOM once at the end.

Frequently Asked Questions

How do I tell whether a site uses pagination or infinite scroll? Open the network panel, filter to Fetch/XHR, and scroll or click to the next page. A URL change with a full document response is offset pagination; a background call returning JSON while the URL stays the same is infinite scroll backed by an API; no network activity at all means the whole set arrived in the first response and is being revealed client-side.

Can I scrape an infinite scroll feed without a browser? Usually yes, and you should try first. The scroll handler almost always calls an HTTP endpoint you can call directly, which is dramatically cheaper. A browser is only necessary when the response is signed by client-side JavaScript or the payload is rendered rather than returned as data.

How many pages should I request in parallel? For offset pagination where page numbers are computable, four to eight concurrent requests per domain is a reasonable ceiling for a site with no stated policy. Cursor pagination cannot be parallelised at all within one sequence, though you can run several independent sequences — one per category or filter — concurrently.

What should I do when a page fails in the middle of a crawl? Retry that page with backoff up to a small limit, and if it still fails, record the page number in a failures list and continue rather than aborting. A crawl that stops on the first bad page wastes everything it already collected; one that records and continues gives you a short list to re-run afterwards.

Should I trust the total count the site reports? Treat it as a hint, not a contract. Reported totals are frequently cached, approximate, or counted before filters are applied. Use it to detect gross under-collection — if the site says 4,000 and you have 900, something is wrong — but never as a loop termination condition.

Related

• The Complete Guide to Python Web Scraping — the parent guide this page belongs to.
• How to Scrape a Static Website Without Getting Blocked — staying welcome across a long crawl.
• Managing Cookies and Sessions — keeping one connection and one identity across hundreds of pages.
• Caching and Incremental Crawling — re-running a paginated crawl without re-fetching everything.
• Parsing HTML with BeautifulSoup — turning each page into records.