Mastering Selenium for Dynamic Websites

Selenium is the browser-automation half of Advanced Scraping Techniques and Anti-Bot Evasion, and it exists for one job: reaching content that only appears after the page's JavaScript has run. This guide covers the WebDriver architecture, the synchronization model that makes scripts reliable, DevTools-level network capture, and the operational habits that keep a fleet of browsers from eating your budget. Everything here assumes you are collecting data you are entitled to collect: check robots.txt, read the terms of service, and keep your request rate well inside what the site publishes.

When to Use Selenium

A browser is the most expensive tool in the extraction toolkit. Reach for it only when cheaper options genuinely cannot see the data.

Selenium's specific strengths are its maturity and its reach. It is the only mainstream framework that speaks the W3C WebDriver protocol to Chrome, Firefox, Edge and Safari with a single API, and it has the largest ecosystem of grid, cloud and reporting integrations. Its weakness is the protocol itself: every command is a separate round trip, which makes chatty scripts slow. If throughput matters more than browser coverage, compare the numbers in Playwright vs Selenium Performance Benchmarks before committing, and read Using Playwright for Modern Web Automation for the alternative model.

Volume is the other deciding factor. A single headless Chrome holds roughly 180-400 MB resident once a content-heavy page has loaded, so a 4 GB worker realistically runs six to ten browsers. If your target list runs to millions of URLs, budget for that arithmetic before writing any code.

Prerequisites

Python 3.10 or newer, and a local Chrome or Chromium installation. Selenium 4.6 and later ship Selenium Manager, which downloads a matching chromedriver automatically, so you no longer need to pin driver binaries by hand.

pip install "selenium>=4.20" "lxml>=5.2" "tenacity>=8.3"

On a headless Linux server, Chrome also needs a handful of shared libraries that minimal container images omit:

sudo apt-get update && sudo apt-get install -y \
    libnss3 libatk-bridge2.0-0 libgtk-3-0 libgbm1 libasound2

Confirm the toolchain resolves a driver before writing scraping logic:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
print(driver.capabilities["browserVersion"], driver.capabilities["chrome"]["chromedriverVersion"])
driver.quit()

If those two versions disagree on the major number, Selenium Manager has picked up a stale cached driver; clear ~/.cache/selenium and rerun.

Step-by-Step: Building a Reliable Selenium Scraper

1. Launch a driver with explicit, deliberate options

Default options are tuned for testing, not for unattended scraping. The set below fixes the four things that break server-side runs most often: an undersized viewport that hides responsive content, /dev/shm exhaustion inside containers, an unbounded page-load wait, and the automation banner.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.webdriver import WebDriver

USER_AGENT = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
)

def build_driver(headless: bool = True) -> WebDriver:
    """Return a Chrome WebDriver configured for unattended scraping."""
    options = Options()
    if headless:
        options.add_argument("--headless=new")
    options.add_argument("--window-size=1920,1080")
    options.add_argument("--disable-dev-shm-usage")
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-blink-features=AutomationControlled")
    options.add_argument(f"--user-agent={USER_AGENT}")
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.page_load_strategy = "eager"

    driver = webdriver.Chrome(options=options)
    driver.set_page_load_timeout(45)
    driver.implicitly_wait(0)
    return driver

if __name__ == "__main__":
    driver = build_driver()
    driver.get("https://books.toscrape.com/")
    print(driver.title)
    driver.quit()

Two of those lines deserve attention. page_load_strategy = "eager" returns control at DOMContentLoaded instead of waiting for every image and tracking beacon, which typically halves navigation time on ad-heavy pages. And implicitly_wait(0) is deliberate: mixing implicit and explicit waits produces wait times that are the product rather than the maximum of the two, a trap explained in detail in Explicit vs Implicit Waits in Selenium.

2. Wait for the state you actually need

An element passes through several distinct states, and the condition you wait on decides which one you get. Waiting for presence and then reading .text is the single most common source of empty strings in Selenium scrapers: the node exists in the DOM but has no layout box yet, so its text is empty.

from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.remote.webdriver import WebDriver
from selenium.webdriver.support.ui import WebDriverWait

def wait_for_rows(driver: WebDriver, selector: str, minimum: int = 1, timeout: float = 20.0) -> list:
    """Block until at least `minimum` visible elements match `selector`."""
    wait = WebDriverWait(driver, timeout, poll_frequency=0.25)
    try:
        wait.until(
            lambda d: len(d.find_elements(By.CSS_SELECTOR, selector)) >= minimum
            and d.find_elements(By.CSS_SELECTOR, selector)[0].is_displayed()
        )
    except TimeoutException:
        raise TimeoutException(
            f"{selector!r} never reached {minimum} visible nodes in {timeout}s"
        ) from None
    return driver.find_elements(By.CSS_SELECTOR, selector)

if __name__ == "__main__":
    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options

    opts = Options()
    opts.add_argument("--headless=new")
    d = webdriver.Chrome(options=opts)
    d.get("https://books.toscrape.com/")
    rows = wait_for_rows(d, "article.product_pod h3 a", minimum=20)
    print(len(rows), "products")
    d.quit()

Waiting for a count rather than a single node matters on paginated grids, where the first card renders long before the rest of the batch. Selenium ships ready-made conditions for the common cases — presence_of_element_located, visibility_of_element_located, element_to_be_clickable, text_to_be_present_in_element — and a custom lambda for everything else. poll_frequency defaults to 0.5 s; lowering it to 0.25 s costs a few extra WebDriver round trips but shaves real time off short waits.

3. Extract from a snapshot rather than from live handles

Every element.text call is a WebDriver round trip. Extracting thirty fields from a product card individually costs thirty round trips; pulling page_source once and parsing it with lxml costs one. On a typical listing page this is the difference between eight seconds and half a second.

from lxml import html as lxml_html
from selenium.webdriver.remote.webdriver import WebDriver

def snapshot_products(driver: WebDriver) -> list[dict[str, str]]:
    """Parse the rendered DOM once and return every product card as a dict."""
    tree = lxml_html.fromstring(driver.page_source)
    products: list[dict[str, str]] = []
    for card in tree.cssselect("article.product_pod"):
        link = card.cssselect("h3 a")[0]
        price = card.cssselect("p.price_color")[0]
        products.append(
            {
                "title": link.get("title", "").strip(),
                "url": link.get("href", "").strip(),
                "price": price.text_content().strip(),
            }
        )
    return products

Use the browser only for the things a browser is uniquely good at — running scripts, holding session state, dispatching input — and hand the resulting HTML to a fast parser. Selector strategy is the same as for static pages; Selecting Elements with XPath and CSS Selectors covers writing expressions that survive a redesign.

4. Drive lazy loading with a content-based stop condition

Scroll loops that sleep for a fixed interval fail in both directions: too short and you truncate the dataset, too long and you waste minutes per page. Anchor the loop on observed item count and stop when it stops growing.

import time
from selenium.webdriver.common.by import By
from selenium.webdriver.remote.webdriver import WebDriver

def scroll_until_stable(
    driver: WebDriver,
    item_selector: str,
    max_rounds: int = 25,
    settle_seconds: float = 1.5,
) -> int:
    """Scroll to the bottom repeatedly until the item count stops increasing."""
    seen = len(driver.find_elements(By.CSS_SELECTOR, item_selector))
    stalled = 0
    for _ in range(max_rounds):
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(settle_seconds)
        current = len(driver.find_elements(By.CSS_SELECTOR, item_selector))
        if current == seen:
            stalled += 1
            if stalled >= 2:
                break
        else:
            stalled = 0
            seen = current
    return seen

Requiring two consecutive stalled rounds absorbs a single slow network response without running the loop forever. The wider set of pagination shapes — numbered pages, cursor parameters, "load more" buttons — is catalogued in Handling Pagination and Infinite Scroll.

5. Capture the JSON the page fetches for itself

When a page renders from an XHR response, the DOM is a lossy re-encoding of data you could have had directly. Chrome DevTools Protocol lets you enable network tracking and read response bodies without a proxy, which usually gives you cleaner fields and full precision on numbers the UI rounds.

import json
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

def capture_json(url: str, url_fragment: str) -> list[dict]:
    """Load a page and return the parsed bodies of matching XHR responses."""
    options = Options()
    options.add_argument("--headless=new")
    options.set_capability("goog:loggingPrefs", {"performance": "ALL"})
    driver = webdriver.Chrome(options=options)
    captured: list[dict] = []
    try:
        driver.get(url)
        driver.execute_cdp_cmd("Network.enable", {})
        for entry in driver.get_log("performance"):
            message = json.loads(entry["message"])["message"]
            if message.get("method") != "Network.responseReceived":
                continue
            response = message["params"]["response"]
            if url_fragment not in response["url"]:
                continue
            body = driver.execute_cdp_cmd(
                "Network.getResponseBody", {"requestId": message["params"]["requestId"]}
            )
            captured.append(json.loads(body["body"]))
    finally:
        driver.quit()
    return captured

Network.getResponseBody only works while the response is still in the renderer's buffer, so read it during or immediately after the page load rather than at the end of a long session. Once you have the endpoint shape, ask whether you still need the browser at all — replaying the call from a plain HTTP client is faster and far cheaper.

6. Persist cookies so logins survive a restart

Re-authenticating on every run wastes time and generates exactly the burst of login traffic that anti-abuse systems watch for. Dump the cookie jar after a successful session and reload it on the next start.

import json
from pathlib import Path
from selenium.webdriver.remote.webdriver import WebDriver

COOKIE_FILE = Path("session_cookies.json")

def save_cookies(driver: WebDriver) -> None:
    """Write the current cookie jar to disk."""
    COOKIE_FILE.write_text(json.dumps(driver.get_cookies(), indent=2), encoding="utf-8")

def load_cookies(driver: WebDriver, origin: str) -> bool:
    """Restore a saved cookie jar; returns False when nothing was saved."""
    if not COOKIE_FILE.exists():
        return False
    driver.get(origin)
    for cookie in json.loads(COOKIE_FILE.read_text(encoding="utf-8")):
        cookie.pop("sameSite", None)
        driver.add_cookie(cookie)
    driver.get(origin)
    return True

Cookies can only be added for the domain the browser is currently on, which is why load_cookies navigates to the origin first, injects, then reloads. The cross-library view of the same problem lives in Managing Cookies and Sessions.

7. Reach content inside iframes and shadow roots

Two DOM boundaries silently break otherwise correct selectors. An <iframe> hosts a separate document that find_element cannot see from the parent, and a shadow root encapsulates a component's internals so they never appear in page_source at all. Both are common in embedded players, payment widgets, review sections and design-system components.

from selenium.webdriver.common.by import By
from selenium.webdriver.remote.webdriver import WebDriver

def text_inside_frame(driver: WebDriver, frame_css: str, target_css: str) -> str:
    """Switch into an iframe, read one element, and switch back out."""
    frame = driver.find_element(By.CSS_SELECTOR, frame_css)
    driver.switch_to.frame(frame)
    try:
        return driver.find_element(By.CSS_SELECTOR, target_css).text
    finally:
        driver.switch_to.default_content()

def text_inside_shadow_root(driver: WebDriver, host_css: str, target_css: str) -> str:
    """Pierce a component's shadow root and read one element inside it."""
    script = (
        "const host = document.querySelector(arguments[0]);"
        "if (!host || !host.shadowRoot) { return ''; }"
        "const node = host.shadowRoot.querySelector(arguments[1]);"
        "return node ? node.textContent.trim() : '';"
    )
    return driver.execute_script(script, host_css, target_css)

Always return to default_content() in a finally block: leaving the driver focused on a frame makes every subsequent selector fail with a confusing NoSuchElementException on a page that visibly contains the element. Nested frames need one switch_to.frame call per level. Closed shadow roots (mode: "closed") expose no shadowRoot property at all, and in that case the only route to the data is the network response that populated the component.

Performance and Scaling Considerations

The dominant cost in Selenium is not rendering, it is the number of commands you send. Each find_element, .text and .get_attribute becomes a JSON document travelling over a local HTTP connection to a separate driver process, which forwards it into Chrome and waits for the reply.

A single command costs roughly 2-8 ms end to end on a warm loopback connection. That sounds negligible until a loop over 500 rows with six fields each turns into 3,000 round trips and twelve seconds of pure protocol overhead. Three habits recover most of that time:

• Batch through the DOM. One page_source snapshot plus lxml parsing replaces hundreds of round trips, as in step 3. Where you must stay in the browser, driver.execute_script can return a whole list of dictionaries in a single call.
• Block what you do not need. Images, fonts, media and third-party analytics often account for 70-80% of bytes on a commercial page. Network.setBlockedURLs over CDP or a Chrome preference that disables images reduces both bandwidth and layout work.
• Reuse the process. Launching Chrome costs 250-600 ms and a fresh profile directory. Keep one driver per worker for the lifetime of a batch and call driver.delete_all_cookies() between targets rather than restarting.

For concurrency, run one driver per OS process rather than per thread. Selenium's Python bindings are thread-safe enough for simple use, but Chrome's memory profile makes processes the natural unit of isolation and the natural unit of failure containment. A practical starting point on an 8 GB, 4-core worker is six concurrent drivers with a hard per-URL timeout; past that, page-load latency rises faster than throughput. Wrap navigation in a retry policy with jittered backoff — Retrying Failed Requests with Tenacity covers the decorator pattern — and always call driver.quit() in a finally block, because an orphaned Chrome process keeps its full memory footprint until the machine reboots. When you move the fleet off your laptop, the packaging and memory limits in Deploying Scrapers to the Cloud apply directly.

Rate limiting deserves its own budget line. A browser issues dozens of sub-requests per page, so ten "pages per minute" can be several hundred requests per minute at the server. Measure what you actually emit before deciding your crawl is polite.

Common Errors and Fixes

StaleElementReferenceException: stale element not found in the current frame. You held a WebElement across a re-render. React, Vue and Svelte replace nodes rather than mutating them, so any handle taken before the update points at a detached element. Never store element references across a navigation, a scroll that triggers loading, or a state change — re-find inside the retry loop:

from selenium.common.exceptions import StaleElementReferenceException
from selenium.webdriver.common.by import By
from selenium.webdriver.remote.webdriver import WebDriver

def read_text(driver: WebDriver, selector: str, attempts: int = 3) -> str:
    """Read element text, re-finding the node if the DOM re-renders."""
    for _ in range(attempts):
        try:
            return driver.find_element(By.CSS_SELECTOR, selector).text
        except StaleElementReferenceException:
            continue
    raise StaleElementReferenceException(f"{selector!r} kept going stale")

ElementClickInterceptedException: Other element would receive the click. A cookie banner, sticky header or modal overlays your target. The exception message names the intercepting element, so dismiss it first; driver.execute_script("arguments[0].click();", el) also works but skips the real event sequence, which some applications depend on.

TimeoutException with no message. Your condition never became true. Before increasing the timeout, save driver.page_source and a driver.save_screenshot("debug.png") — nine times out of ten the page is a consent wall, a geographic redirect or a challenge page rather than the content you expected.

SessionNotCreatedException: This version of ChromeDriver only supports Chrome version N. Chrome auto-updated underneath a cached driver. Upgrade Selenium so Selenium Manager refreshes the binary, or pin both Chrome and the driver inside your container image so background updates cannot break a running fleet.

WebDriverException: unknown error: DevToolsActivePort file doesn't exist. Chrome failed to start, almost always in a container without --no-sandbox or with a /dev/shm smaller than 64 MB. Add --no-sandbox and --disable-dev-shm-usage, or raise the shared-memory size with docker run --shm-size=1g.

InvalidArgumentException: invalid argument: invalid 'expiry' when restoring cookies. Selenium requires expiry to be an integer; some sources serialize it as a float. Cast it with cookie["expiry"] = int(cookie["expiry"]) before add_cookie, or drop the key to create a session cookie.

ElementNotInteractableException: element not interactable. The node exists and is visible to is_displayed() but has zero size, is behind a pointer-events: none layer, or is a hidden file input. Scroll it into view with driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", el) first; for file inputs, send the path with send_keys rather than clicking.

urllib3.exceptions.ReadTimeoutError from the driver connection. The browser is busy longer than the command timeout — usually a synchronous script or a very large DOM. Raise the client timeout with driver.command_executor.set_timeout(120) and investigate why a single command takes that long.

Pages render blank in headless but fine headed. The site keys layout off viewport size or refuses to serve without a GPU. Set an explicit --window-size, and if the page still differs, run headful under a virtual display. Detection-driven differences are a separate matter, addressed in Browser Fingerprint and Stealth Configuration and, for WebDriver stacks specifically, in How to Configure Selenium Stealth to Avoid Detection.

Frequently Asked Questions

Do I still need explicit waits if I set an implicit wait? Yes, and mixing them is actively harmful. An implicit wait makes every find_element poll internally, so an explicit wait wrapped around a find can multiply the two timeouts instead of respecting the larger one. Set implicitly_wait(0) and use WebDriverWait everywhere so each wait has one clearly defined budget.

How much memory should I budget per headless Chrome? Plan for 180 MB on a simple page and 400 MB or more once a content-heavy site with several iframes has loaded. Add roughly 50 MB for the driver process itself. On an 8 GB worker that means six to ten concurrent browsers with headroom, and you should monitor resident size rather than assume it stays flat across a long run.

Can Selenium read responses from XHR requests the page makes? Yes, through the Chrome DevTools Protocol. Enable performance logging, watch for Network.responseReceived events, and call Network.getResponseBody with the request id while the body is still buffered. This often returns richer data than the DOM, and it frequently reveals that you could skip the browser entirely.

Why does the same script pass locally and fail in CI? Containers change three things at once: no GPU, a tiny /dev/shm, and a different default window size. Add --no-sandbox, --disable-dev-shm-usage and an explicit --window-size, then pin the Chrome version in the image so an upstream update cannot silently change rendering behaviour mid-week.

Is Selenium slower than Playwright for scraping? Usually, because the WebDriver protocol adds a process hop and an HTTP round trip to every command, while Playwright multiplexes over one persistent connection. The gap is largest for chatty scripts and smallest for scripts that navigate once and take a single DOM snapshot, which is the pattern this guide recommends regardless of framework.

Related

• Advanced Scraping Techniques and Anti-Bot Evasion — the parent section for this topic
• Explicit vs Implicit Waits in Selenium — why the two wait models must not be mixed
• How to Configure Selenium Stealth to Avoid Detection — patching the automation tells a WebDriver session exposes
• Using Playwright for Modern Web Automation — the newer automation model and when it wins
• Rotating Proxies and Managing IP Blocks — giving each browser worker its own exit address