Python Web Scraping
Reading layout

Explicit vs Implicit Waits in Selenium

Synchronisation is where most Selenium scrapers actually break, so this page settles the question the parent guide Mastering Selenium for Dynamic Websites raises: which wait mechanism to use, and what happens when you use both.

Cost of probing twenty absent fields under different wait settings With a ten second implicit wait, twenty absent fields cost two hundred seconds per page. A two second implicit wait costs forty seconds. With no implicit wait and a single explicit wait, the same page costs ten seconds. Probing 20 optional fields that are all absentimplicitly_wait(10)global, every lookup200 s per pageimplicitly_wait(2)still global40 s per pageno implicit waitone WebDriverWait10 s once, then instant misses
An implicit wait charges its full timeout on every lookup that finds nothing, so the cost scales with the number of optional fields you probe.

The answer in one line: set the implicit wait to zero and use WebDriverWait with expected_conditions everywhere. An implicit wait is a single global polling timeout applied inside the driver to every element lookup you will ever make; it cannot express what you are waiting for, it charges its full duration on every lookup that legitimately finds nothing, and combined with an explicit wait it produces timeouts that are longer than either value and hard to reason about. The Selenium documentation itself warns against mixing them. An explicit wait is scoped to one condition, states the intent in the code, and fails with an error message that tells you which condition never became true.

How Each Mechanism Actually Works

An implicit wait is configured once with driver.implicitly_wait(10) and lives inside the driver process, not your script. From that point on, every find_element and find_elements call that does not immediately match enters a polling loop in the driver, retrying roughly every 500 ms until the element appears or the timeout expires. Two properties follow from that, and both cause trouble.

It is global and invisible. Nothing at the call site tells a reader that this particular find_element may block for ten seconds. A helper written months ago in another module inherits the setting, and changing the value to debug one page silently changes the behaviour of every page.

It only ever waits for presence in the DOM. That is the one condition it knows. It cannot wait for an element to become visible, to become enabled, to stop moving, to contain particular text, or to disappear. On any framework that renders a container first and populates it a moment later — which is to say all of them — the implicit wait returns an element that exists and is empty, and your extraction reads an empty string with no error at all. That silent-wrong-answer failure is worse than a crash because nothing in the run signals it.

An explicit wait is constructed per use: WebDriverWait(driver, 10, poll_frequency=0.5) returns an object whose .until() method takes a callable, invokes it against the driver every poll interval, and returns the first truthy result. expected_conditions is simply a library of such callables. Because the condition is a parameter, the wait can express anything — visible, clickable, invisible, text present, count reached, your own predicate — and because it is local, a reader sees the cost and the intent at the call site.

Why time.sleep Is a Smell, Not a Wait

time.sleep(3) between actions is a symptom that the script has no model of when the page is ready. It pays the full three seconds when the element arrived in 200 ms, and it still fails when a slow response takes four. On a crawl of 5,000 pages, three seconds of unnecessary sleep per page is more than four hours of wall clock burnt waiting for nothing.

There is exactly one honest use for a fixed sleep, and it is not synchronisation: rate limiting. Pausing deliberately between requests to stay inside a site's tolerance is a policy decision about politeness, and a fixed delay is the right tool. Waiting for the DOM is a different problem, and a fixed delay is the wrong tool for it every time.

The Compounding Cost of a Negative Lookup

The strongest practical argument against implicit waits is what they do when an element is genuinely absent — the case that occurs constantly in real scraping, because optional fields are optional.

Consider a product page where you probe twenty optional attributes: a sale badge, a stock warning, a delivery estimate, a size guide, and so on. On a product that has none of them, find_elements returns an empty list — but with a ten-second implicit wait, each of those twenty calls first spends ten seconds in the driver's polling loop hoping the element shows up. That is 200 seconds on a page that rendered fully in under a second. Drop the implicit wait to two seconds and it is still 40 seconds. With the implicit wait at zero, all twenty calls return empty in a few milliseconds each and a single explicit wait handles the one element you actually depend on.

This is why the cost is invisible in development and brutal in production: you notice it only when the field is missing, and in development you test against the page where the field is present. A crawl that ran at two seconds per page in testing runs at ninety in production, and the profile shows the time inside the driver, not inside your code.

Never Mix Them

Implicit and explicit waits composing into a longer timeout A ten second WebDriverWait polls repeatedly, and each poll calls a finder that itself blocks for the five second implicit timeout, so the total elapsed time overruns the ten second budget. WebDriverWait(driver, 10)poll 1 → find_elementblocks 5 s inside the driverpoll 2 → find_elementblocks 5 s inside the driverpoll 3 → find_elementblocks 5 s inside the driverThe two timers composenothing takes the shorter oneElapsed: 15 s, not 10the 10 s budget is checkedbetween polls, not during them
An explicit wait polls by calling a finder, and each of those calls re-enters the implicit timeout, so the two budgets multiply instead of taking the shorter one.

The mixing failure is subtle enough that it survives code review. An explicit wait works by polling a finder. If an implicit wait is also set, each of those polls enters the driver's own polling loop first and blocks for the implicit timeout before returning "not found" to the explicit wait, which then decides whether to poll again.

The two timers therefore compose rather than taking the shorter value. WebDriverWait(driver, 10) combined with implicitly_wait(5) does not time out at 5 seconds or at 10; the explicit wait checks its own deadline only between polls, so a poll that starts at 9.9 seconds still blocks a further 5 seconds inside the driver before the deadline is even re-examined. You observe roughly 15 seconds and the traceback claims a 10-second timeout. Vary the implicit value or the poll frequency and the observed total moves around unpredictably, which is exactly why the resulting flakiness is so hard to diagnose from logs.

Conditions that wait for something to stop being true — invisibility_of_element_located, staleness_of — invert the pathology. Each poll asks "is it gone yet?", and each poll first waits out the implicit timeout hoping the element appears. A condition that should resolve the instant a spinner disappears now resolves no faster than the implicit timeout allows.

The remedy is one line at driver construction: leave implicitly_wait unset, or set it explicitly to zero if some inherited helper might set it. Then every wait in the codebase is visible at its call site.

Choosing the Right Condition, and Writing Your Own

Four intents mapped to expected conditions Reading rendered text maps to presence of element located, clicking maps to element to be clickable, waiting for a spinner maps to invisibility of element located, and anything else maps to a custom callable. Pick the condition that matches the intentYou only need to read textpresence_of_element_locatedcheapest; ignores visibilityYou are about to clickelement_to_be_clickablevisible and enabled, not just thereA spinner must go awayinvisibility_of_element_locatedstops you reading stale valuesNothing built in fitsyour own callable(driver)return a value, or False to keep polling
Every explicit wait states an intent. Pick the condition that matches what you are about to do with the element, not the one that returns soonest.

presence_of_element_located is the cheapest condition and the most over-used. It resolves as soon as the node exists in the DOM, which is fine when you only need an attribute value, and wrong whenever the element is rendered before its content or is still hidden behind a CSS transition. visibility_of_element_located additionally requires a non-zero bounding box; element_to_be_clickable requires visible and enabled, which is what you want before any interaction; invisibility_of_element_located is how you wait out a loading overlay before reading values that would otherwise be stale placeholders.

When none of them fits, write a callable. An expected condition is any object callable with the driver that returns a truthy value on success or a falsy value to keep polling. The example below waits for a table to reach a stable row count — an assertion that no built-in condition expresses, and one that comes up constantly on paginated grids.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.remote.webdriver import WebDriver
from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait


class row_count_stable:
    """Wait until a selector's match count stops changing between polls."""

    def __init__(self, css: str, required_repeats: int = 2) -> None:
        self.css = css
        self.required_repeats = required_repeats
        self._last = -1
        self._repeats = 0

    def __call__(self, driver: WebDriver) -> int | bool:
        count = len(driver.find_elements(By.CSS_SELECTOR, self.css))
        if count == self._last and count > 0:
            self._repeats += 1
        else:
            self._last = count
            self._repeats = 0
        return count if self._repeats >= self.required_repeats else False


def build_driver() -> WebDriver:
    """Create a Chrome driver with the implicit wait explicitly disabled."""
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument("--window-size=1440,900")
    options.add_argument(
        "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
    )
    driver = webdriver.Chrome(options=options)
    driver.implicitly_wait(0)
    return driver


def scrape_titles(url: str) -> list[str]:
    """Load a catalogue page and read every product title once the grid settles."""
    driver = build_driver()
    try:
        driver.get(url)
        wait = WebDriverWait(driver, 15, poll_frequency=0.4)
        wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "article.product_pod")))
        wait.until(row_count_stable("article.product_pod h3 a"))
        links: list[WebElement] = driver.find_elements(
            By.CSS_SELECTOR, "article.product_pod h3 a"
        )
        return [link.get_attribute("title") or "" for link in links]
    finally:
        driver.quit()


if __name__ == "__main__":
    titles = scrape_titles("https://books.toscrape.com/catalogue/page-1.html")
    print(f"{len(titles)} titles")
    print(titles[:3])

Note the poll_frequency=0.4 and the driver.implicitly_wait(0). The first tightens the loop so a stability check costs less wall clock; the second is what makes the custom condition's find_elements return instantly when the count is momentarily zero, instead of blocking on a hidden global timeout.

Handling Staleness

StaleElementReferenceException is the other half of the synchronisation problem, and no amount of waiting before a lookup prevents it. A WebElement is a handle to a node in a particular document; if the framework re-renders that subtree, the handle points at a node no longer attached and every method on it raises.

The distinction that matters: waits protect you before you obtain a reference, staleness bites after. The three practical defences are to keep references short-lived — find the element immediately before using it rather than caching a list across a re-render; to re-find after any action that triggers a render, such as a sort control or a filter checkbox; and to wait for the old node to go stale before hunting for the new one, using EC.staleness_of(old_element) so you do not race the framework and grab the outgoing node.

For loops over a re-rendering list, extract the identifying data first — hrefs, ids — and then iterate over those plain strings, re-finding each element by its identifier when you need it. Iterating over a cached list of WebElement objects across a re-render is the single most common source of staleness in scraping code.

Edge Cases and Caveats

  • Selenium 4.x timeout units. implicitly_wait() and WebDriverWait take seconds in Python. The driver.timeouts interface and several other language bindings use milliseconds; a value copied from a Java example becomes a 10,000-second wait.
  • ignored_exceptions widens the net. WebDriverWait(driver, 10, ignored_exceptions=[StaleElementReferenceException]) lets a condition survive a re-render mid-poll instead of aborting. NoSuchElementException is ignored by default.
  • Remote grids add latency per poll. Against a remote Selenium Grid every poll is a network round-trip. A poll_frequency of 0.1 that is free locally becomes significant load remotely; 0.4 to 0.5 is a sensible floor.
  • Conditions returning elements versus booleans. presence_of_element_located returns the element, so element = wait.until(...) is idiomatic. invisibility_of_element_located and text_to_be_present_in_element return booleans — assigning them and calling .text produces a confusing AttributeError.
  • A page-load timeout is separate. driver.set_page_load_timeout() governs navigation, not element lookups. A hanging third-party script can block driver.get() regardless of any element wait.
  • Waits do not defeat detection. Perfect synchronisation still leaves a machine-speed interaction rhythm. Pair it with the measures in how to configure Selenium stealth to avoid detection when the target scores behaviour.

Frequently Asked Questions

Can I set an implicit wait as a safety net and still use explicit waits? No — that is precisely the combination to avoid. Each poll of the explicit wait re-enters the implicit timeout inside the driver, so the observed timeout exceeds both configured values and varies between runs. Set the implicit wait to zero and let every wait be explicit and visible.

Which expected condition should I use by default? Use presence_of_element_located when you only need to read an attribute or text from a node that is populated on arrival, and element_to_be_clickable before any interaction. Reach for visibility_of_element_located when the element is rendered hidden and revealed by a transition.

How do I avoid StaleElementReferenceException in a loop? Do not hold WebElement references across anything that re-renders the list. Collect identifying strings such as hrefs or ids first, then re-find each element by its identifier inside the loop, and use EC.staleness_of(old_element) when you need to wait for a re-render to complete.

Is time.sleep ever acceptable in a Selenium scraper? Only for rate limiting, where a deliberate pause between requests is the point. Using it to wait for the DOM is always either slower than necessary or unreliable, and often both on the same page.

Back to home