Asynchronous Scraping with Asyncio and HTTPX

Asynchronous I/O is the cheapest speed-up available to a Python scraper, and this guide — part of Scaling Python Web Scrapers — covers how to get it without turning a polite crawler into an accidental flood. The scope is a single process fetching many URLs at once: the event loop model, how to bound concurrency, and how to keep one failed request from destroying a batch of two hundred.

The reason async wins is arithmetic, not magic. A request to a typical page spends roughly 20–40 ms on DNS, 40–120 ms on the TCP and TLS handshake, 100–400 ms waiting for the server to generate a response, and 5–20 ms actually reading bytes off the socket. Python is idle for well over 90% of that. A sequential loop over 1,000 URLs at 350 ms each takes just under six minutes; the same 1,000 URLs at a concurrency of 20 take about eighteen seconds, on the same machine, over the same connection, with the same parser. Nothing got faster — the process simply stopped waiting one request at a time. The request and response fundamentals underneath all of this are covered in Understanding HTTP Requests and Responses.

When to Use Async Scraping

Async pays off when the workload is I/O-bound and the number of outstanding operations is large. It costs you a stricter set of rules about what may be called inside a coroutine, and that cost is not always worth paying.

Two numbers usually decide it. If the total request count is below about 200, a synchronous script finishes before you have finished writing the async version. Above a few thousand, and especially if the pages are on a handful of hosts where connection reuse matters, async is the correct default. Above roughly 50,000 URLs per run you are into scheduling and resume territory, which is where Web Scraping with Scrapy or a distributed queue starts to earn its keep.

There is a third factor that is easy to miss: async also gives you a single, cheap place to enforce politeness. One semaphore in one process controls every outbound request, which is far easier to reason about than coordinating rate limits across a thread pool.

Prerequisites

You need Python 3.10 or newer. Python 3.11 is strongly preferred because it adds asyncio.TaskGroup and ExceptionGroup, which remove an entire category of orphaned-task bugs described later on this page.

python3 --version          # 3.10 or newer
pip install "httpx[http2]==0.27.2" "selectolax==0.3.21" "tenacity==9.0.0"

The [http2] extra pulls in h2 and enables HTTP/2 multiplexing, which matters when many requests target the same host. selectolax is a fast C-backed HTML parser used later for the parsing step; if you prefer a more forgiving API, Parsing HTML with BeautifulSoup covers the alternative. On Linux, raise the open-file limit before running with high concurrency, since every in-flight connection consumes a file descriptor:

ulimit -n 4096

Step-by-Step: Building a Concurrent HTTPX Scraper

1. Create One Client and Reuse It

The single most common async performance bug is creating a client per request. A fresh AsyncClient has an empty connection pool, so every request pays a full DNS lookup plus TCP and TLS handshake — typically 150 ms of pure overhead that a reused client pays once.

import asyncio
import httpx

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
}

LIMITS = httpx.Limits(max_connections=50, max_keepalive_connections=20)
TIMEOUT = httpx.Timeout(connect=5.0, read=15.0, write=5.0, pool=30.0)


async def fetch(client: httpx.AsyncClient, url: str) -> str:
    response = await client.get(url)
    response.raise_for_status()
    return response.text


async def main() -> None:
    urls = [f"https://books.toscrape.com/catalogue/page-{n}.html" for n in range(1, 11)]
    async with httpx.AsyncClient(
        headers=HEADERS, limits=LIMITS, timeout=TIMEOUT, http2=True, follow_redirects=True
    ) as client:
        pages = [await fetch(client, url) for url in urls]
    print(f"fetched {len(pages)} pages, {sum(len(p) for p in pages)} bytes")


asyncio.run(main())

That code is correct but not yet concurrent — the list comprehension awaits each fetch in turn. It is worth running first anyway, because it establishes that the client, headers and timeouts work before concurrency hides the errors.

Note the explicit httpx.Timeout with four separate values. HTTPX's default is a flat five seconds on every phase, which is too aggressive for a slow origin server and too generous for a black-holed connection. Splitting them means a dead host fails in five seconds while a slow-but-alive page still gets fifteen to respond.

The await in fetch is what makes the whole scheme work. When the coroutine reaches it, HTTPX hands the socket to the loop's selector and returns control; the loop is then free to advance any other ready task. Nothing runs in parallel — there is exactly one thread — but the thread stops being blocked. That is also why a single blocking call anywhere in the chain is so destructive: it never yields, so the loop cannot advance anything else until it returns.

2. Run Requests Concurrently and Gate Them with a Semaphore

asyncio.gather schedules every coroutine at once. Against ten URLs that is fine; against ten thousand it opens ten thousand sockets, exhausts your file descriptors, and hits the target harder than most load tests. An asyncio.Semaphore caps how many coroutines may be inside the request section at any moment.

import asyncio
import httpx


async def fetch_limited(
    client: httpx.AsyncClient, url: str, gate: asyncio.Semaphore
) -> tuple[str, str]:
    async with gate:
        response = await client.get(url)
        response.raise_for_status()
        await asyncio.sleep(0.15)          # politeness pause, still inside the gate
        return url, response.text


async def crawl(urls: list[str], concurrency: int = 8) -> list[tuple[str, str]]:
    gate = asyncio.Semaphore(concurrency)
    async with httpx.AsyncClient(
        headers=HEADERS, limits=LIMITS, timeout=TIMEOUT, http2=True, follow_redirects=True
    ) as client:
        tasks = [fetch_limited(client, url, gate) for url in urls]
        return await asyncio.gather(*tasks)


if __name__ == "__main__":
    targets = [f"https://books.toscrape.com/catalogue/page-{n}.html" for n in range(1, 26)]
    results = asyncio.run(crawl(targets, concurrency=8))
    print(f"{len(results)} pages, mean size {sum(len(h) for _, h in results) // len(results)}")

Keeping asyncio.sleep inside the async with gate block is deliberate. If the sleep happens after the semaphore is released, the next task starts immediately and the delay does nothing; inside the block, the gate is held for the request plus the pause, so eight permits and a 0.15 s pause give a ceiling of roughly 53 requests per second even if the server responds instantly. Sizing and layering these limits is covered in depth in Limiting Concurrency with Semaphores.

One semaphore per host is usually what you want rather than one global one, because a crawl that touches thirty domains should not throttle domain B because domain A is slow. A defaultdict keyed on urlsplit(url).netloc is enough.

3. Contain Failures So One Timeout Does Not Cost the Batch

By default asyncio.gather re-raises the first exception to the caller — and leaves every sibling task running, unwatched. Their results are discarded and their own exceptions surface later as Task exception was never retrieved warnings. On a 200-URL batch, one httpx.ReadTimeout can therefore throw away 199 successful fetches.

There are two correct fixes. return_exceptions=True turns failures into ordinary list entries so you can partition the results:

import asyncio
import httpx


async def crawl_tolerant(urls: list[str], concurrency: int = 8) -> dict[str, str]:
    gate = asyncio.Semaphore(concurrency)
    async with httpx.AsyncClient(
        headers=HEADERS, limits=LIMITS, timeout=TIMEOUT, follow_redirects=True
    ) as client:
        tasks = [fetch_limited(client, url, gate) for url in urls]
        settled = await asyncio.gather(*tasks, return_exceptions=True)

    pages: dict[str, str] = {}
    failures: list[tuple[str, str]] = []
    for url, outcome in zip(urls, settled):
        if isinstance(outcome, BaseException):
            failures.append((url, type(outcome).__name__))
        else:
            pages[outcome[0]] = outcome[1]

    print(f"ok={len(pages)} failed={len(failures)}")
    for url, kind in failures[:5]:
        print(f"  {kind}: {url}")
    return pages

On Python 3.11+, asyncio.TaskGroup is the better choice when partial results are useless — it cancels the remaining siblings and raises a single ExceptionGroup containing everything that went wrong, with no orphaned tasks left behind:

import asyncio
import httpx


async def crawl_strict(urls: list[str], concurrency: int = 8) -> list[str]:
    gate = asyncio.Semaphore(concurrency)
    collected: list[str] = []

    async def worker(client: httpx.AsyncClient, url: str) -> None:
        async with gate:
            response = await client.get(url)
            response.raise_for_status()
            collected.append(response.text)

    async with httpx.AsyncClient(headers=HEADERS, timeout=TIMEOUT) as client:
        async with asyncio.TaskGroup() as group:
            for url in urls:
                group.create_task(worker(client, url))
    return collected

Pick tolerant collection for a wide crawl where 3% loss is acceptable and retried later, and strict cancellation for a pipeline where a partial dataset is worse than no dataset.

4. Retry Transient Failures with Backoff and Jitter

Rate limits, gateway hiccups and connection resets are routine above a few thousand requests. The rule is to retry only what is plausibly transient — 429, 502, 503, 504, and transport errors — and never a 404 or 403, which will fail identically forever.

import asyncio
import random
import httpx

RETRYABLE_STATUS = {429, 500, 502, 503, 504}


async def fetch_retrying(
    client: httpx.AsyncClient, url: str, gate: asyncio.Semaphore, attempts: int = 4
) -> str | None:
    for attempt in range(attempts):
        try:
            async with gate:
                response = await client.get(url)
            if response.status_code in RETRYABLE_STATUS:
                raise httpx.HTTPStatusError(
                    "retryable", request=response.request, response=response
                )
            response.raise_for_status()
            return response.text
        except httpx.HTTPStatusError as exc:
            if exc.response.status_code not in RETRYABLE_STATUS:
                return None
            retry_after = exc.response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after and retry_after.isdigit() else 2**attempt
        except httpx.TransportError:
            delay = 2**attempt
        if attempt == attempts - 1:
            return None
        await asyncio.sleep(delay + random.uniform(0, 0.5))
    return None

Two details matter more than the loop itself. Honouring Retry-After on a 429 is the difference between a server that tolerates you and one that escalates to a block. And the random.uniform(0, 0.5) jitter breaks up the thundering herd: without it, fifty tasks that all hit a 503 at the same instant will all retry at exactly the same instant, reproducing the overload. If you would rather declare the policy than hand-roll it, Retrying Failed Requests with Tenacity covers the decorator approach.

Note the semaphore is released before the sleep here, unlike in step 2. A task that is backing off is not making requests, so it should not hold a permit that a healthy task could use.

5. Keep Parsing Off the Event Loop

Parsing is CPU work. A coroutine that spends 40 ms in lxml blocks every other task for 40 ms, and at a concurrency of 50 that latency compounds badly. For light extraction, parse inline and accept it. For heavy documents, push the work to a process pool with run_in_executor so the loop keeps servicing sockets.

import asyncio
from concurrent.futures import ProcessPoolExecutor

from selectolax.parser import HTMLParser


def extract_books(html: str) -> list[dict[str, str]]:
    tree = HTMLParser(html)
    rows: list[dict[str, str]] = []
    for card in tree.css("article.product_pod"):
        title = card.css_first("h3 a")
        price = card.css_first("p.price_color")
        rows.append(
            {
                "title": title.attributes.get("title", "") if title else "",
                "price": price.text(strip=True) if price else "",
            }
        )
    return rows


async def parse_all(pages: list[str]) -> list[dict[str, str]]:
    loop = asyncio.get_running_loop()
    with ProcessPoolExecutor(max_workers=4) as pool:
        batches = await asyncio.gather(
            *(loop.run_in_executor(pool, extract_books, html) for html in pages)
        )
    return [row for batch in batches for row in batch]

The pickling cost of shipping HTML strings to worker processes is real — roughly 1–3 ms per 100 KB document — so this only wins when parsing takes appreciably longer than that. Measure before adopting it.

6. Emit Results as They Complete

gather returns only when the last coroutine finishes, so a batch of 5,000 pages holds 5,000 response bodies in memory and produces nothing until the slowest one lands. asyncio.as_completed yields each future the moment it resolves, which lets you write rows out continuously and keeps peak memory flat regardless of batch size.

import asyncio
import json

import httpx


async def crawl_streaming(urls: list[str], out_path: str, concurrency: int = 8) -> int:
    gate = asyncio.Semaphore(concurrency)
    written = 0

    async with httpx.AsyncClient(headers=HEADERS, timeout=TIMEOUT) as client:
        tasks = [
            asyncio.create_task(fetch_retrying(client, url, gate)) for url in urls
        ]
        with open(out_path, "w", encoding="utf-8") as handle:
            for future in asyncio.as_completed(tasks):
                html = await future
                if html is None:
                    continue
                for row in extract_books(html):
                    handle.write(json.dumps(row, ensure_ascii=False) + "\n")
                    written += 1
                handle.flush()
    return written


if __name__ == "__main__":
    pages = [f"https://books.toscrape.com/catalogue/page-{n}.html" for n in range(1, 51)]
    total = asyncio.run(crawl_streaming(pages, "books.jsonl"))
    print(f"wrote {total} rows")

The handle.flush() after each page costs a syscall but means a crash at page 900 of 1,000 still leaves 900 pages of usable output on disk. That trade is almost always worth making for a long crawl.

Performance and Scaling Considerations

Throughput is bounded by the smallest limit, not the largest. Requests per second is concurrency / mean_latency, but only if the connection pool allows it. Setting Semaphore(50) while max_connections=10 means forty tasks sit blocked in the pool queue and your effective concurrency is ten. Keep max_connections at or above the semaphore value, and set max_keepalive_connections to roughly the per-host concurrency so sockets are reused rather than reopened.

HTTP/2 changes the arithmetic for single-host crawls. With http2=True, HTTPX multiplexes many streams over one connection, so a hundred requests to one origin need one handshake instead of a hundred. Against a many-host crawl it makes little difference. The measured comparison between clients, including where HTTP/2 helps and where it does not, is in httpx vs aiohttp Async Performance.

Memory is dominated by response bodies, not tasks. A pending coroutine costs a few kilobytes; a 500 KB HTML page held in a list costs 500 KB. Fetching 20,000 pages into memory before writing anything will use several gigabytes. Stream results out as they complete — write to disk or a database inside the task — rather than accumulating a giant list and processing at the end.

The event loop is a single core. One async process will saturate roughly one CPU core on TLS and parsing. Past that, run several processes, each with its own loop and its own URL shard, rather than raising concurrency further. Beyond a single machine, the coordination problem is a different one entirely — see Distributed Crawling with Celery and Redis.

Not re-fetching is faster than fetching fast. On repeat runs, conditional requests and a local response cache typically eliminate 60–90% of traffic, which beats any concurrency tuning. That is the subject of Caching and Incremental Crawling.

Watch the ratio, not the total. A crawl that suddenly runs twice as fast has usually started receiving small block pages instead of real content. Track success rate, mean response size and status-code distribution per run, as described in Monitoring and Alerting for Scrapers.

Common Errors and Fixes

RuntimeError: asyncio.run() cannot be called from a running event loop Raised when asyncio.run is called inside Jupyter, IPython, or another framework that already owns a loop. In a notebook, await the coroutine directly in the cell instead of wrapping it, or install nest_asyncio. In application code the fix is architectural: asyncio.run belongs exactly once, at the top-level entry point.

httpx.PoolTimeout: pool timeout Every connection in the pool is busy and a task waited longer than timeout.pool for a free one. The cause is almost always concurrency above max_connections. Raise httpx.Limits(max_connections=...) to match the semaphore, or lower the semaphore. Raising only the pool timeout hides the queue instead of draining it.

httpx.ConnectError: [Errno 24] Too many open files Each connection holds a file descriptor and the process limit is often 256 or 1024. Cap concurrency, set max_connections explicitly, and raise the limit with ulimit -n 4096. A client that is never closed leaks descriptors too — always use async with httpx.AsyncClient(...).

RuntimeWarning: coroutine 'fetch' was never awaited The coroutine object was created but never scheduled, usually by calling fetch(client, url) without await or without passing it to gather/create_task. Nothing was fetched. Check for a stray call that discards its return value.

asyncio.CancelledError appearing during shutdown Tasks still running when the loop closes get cancelled. Inside a TaskGroup this is normal and expected after a sibling failed. Outside one, it means work was abandoned — collect your tasks and await them, and never swallow CancelledError in a bare except Exception, because that breaks cancellation semantics.

The whole crawl hangs with no error Something blocking is running inside a coroutine: time.sleep, a synchronous requests call, a psycopg2 query, or a large re.sub. Run with asyncio.run(main(), debug=True) and the loop will log any callback that takes longer than 100 ms, naming the offending function.

Every request returns 403 once concurrency rises This is not an async bug. The target is fingerprinting the traffic pattern or the client. Lower concurrency first to confirm, then distribute requests across addresses as described in Rotating Proxies and Managing IP Blocks.

Frequently Asked Questions

How many concurrent requests is safe against one domain? Start at 4 to 8 and treat the server's behaviour as the signal. If the 429 rate stays at zero and mean latency does not climb, step up gradually; if latency rises as you add concurrency, you are already past the point where the server is comfortable. A useful ceiling for an unknown site is one request per second per domain sustained.

Does async make parsing faster? No. Async only removes waiting, and parsing never waits — it uses the CPU continuously. If a profile shows most time inside lxml or BeautifulSoup rather than in socket reads, concurrency will not help and may hurt by starving the loop. Move parsing to a process pool instead.

Should I use HTTPX or aiohttp? HTTPX has a requests-shaped API, supports sync and async from the same code, and speaks HTTP/2, which makes it the easier migration. aiohttp is async-only and slightly faster at very high request rates. For most crawls the difference is under 10% and the API you will maintain matters more.

Can I mix async fetching with a synchronous database driver? Only via an executor. Calling psycopg2 or sqlite3 directly inside a coroutine blocks the loop for the duration of the query. Either use an async driver such as asyncpg, or wrap the write in loop.run_in_executor so it runs on a thread.

Why does my scraper get slower as it runs longer? Usually memory. Accumulating responses, parsed trees or per-task exception objects in a list makes the process grow until it starts swapping. Write results out incrementally and hold only what the next step needs.

Related

• Scaling Python Web Scrapers — where async fits among the other scaling strategies
• Limiting Concurrency with Semaphores — sizing and layering the gate correctly
• httpx vs aiohttp Async Performance — measured throughput on the same workload
• Retrying Failed Requests with Tenacity — declarative retry policy for async code
• Caching and Incremental Crawling — skipping the requests you do not need to make