Limiting Concurrency with Semaphores

Handing ten thousand coroutines to asyncio.gather is the single most common way an otherwise correct async scraper destroys itself, and this article — part of Asynchronous Scraping with asyncio and HTTPX — covers exactly where the bound belongs, what else in the stack is already imposing one, and how to keep several of them consistent.

The short version: create one asyncio.Semaphore sized to what the target tolerates, acquire it around the network call only, and set HTTPX's max_connections to the same number or slightly higher so the pool is never the thing that queues your work. For a crawl that spans several hostnames, keep one global semaphore for your own resource ceiling and a separate, smaller semaphore per host. Anything more elaborate than that — a worker pool over an asyncio.Queue — is worth reaching for only when the URL list is discovered while the crawl runs rather than known up front.

Why Unbounded gather Fails in Three Places at Once

asyncio.gather(*coros) schedules every coroutine immediately. There is no queue, no admission control, and no back-pressure: the event loop starts all of them, and each one runs until it hits its first real await. For a fetch coroutine, that first await is the connection attempt, so within a few milliseconds the process has tried to open one socket per URL.

That collides with three separate ceilings, and which one you hit first depends on the machine and the target.

The first is the file-descriptor limit. Every TCP socket consumes a descriptor. On most Linux distributions the soft limit reported by ulimit -n is 1024, and macOS defaults lower still at 256 for a login shell. Crossing it produces OSError: [Errno 24] Too many open files, which surfaces from deep inside the transport layer rather than from your code, so the traceback rarely points at the line that caused it. Raising the limit with ulimit -n 65535 makes the symptom go away and the underlying problem worse, because the next ceiling is the one that matters.

The second is connection-pool saturation inside the client. An httpx.AsyncClient does not open unlimited sockets; it defers to an httpx.Limits object whose defaults are 100 total connections and 20 idle keep-alive connections. Requests beyond that do not fail — they wait for a free slot, and by default they wait forever, because pool timeout inherits from the timeout argument only if you pass a bare number. Ten thousand coroutines against a hundred pool slots means 9,900 coroutines sitting in a pool queue, each holding a fully materialised Request object plus its headers and its slice of the URL list in memory. The crawl appears to hang while resident memory climbs.

The third, and the only one the target cares about, is its own rate limit. A server that answers comfortably at 8 requests per second will start returning 429 Too Many Requests or 503 Service Unavailable somewhere above that, and the edge in front of it may simply blackhole your IP. This is the ceiling that should determine your number; the other two are just failure modes you meet on the way there if you ignore it. The same politeness reasoning is developed for synchronous crawls in How to Scrape a Static Website Without Getting Blocked.

Because these limits stack, the effective concurrency of a crawl is the minimum of all of them. Setting Semaphore(200) on a host that tolerates eight requests per second does not make the crawl faster; it just moves the queue from your process into the target's error path, where every rejected request still costs a round trip and still counts against your reputation.

Where the Acquire Belongs

The semaphore should be held for the network call and released before parsing. This sounds like a detail and is not: parsing is CPU work, and CPU work inside the guarded block reduces your effective request rate without reducing load on the target.

import asyncio
import httpx
from selectolax.parser import HTMLParser

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}

async def fetch(client: httpx.AsyncClient, url: str, gate: asyncio.Semaphore) -> str:
    async with gate:                      # held only across the network call
        response = await client.get(url)
        response.raise_for_status()
        await asyncio.sleep(0.25)         # politeness, inside the gate
        return response.text

def parse(html: str) -> dict[str, str]:
    tree = HTMLParser(html)               # CPU work, outside the gate
    title = tree.css_first("h1")
    price = tree.css_first("p.price_color")
    return {
        "title": title.text(strip=True) if title else "",
        "price": price.text(strip=True) if price else "",
    }

async def scrape_one(client: httpx.AsyncClient, url: str, gate: asyncio.Semaphore) -> dict[str, str]:
    html = await fetch(client, url, gate)
    return parse(html)

Two details in fetch are deliberate. The await asyncio.sleep(0.25) sits inside the async with gate block, which is what makes it a rate limiter rather than decoration. With N permits and a delay of D seconds, the steady-state request rate is bounded above by N / (D + latency) — with 10 permits, a 0.25 s delay and 0.4 s of server latency, that is about 15 requests per second. Move the sleep outside the block and the permit is released the instant the response arrives, so the delay throttles nothing at all.

The second detail is that raise_for_status() is inside the guarded block too. If it raised outside, the permit would already be free and a burst of 429s would immediately be replaced by a burst of new requests — precisely the wrong response to a server that just asked you to slow down. Handling those statuses properly means backing off, which is covered in Retrying Failed Requests with Tenacity.

Making the Pool Agree with the Gate

The semaphore and httpx.Limits are independent mechanisms, and leaving them mismatched produces confusing behaviour. If the pool is smaller than the semaphore, coroutines that hold a permit still block waiting for a socket, so your permit count no longer describes anything real. If the pool is very much larger, nothing breaks, but the idle keep-alive connections accumulate descriptors for no benefit.

import asyncio
import httpx

CONCURRENCY = 10

async def crawl(urls: list[str]) -> list[dict[str, str] | BaseException]:
    limits = httpx.Limits(
        max_connections=CONCURRENCY,          # match the gate
        max_keepalive_connections=CONCURRENCY,
        keepalive_expiry=30.0,
    )
    timeout = httpx.Timeout(connect=5.0, read=15.0, write=10.0, pool=30.0)
    gate = asyncio.Semaphore(CONCURRENCY)

    async with httpx.AsyncClient(
        headers=HEADERS, limits=limits, timeout=timeout, http2=True, follow_redirects=True
    ) as client:
        tasks = [scrape_one(client, url, gate) for url in urls]
        return await asyncio.gather(*tasks, return_exceptions=True)

urls = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 51)]
results = asyncio.run(crawl(urls))
ok = [r for r in results if isinstance(r, dict)]
print(f"{len(ok)} of {len(urls)} pages parsed")

The explicit pool=30.0 timeout is worth setting even when the numbers match. Without it, a bug that leaks connections turns into a silent hang rather than a httpx.PoolTimeout you can log and alert on. Note also that gather still creates one Task per URL — the semaphore bounds sockets, not tasks. Fifty thousand pending tasks is roughly 50 MB of Python objects before a single byte moves, which is fine at that scale and is not fine at ten million.

Setting http2=True changes the arithmetic slightly. Over HTTP/2 a single connection multiplexes many streams, so max_connections no longer maps one-to-one onto in-flight requests. The semaphore keeps doing exactly what it did, which is another argument for treating it, not the pool, as the authoritative bound. If you are still deciding between clients, httpx vs aiohttp Async Performance measures what that choice actually costs.

Bounding Per Host, Not Just Overall

A single global semaphore is the right tool for a crawl of one site. Point the same code at a mixed list covering forty domains and it becomes wrong in both directions at once: ten global permits is far too slow across forty hosts, and if the list happens to be sorted so that a thousand consecutive URLs share a hostname, all ten permits land on one server anyway.

The fix is a small registry of semaphores keyed by hostname, created lazily, sitting underneath one global gate that protects your own descriptors and memory.

import asyncio
from collections import defaultdict
from urllib.parse import urlsplit

import httpx

class HostGates:
    """One global bound on sockets, plus a per-host bound on politeness."""

    def __init__(self, total: int = 40, per_host: int = 4) -> None:
        self._global = asyncio.Semaphore(total)
        self._per_host = per_host
        self._hosts: dict[str, asyncio.Semaphore] = {}
        self._counts: defaultdict[str, int] = defaultdict(int)

    def _for(self, url: str) -> asyncio.Semaphore:
        host = urlsplit(url).netloc
        if host not in self._hosts:
            self._hosts[host] = asyncio.Semaphore(self._per_host)
        self._counts[host] += 1
        return self._hosts[host]

    async def get(self, client: httpx.AsyncClient, url: str, delay: float = 0.5) -> httpx.Response:
        async with self._global:
            async with self._for(url):
                response = await client.get(url)
                await asyncio.sleep(delay)
                return response

async def main() -> None:
    urls = [
        "https://books.toscrape.com/",
        "https://httpbin.org/headers",
        "https://httpbin.org/user-agent",
        "https://quotes.toscrape.com/",
    ]
    gates = HostGates(total=40, per_host=4)
    async with httpx.AsyncClient(headers=HEADERS, timeout=20.0) as client:
        responses = await asyncio.gather(
            *(gates.get(client, u) for u in urls), return_exceptions=True
        )
    for url, resp in zip(urls, responses):
        status = resp.status_code if isinstance(resp, httpx.Response) else type(resp).__name__
        print(f"{status}  {url}")

asyncio.run(main())

Acquire order matters here. Taking the global permit first and the host permit second means a coroutine can hold a global permit while queued for a busy host, which wastes a slot but cannot deadlock. Reversing the order — host first, then global — allows a coroutine holding the last host permit to wait indefinitely for a global permit that another coroutine is holding while waiting for the same host permit. Consistent acquisition order is the standard defence against that, and with only two levels it costs nothing to get right.

The lazy dict creation is safe because coroutines in one event loop are not preempted between the in check and the assignment; there is no await in that window. If you ever move this to a threaded design, that assumption disappears and you need a lock.

The asyncio.Queue Alternative

A semaphore bounds a fixed set of tasks. A queue with a fixed number of long-lived workers bounds the tasks themselves, which matters when the crawl discovers new URLs as it goes — following pagination, expanding a sitemap, or crawling links out of pages it has already fetched.

import asyncio
import httpx

async def worker(
    name: str, queue: asyncio.Queue[str], client: httpx.AsyncClient, out: list[tuple[str, int]]
) -> None:
    while True:
        url = await queue.get()
        try:
            response = await client.get(url)
            out.append((url, response.status_code))
        except httpx.HTTPError as exc:
            out.append((url, -1))
            print(f"{name} failed on {url}: {exc!r}")
        finally:
            await asyncio.sleep(0.3)
            queue.task_done()

async def run(urls: list[str], workers: int = 8) -> list[tuple[str, int]]:
    queue: asyncio.Queue[str] = asyncio.Queue(maxsize=1000)
    out: list[tuple[str, int]] = []
    async with httpx.AsyncClient(headers=HEADERS, timeout=20.0) as client:
        pool = [
            asyncio.create_task(worker(f"w{i}", queue, client, out))
            for i in range(workers)
        ]
        for url in urls:
            await queue.put(url)          # blocks once 1000 are queued
        await queue.join()                # wait for the backlog to drain
        for task in pool:
            task.cancel()
        await asyncio.gather(*pool, return_exceptions=True)
    return out

pages = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 31)]
print(len(asyncio.run(run(pages))))

The maxsize=1000 is the real difference. With a semaphore, the full URL list must exist in memory as tasks before anything is bounded; with a bounded queue, await queue.put(url) blocks the producer once the backlog is full, so back-pressure reaches all the way up to whatever is generating URLs. That property is what makes the queue shape the right one for a crawl whose frontier is not known in advance, and it is the same reasoning that leads to an external broker in Distributed Crawling with Celery and Redis once the frontier outgrows one process.

The cost is that errors are now the worker's problem. A worker that raises escapes its while True loop and silently stops consuming, and with all workers dead queue.join() never returns. The try/finally above exists specifically to guarantee task_done() runs, and catching httpx.HTTPError rather than bare Exception keeps genuine bugs loud.

Edge Cases and Caveats

• A semaphore is bound to one event loop. Creating it at module import time, before asyncio.run starts a loop, works on Python 3.10+ but ties it to the first loop that touches it. Create semaphores inside the coroutine that uses them, and never share one across two asyncio.run calls.
• Permits are not fair in the way you might assume. asyncio.Semaphore wakes waiters in FIFO order, so a slow host cannot starve a fast one within a single gate — but a single gate shared across hosts still means one slow host occupies permits that a fast host could use. That is the case for splitting the gate per host rather than raising the count.
• Redirects consume permits invisibly. With follow_redirects=True, one client.get may perform four round trips while holding one permit. Against a site that redirects heavily, real request rate is a multiple of what the permit count suggests.
• Proxies change the arithmetic. Routing through a rotating pool means the target sees requests spread across many IPs, so its per-IP limit no longer bounds you. The descriptor and pool ceilings still do, and each proxy hop adds latency that lowers throughput per permit. See Rotating Proxies and Managing IP Blocks.
• return_exceptions=True hides failures unless you inspect them. It prevents one error from cancelling the batch, but the exceptions arrive as ordinary list elements. Filter them and count them, or a crawl that returns 10,000 results of which 9,000 are ConnectTimeout looks identical to a successful one.
• Windows caps the default selector loop. The ProactorEventLoop used by default on Windows does not have the 512-descriptor select() limit of the older selector loop, but running under an explicit SelectorEventLoop reintroduces it, and it fails well below the ulimit numbers quoted above.

Frequently Asked Questions

What concurrency number should I start with? Five to ten permits per hostname is a safe opening position for a site you do not control, with a delay inside the gate long enough to keep the observed rate under a few requests per second. Raise it only after a full run produces no 429 or 503 responses, and treat any increase as something to re-verify rather than a permanent setting.

Do I need a semaphore if I already set max_connections in HTTPX? In a single-host crawl the pool limit alone will bound your sockets, so it is not strictly required. It is still worth having, because the semaphore is where the politeness delay lives, it survives a switch to HTTP/2 where connections no longer map to requests, and it makes the intended bound explicit at the call site instead of buried in client construction.

Why is my crawl slower than the permit count suggests? Usually because something CPU-bound is inside the guarded block. Parsing, JSON decoding, and writing to disk all hold the permit while doing no network work. Move them outside the async with, and if parsing dominates, hand it to a process pool so the event loop is not blocked at all.

Can one semaphore cover several AsyncClient instances? Yes, as long as they all run on the same event loop. The semaphore counts permits, not connections, so it will happily bound work spread across two clients — for example one client routed through a proxy and one direct. What it cannot do is bound work in another process, which needs a shared counter in Redis or a broker instead.

Related

• Asynchronous Scraping with asyncio and HTTPX — the parent topic covering the event loop, clients, and when async is the wrong tool.
• Retrying Failed Requests with Tenacity — what to do with the 429s a too-generous gate produces.
• httpx vs aiohttp Async Performance — measured throughput differences between the two async clients.
• Web Scraping with Scrapy — a framework that applies the same bounds through settings rather than code.