Rotating Proxies and Managing IP Blocks

Proxy rotation is the network-layer half of Advanced Scraping Techniques and Anti-Bot Evasion: it spreads your traffic across many exit addresses so that no single IP accumulates enough request volume to trip a rate limit. This guide covers how to pick a rotation policy, build a pool that measures its own health, detect a block reliably rather than guessing, and return a cooled-down endpoint to service. Rotation is a way to stay inside a site's limits at scale, not a way around a refusal — if a site has told you not to crawl it, more IP addresses do not change that answer.

When to Use Proxy Rotation

Rotation solves exactly one problem: per-IP accounting. It does nothing for detection that keys on your TLS handshake, your headers, or your browser environment. Match the symptom to the layer before buying bandwidth.

The choice between residential and datacenter exits is a cost-versus-reputation trade, not a quality ranking. Datacenter IPs are fast, cheap and stable, but they arrive in contiguous, publicly documented ranges that any edge provider can classify in a single lookup. Residential exits borrow the reputation of consumer ISP allocations and are typically ten to forty times more expensive per gigabyte, with higher and much more variable latency. Residential vs Datacenter Proxies works through the numbers, and Best Free and Paid Proxy Providers for Scraping covers what to check before signing a contract.

The rotation policy follows from the shape of the request, not from the size of the pool.

Session-bound work — a login, a multi-step form, a cart — must keep the same exit for the duration, because switching IP mid-session invalidates most session cookies and is itself a strong bot signal. Stateless fetches should rotate as widely as the pool allows. Most real crawls contain both, so a usable pool exposes two acquisition modes rather than one.

Prerequisites

Python 3.10 or newer, plus a source of proxy endpoints. The code below uses requests for the synchronous examples and redis for shared state across workers.

pip install "requests[socks]>=2.32" "redis>=5.0" "tenacity>=8.3"

The [socks] extra pulls in PySocks, which requests needs for socks5:// URLs. Keep credentials out of source control:

export PROXY_USER="your-username"
export PROXY_PASS="your-password"

Confirm an endpoint works before wiring it into anything:

curl -s -x "http://$PROXY_USER:$PROXY_PASS@gateway.example.com:8000" https://httpbin.org/ip

If that returns your own address rather than the exit's, the proxy is not being applied — usually a scheme mismatch between http:// and https:// entries in the proxies mapping.

Step-by-Step: Building a Rotating Proxy Layer

1. Model an endpoint with its own health record

A bare list of URL strings cannot express "this one has failed twice in the last minute". Give each endpoint a small record so rotation decisions have something to read.

from dataclasses import dataclass, field
from time import monotonic

@dataclass
class ProxyEndpoint:
    """One exit address plus the health data rotation decisions depend on."""
    url: str
    kind: str = "datacenter"
    successes: int = 0
    failures: int = 0
    consecutive_blocks: int = 0
    cooldown_until: float = 0.0
    latencies: list[float] = field(default_factory=list)

    @property
    def available(self) -> bool:
        """True when the endpoint is not serving a cooldown."""
        return monotonic() >= self.cooldown_until

    @property
    def success_rate(self) -> float:
        """Share of attempts that returned usable content."""
        total = self.successes + self.failures
        return 1.0 if total == 0 else self.successes / total

    @property
    def median_latency(self) -> float:
        """Median of the last observations, or 0.0 before any data."""
        if not self.latencies:
            return 0.0
        ordered = sorted(self.latencies)
        return ordered[len(ordered) // 2]

    def record(self, ok: bool, seconds: float) -> None:
        """Update counters after one attempt."""
        if ok:
            self.successes += 1
            self.consecutive_blocks = 0
        else:
            self.failures += 1
        self.latencies = (self.latencies + [seconds])[-20:]

Capping latencies at twenty samples keeps memory flat over a long run while still letting a degrading endpoint show up quickly in the median.

2. Choose an endpoint by weight, not by turn

Strict round-robin sends exactly as much traffic to your slowest, least reliable exit as to your best one. Weighting by observed success rate and latency fixes that without the starvation a pure "always pick the best" strategy causes.

import random
from time import monotonic

class ProxyPool:
    """A weighted pool of exits with cooldown support."""

    def __init__(self, endpoints: list[ProxyEndpoint]) -> None:
        self._endpoints = endpoints
        self._sticky: dict[str, ProxyEndpoint] = {}

    def acquire(self) -> ProxyEndpoint:
        """Return a healthy endpoint chosen by weighted random selection."""
        live = [e for e in self._endpoints if e.available]
        if not live:
            soonest = min(self._endpoints, key=lambda e: e.cooldown_until)
            raise RuntimeError(
                f"every exit is cooling down; next free in "
                f"{soonest.cooldown_until - monotonic():.0f}s"
            )
        weights = [
            max(0.05, e.success_rate) / (1.0 + e.median_latency)
            for e in live
        ]
        return random.choices(live, weights=weights, k=1)[0]

    def acquire_sticky(self, session_key: str) -> ProxyEndpoint:
        """Return the endpoint bound to `session_key`, creating the binding once."""
        bound = self._sticky.get(session_key)
        if bound is None or not bound.available:
            bound = self.acquire()
            self._sticky[session_key] = bound
        return bound

    def penalise(self, endpoint: ProxyEndpoint, base_seconds: float = 1800.0) -> None:
        """Cool an endpoint down, doubling the wait for repeat offences."""
        endpoint.consecutive_blocks += 1
        factor = 2 ** (endpoint.consecutive_blocks - 1)
        endpoint.cooldown_until = monotonic() + base_seconds * min(factor, 8)

The floor of 0.05 on the weight is deliberate: an endpoint that has failed recently still gets a small share of traffic, so it can recover its rating instead of being permanently exiled by one bad minute.

3. Detect a block properly, not by status code alone

Status codes tell you less than they should. Plenty of protected sites return 200 with a challenge page, and plenty of healthy sites return 404 legitimately. Check the status, the headers and a cheap content signal together.

import requests

BLOCK_STATUSES = {403, 407, 429, 503}
CHALLENGE_MARKERS = (
    "just a moment",
    "checking your browser",
    "verify you are human",
    "access denied",
)

def looks_blocked(response: requests.Response) -> bool:
    """True when a response is a block or a challenge rather than content."""
    if response.status_code in BLOCK_STATUSES:
        return True
    if response.status_code >= 500:
        return True
    body = response.text[:4000].lower()
    if any(marker in body for marker in CHALLENGE_MARKERS):
        return True
    # A content page that suddenly collapses to a few hundred bytes is suspect.
    return response.status_code == 200 and len(response.content) < 512

The last check catches the quiet failure mode that costs the most: an edge that returns 200 with an empty shell, so your parser silently writes thousands of empty rows. Assert on a field you always expect and treat its absence as a block.

4. Wire the pool into a request function

Put rotation, block detection, penalties and backoff in one place so no call site can forget a step.

import random
import time
import requests

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Connection": "keep-alive",
}

def fetch(pool: ProxyPool, url: str, attempts: int = 4) -> requests.Response:
    """Fetch a URL through the pool, rotating and backing off on blocks."""
    last_error: Exception | None = None
    for attempt in range(attempts):
        endpoint = pool.acquire()
        started = time.monotonic()
        try:
            response = requests.get(
                url,
                headers=HEADERS,
                proxies={"http": endpoint.url, "https": endpoint.url},
                timeout=(5, 20),
            )
        except requests.RequestException as exc:
            endpoint.record(False, time.monotonic() - started)
            pool.penalise(endpoint, base_seconds=120.0)
            last_error = exc
        else:
            elapsed = time.monotonic() - started
            if looks_blocked(response):
                endpoint.record(False, elapsed)
                pool.penalise(endpoint)
                last_error = requests.HTTPError(
                    f"blocked with {response.status_code} via {endpoint.url}"
                )
            else:
                endpoint.record(True, elapsed)
                return response
        time.sleep(min(2 ** attempt + random.uniform(0, 1.5), 30))
    raise RuntimeError(f"gave up on {url} after {attempts} attempts") from last_error

Note the two-part timeout: (5, 20) means five seconds to establish the connection and twenty to read the body. A single scalar timeout lets a proxy that accepts connections but never responds hold a worker for the full duration. Backoff uses full jitter, because a fleet of workers that all back off by exactly four seconds simply re-synchronises and hits the origin in a second thundering herd.

5. Probe recovered endpoints before trusting them

A cooldown that expires is not evidence of health. Send one cheap request first, and only return the endpoint to the pool if it passes.

import requests

def probe(endpoint: ProxyEndpoint, timeout: float = 10.0) -> bool:
    """Send one cheap request to check an endpoint before returning it to the pool."""
    try:
        response = requests.get(
            "https://httpbin.org/ip",
            headers={"User-Agent": HEADERS["User-Agent"], "Accept": "application/json"},
            proxies={"http": endpoint.url, "https": endpoint.url},
            timeout=timeout,
        )
        response.raise_for_status()
        return "origin" in response.json()
    except (requests.RequestException, ValueError):
        return False

def rehabilitate(pool: ProxyPool, endpoint: ProxyEndpoint, max_failures: int = 3) -> bool:
    """Probe a cooled-down endpoint; retire it after repeated failures."""
    if probe(endpoint):
        endpoint.consecutive_blocks = 0
        endpoint.cooldown_until = 0.0
        return True
    if endpoint.consecutive_blocks >= max_failures:
        pool._endpoints.remove(endpoint)
    else:
        pool.penalise(endpoint)
    return False

Probe against a neutral echo service rather than the target. Sending a probe to the site that just blocked you turns a recoverable cooldown into a permanent one.

6. Share pool state across workers

Per-process state means eight workers each discover the same dead endpoint independently, at the cost of eight blocked requests. A Redis-backed cooldown set makes one worker's discovery immediately visible to all of them.

import redis

class SharedCooldown:
    """Cooldown registry shared by every worker through Redis."""

    def __init__(self, client: redis.Redis, prefix: str = "proxy:cool:") -> None:
        self._client = client
        self._prefix = prefix

    def mark(self, proxy_url: str, seconds: int) -> None:
        """Mark an endpoint as cooling down for `seconds`."""
        self._client.setex(f"{self._prefix}{proxy_url}", seconds, "1")

    def is_cooling(self, proxy_url: str) -> bool:
        """True when another worker has already sidelined this endpoint."""
        return self._client.exists(f"{self._prefix}{proxy_url}") == 1

    def available(self, proxy_urls: list[str]) -> list[str]:
        """Filter a candidate list down to endpoints nobody has sidelined."""
        pipe = self._client.pipeline()
        for url in proxy_urls:
            pipe.exists(f"{self._prefix}{url}")
        return [url for url, hit in zip(proxy_urls, pipe.execute()) if not hit]

Redis TTLs expire the entry automatically, so the cooldown needs no sweeper process. The same coordination primitive underpins distributed crawls generally, as described in Distributed Crawling with Celery and Redis.

7. Verify where each exit actually lands

Providers advertise country targeting; they do not always deliver it. An exit that claims Germany and resolves in Singapore produces two problems at once: the site serves the wrong regional content, and the geography contradicts whatever locale and timezone your client declares. Verify on acquisition rather than trusting the label.

import requests

GEO_URL = "https://ipinfo.io/json"

def locate(endpoint: ProxyEndpoint, timeout: float = 12.0) -> dict[str, str]:
    """Return the country, region and ASN an endpoint actually exits from."""
    response = requests.get(
        GEO_URL,
        headers={"User-Agent": HEADERS["User-Agent"], "Accept": "application/json"},
        proxies={"http": endpoint.url, "https": endpoint.url},
        timeout=timeout,
    )
    response.raise_for_status()
    payload = response.json()
    return {
        "ip": payload.get("ip", ""),
        "country": payload.get("country", ""),
        "region": payload.get("region", ""),
        "org": payload.get("org", ""),
    }

def verify_country(endpoint: ProxyEndpoint, expected: str) -> bool:
    """True when the endpoint exits in the country it was sold as."""
    try:
        return locate(endpoint)["country"].upper() == expected.upper()
    except (requests.RequestException, ValueError):
        return False

The org field is the one to record. It carries the ASN and the network name, which tells you whether a "residential" exit is really a consumer ISP allocation or a hosting provider being sold as one — a distinction the price does not always reflect. Cache the result per endpoint rather than probing on every acquisition; geolocation is stable for the life of an exit and the lookup itself consumes billed bandwidth.

Store the verified country alongside the endpoint and use it to pick a matching locale, Accept-Language and timezone for whatever client uses it. An exit in Tokyo paired with Accept-Language: en-US and a New York timezone is a contradiction any correlation rule can read, and it undoes the work the rotation was supposed to do.

Performance and Scaling Considerations

Pool size follows from your target throughput and the per-IP budget you are willing to spend. If a site tolerates roughly one request per IP every ten seconds and you want sixty requests per second, you need at least six hundred concurrent exits — and you should treat that arithmetic as a ceiling, not a goal. Running comfortably below what a site tolerates keeps you off its incident dashboards.

Latency is the second budget. Datacenter exits typically add 20-60 ms, residential exits 150-800 ms with outliers beyond two seconds. Because request time is dominated by those outliers, concurrency rather than raw speed is what recovers throughput: sixty concurrent workers on 400 ms exits deliver more than fifteen workers on 100 ms exits. Keep connections alive with a requests.Session per endpoint so each request does not repay the TLS handshake, and cap pool_maxsize on the adapter to match your worker count.

Bandwidth is the third, and it is the one that surprises people. Residential traffic is usually billed per gigabyte, so blocking images and stylesheets in browser workloads and requesting gzip compression in HTTP workloads directly reduces the invoice. Where a JSON endpoint exists, a 4 KB API response replaces a 900 KB rendered page — a two-hundred-fold saving that no amount of proxy tuning can match. Caching has the same effect from the other direction; Caching and Incremental Crawling covers not re-fetching what you already hold.

One cost that is easy to miss is DNS. Every new connection through a fresh exit repeats name resolution, and some gateways resolve on your behalf while others do not. Use socks5h:// where you want the proxy to resolve, keep a local resolver cache for the hostnames you visit repeatedly, and check whether your provider bills DNS traffic separately.

Finally, instrument the pool. Track success rate, median latency and cooldown count per endpoint, and alert when the pool-wide success rate drops below a threshold rather than when individual requests fail. A gradual slide from 95% to 60% over an hour is the signature of a site tightening its limits, and it is invisible if you only look at exceptions. Monitoring and Alerting for Scrapers covers the metrics plumbing.

Common Errors and Fixes

requests.exceptions.ProxyError: HTTPSConnectionPool(...): Max retries exceeded ... 407 Proxy Authentication Required. Credentials were not sent or were mangled. Percent-encode special characters in the password — @, : and / all break URL parsing — with urllib.parse.quote(password, safe="") before building the proxy URL.

requests.exceptions.InvalidSchema: Missing dependencies for SOCKS support. You passed a socks5:// URL without PySocks. Install requests[socks]. Prefer socks5h:// over socks5:// so DNS resolution happens at the proxy; resolving locally leaks the hostnames you are visiting to your own resolver and can return the wrong regional IP.

Every request returns your real IP. The proxies mapping keys must match the scheme of the target URL. {"http": proxy} alone does nothing for an https:// request. Set both keys, and verify against https://httpbin.org/ip rather than assuming.

requests.exceptions.SSLError: certificate verify failed through the proxy. A man-in-the-middle proxy is presenting its own certificate. Add its CA to the trust store with verify="/path/to/ca.pem". Do not reach for verify=False: it disables authentication of the endpoint entirely, and any intermediary can then read and rewrite your traffic.

429 Too Many Requests continues after rotation. The limit is not keyed on IP. It may be keyed on an account, an API key, a session cookie or a device fingerprint. Check whether a Retry-After header is present and honour it, and confirm you are not carrying the same session cookie across every "different" identity.

Latency collapses to timeouts under load. You are exhausting the provider's concurrent-session allowance, and the gateway is queuing you. Read the plan's concurrency limit, set your semaphore below it, and treat ConnectTimeout as a signal to reduce parallelism rather than to retry harder.

A block persists after the cooldown expires. The site is blocking a whole subnet, not one address. Group endpoints by their /24 and cool down the group, or switch that batch to a different provider region.

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected) on the first request only. The gateway is rotating you to a fresh exit and dropping the connection while it does. Retry once on the same endpoint before penalising it; treating a rotation artefact as a block wastes healthy capacity.

Bandwidth spend far exceeds the number of pages fetched. Something is downloading assets you never parse — usually a browser worker without route interception, or a client that follows redirects into large media files. Log response sizes per request and find the top percentile before buying more bandwidth.

Frequently Asked Questions

How large does my proxy pool need to be? Divide your target request rate by the per-IP rate you intend to respect. At one request per IP every ten seconds and a target of ten requests per second, you need about a hundred concurrent exits. Add roughly 20% headroom for endpoints sitting in cooldown at any moment, and revisit the number whenever the target's limits change.

When should I use sticky sessions instead of rotating every request? Whenever the request depends on state the server associates with your address: an authenticated session, a cart, a multi-page form, or any flow whose cookies were issued to a specific IP. Bind a session key to one exit for the whole flow, and rotate only between flows.

Why do free proxy lists fail almost immediately? Public endpoints are shared by thousands of users, so the addresses are already on every reputation list before you use them, and many intercept or modify traffic. They are useful only for testing that your rotation code works, never for collecting data you intend to rely on.

Should I rotate the User-Agent along with the IP? Rotate the whole identity together or not at all. A new IP with the same header set is still recognisably the same client, and a new User-Agent on the same IP is a contradiction. Bind a coherent header profile to each exit, as described in How to Rotate User Agents in Python.

Does rotation help against Cloudflare or Akamai? Only for the rate-limiting component. Those systems score the TLS handshake, header order and browser environment as well, so a fresh IP with a stock Python fingerprint is challenged just as quickly as a stale one. Fix the fingerprint first, then use rotation to stay within the volume the site permits.

Related

• Advanced Scraping Techniques and Anti-Bot Evasion — the parent section for this topic
• Residential vs Datacenter Proxies — cost, latency and reputation compared
• How to Rotate User Agents in Python — keeping headers coherent with each exit
• Best Free and Paid Proxy Providers for Scraping — what to check before you sign
• Using Playwright for Modern Web Automation — attaching a per-context proxy to a browser