Best Free and Paid Proxy Providers for Scraping

Choosing a proxy pool is the first infrastructure decision that shows up in both your success rate and your invoice, and it sits at the centre of Rotating Proxies and Managing IP Blocks.

Proxy options trade cost against stealth — residential paid resists blocks best.

The decision is not "which provider is best" but "which class of IP does this target require, and what does that class cost per gigabyte". Free lists are useful for exactly one thing — confirming that your proxy plumbing works — and should never carry production traffic or credentials. Paid datacenter pools are the right default for permissive targets and cost cents per gigabyte. Residential pools cost several dollars per gigabyte and are worth it only once a target is scoring the ASN your traffic comes from. Mobile pools cost more again and are a niche answer to a niche problem. Provider brand matters far less than pool class, billing model, and how the provider handles a session that goes bad.

Whatever you buy, the terms of service of the site you are fetching still apply. A proxy changes where a request appears to come from; it does not grant access you were not already entitled to, and it does not exempt you from a published rate limit.

Starting From the Constraint That Binds

Most bad proxy purchases start from a price list. Start from the failure you are actually seeing instead.

Proxy shopping goes wrong when the pool is chosen by price alone. Start from the constraint that is actually blocking the crawl and the class follows.

If the target serves you fine and you simply need throughput, you are buying bandwidth, and a datacenter pool wins on every axis that matters. If you are getting 403s or challenge pages from a clean, correctly-headed client, the problem is that your ASN is known to belong to a cloud provider, and no amount of extra datacenter IPs fixes a category judgement — that is when residential becomes necessary. If you are working with an endpoint that only a mobile app calls, carrier IPs may be the only ones the backend accepts. The distinction between the first two cases is worked through in detail in Residential vs Datacenter Proxies.

One more axis is easy to overlook: session control. Some pools give you a new exit IP on every connection; others let you request a sticky session that holds one exit for a bounded period. Anything involving a login, a cart, or a multi-step flow needs sticky sessions, because a session cookie that arrives from three countries in ninety seconds is a stronger signal than any header you could send.

What Free Proxy Lists Actually Cost

Free lists are not free; the price is paid in reliability and in exposure.

A publicly listed open proxy has an unknown operator, and by definition it terminates your TLS connection at the CONNECT layer or, worse, is a transparent HTTP proxy that can read and rewrite plaintext. Anything you send through one — credentials, API keys, session cookies — should be considered disclosed. That alone rules them out for authenticated scraping.

The reliability picture is equally poor and easy to measure. Take any free list and check it: a large fraction of entries will fail to connect at all, a further chunk will connect and then time out, and many of the survivors will be transparent rather than anonymous — passing your real IP in X-Forwarded-For or Via, which defeats the entire purpose. The verification script below tells you which is which in a few seconds per endpoint.

Free endpoints also tend to be heavily used by everyone else who found the same list, so their reputation with any major WAF is already spent. An IP that a thousand people are scraping through is not anonymous; it is pre-flagged.

Checking a Proxy Before You Trust It

Run this against any endpoint — free or paid — before routing real traffic through it. It measures latency, confirms the exit IP, and detects the leak headers that mark a transparent proxy.

import time

import requests

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
    ),
    "Accept": "application/json",
    "Accept-Language": "en-US,en;q=0.9",
}

LEAK_HEADERS = ("X-Forwarded-For", "Via", "X-Real-Ip", "Forwarded", "Client-Ip")


def check_proxy(proxy_url: str, timeout: float = 10.0) -> dict[str, object]:
    """Return latency, exit IP and any headers that leak the origin address."""
    proxies = {"http": proxy_url, "https": proxy_url}
    result: dict[str, object] = {"proxy": proxy_url.split("@")[-1], "ok": False}
    started = time.perf_counter()
    try:
        response = requests.get(
            "https://httpbin.org/anything",
            headers=HEADERS,
            proxies=proxies,
            timeout=timeout,
        )
        response.raise_for_status()
    except requests.exceptions.ProxyError as exc:
        result["error"] = f"proxy refused: {exc.__class__.__name__}"
        return result
    except requests.exceptions.SSLError:
        result["error"] = "TLS interception or broken CONNECT tunnel"
        return result
    except requests.exceptions.Timeout:
        result["error"] = "timed out"
        return result

    payload = response.json()
    seen = {k.lower() for k in payload["headers"]}
    result.update(
        ok=True,
        latency_s=round(time.perf_counter() - started, 3),
        exit_ip=payload["origin"],
        leaks=[h for h in LEAK_HEADERS if h.lower() in seen],
    )
    return result


if __name__ == "__main__":
    for entry in ("http://user:pass@proxy.example.com:8080",):
        print(check_proxy(entry))

An endpoint with leaks non-empty is transparent and useless for anything reputation-sensitive. An SSLError on an HTTPS target usually means the proxy is intercepting TLS rather than tunnelling it, which is both a privacy problem and an immediate fingerprint mismatch, since the certificate chain and handshake will not look like the site's.

The Cost Model Nobody Reads Until Month Two

Almost every residential and mobile provider bills per gigabyte of transfer, not per request. Datacenter pools are more often billed per IP per month or per port. That difference dominates budgeting.

Almost every paid pool bills per gigabyte, so the bill is set by bytes transferred rather than pages fetched. Blocking images and fonts moves this chart more than changing provider does.

The consequence is that your bill is set by bytes, and most of the bytes on a typical page are not the data you want. Blocking images, fonts, media and analytics on a browser-based crawl routinely cuts transfer by half or more, and on a metered residential plan that is a direct halving of the invoice. On an HTTP-client crawl, sending Accept-Encoding: gzip, deflate, br (and actually having Brotli installed) does the same job.

Two billing details are worth checking explicitly in a contract before you sign:

Is failed traffic billed? Many providers meter bytes transferred regardless of status code, so a run that returns 40% 403s still costs you for those responses.
What is the concurrency cap? Pools are usually sold with a maximum number of simultaneous connections. Exceeding it typically returns 407 or a 502 from the gateway rather than queueing, so your client-side concurrency limit must be set below the contracted number — the semaphore pattern in Limiting Concurrency with Semaphores is the clean way to enforce it.

The Errors You Will See, and What They Mean

Proxy failures produce a small, well-defined set of signatures, and knowing them saves hours of guessing.

407 Proxy Authentication Required means the credentials never reached the proxy. In requests this is almost always because the username or password contains a character that needs percent-encoding in the URL — an @ or a : in a password will silently split the URL in the wrong place.

requests.exceptions.ProxyError wrapping NewConnectionError means the endpoint is dead or firewalled. Retry against a different endpoint rather than the same one; a backoff on a dead host just wastes wall-clock time.

502 Bad Gateway or 522 from the proxy usually means the exit node dropped mid-request, which is common on residential pools where the exit is somebody's home connection. This is a retryable error, and it is the case that most justifies the retry discipline in Retrying Failed Requests with Tenacity.

403 from the target with a working proxy means the IP class is wrong for that site, not that the proxy is broken. Rotating harder within the same class will not help; changing class or reducing rate will.

SSLError: certificate verify failed on an HTTPS target through an HTTP proxy means the proxy is not tunnelling with CONNECT. Do not solve it with verify=False — that hides an interception you would rather know about.

A quieter class of failure is worth building a check for: the proxy that returns 200 with the wrong content. Some gateways serve a captive-portal page, an authentication notice, or a cached error body with a success status, and raise_for_status() will pass all of them straight through to your parser. The symptom is a parser that suddenly extracts zero rows without raising. Assert on something structural in the response — an expected element, a minimum body length, a content type — before treating a 200 as data, and count those assertion failures separately from HTTP errors so the two causes stay distinguishable in your metrics.

Provider dashboards are also worth reconciling against your own numbers rather than trusting outright. A dashboard reporting a 97% success rate is usually reporting successful proxy transactions, which includes every 403 the target returned. The metric you care about is pages parsed per gigabyte billed, and only your side can compute it.

Edge Cases and Caveats

Geolocation must match the story you tell. An exit IP in São Paulo with Accept-Language: en-US and a UTC timezone is incoherent. Match locale and timezone to the pool region, alongside the header discipline in How to Rotate User Agents in Python.
A proxy does not change your TLS fingerprint. The handshake is generated by your client, not the proxy, so a Python client behind a residential IP still presents an OpenSSL fingerprint — see TLS and JA3 Fingerprint Evasion.
Rotation frequency can itself be a signal. An IP that changes on literally every request while carrying one session cookie is less plausible than a stable IP. Rotate at session boundaries.
Provider APIs go down too. If you fetch fresh endpoints from a provider REST API at the start of each batch, cache the last good list; otherwise a provider outage becomes a total outage.
Credentials belong in the environment. Proxy URLs with embedded passwords end up in logs, tracebacks and error reporting. Read them from environment variables and redact them before logging — the proxy_url.split("@")[-1] in the script above is the minimum version of that.
Some pools resell each other. Two "different" providers can front the same underlying network, so a second vendor is not automatically a second pool. Test with the exit-IP check before assuming diversity.

Frequently Asked Questions

Are free proxy lists ever appropriate? Only for testing that your own proxy configuration works, and only against a target you control or an endpoint like httpbin.org. The operator is unknown, transparent proxies leak your real IP in forwarding headers, and any credentials you send through one should be treated as disclosed.

How do I know whether I need residential instead of datacenter IPs? Run the same request from a datacenter proxy and from a normal residential connection with identical headers. If the residential attempt succeeds where the datacenter one is challenged, the target is scoring the ASN and you need a residential pool. If both fail, the problem is elsewhere in your client and a more expensive pool will not fix it.

Why does my proxy work in the browser but fail in Python? Usually authentication encoding or protocol handling. Credentials containing reserved characters must be percent-encoded inside the proxy URL, and an HTTPS target through an HTTP proxy requires a CONNECT tunnel — if the proxy does not support it you will see a certificate error rather than a connection error.

What is the biggest lever on proxy cost? Bytes, not requests. Metered pools bill per gigabyte, so blocking images, fonts and third-party scripts on browser-based crawls, and requesting compressed responses on HTTP-client crawls, cuts the bill roughly in proportion to the transfer you avoid.

Rotating Proxies and Managing IP Blocks — the parent topic covering rotation strategy end to end.
Residential vs Datacenter Proxies — how the two classes differ in practice.
Bypassing Cloudflare and Akamai Protections — what happens when the IP is only part of the score.
Asynchronous Scraping with Asyncio and HTTPX — running a pool at concurrency without exceeding its cap.