How to Rotate User-Agents in Python

Rotating the User-Agent header is the first thing most people try to look less like a bot, and this guide — part of Rotating Proxies and Managing IP Blocks — shows how to do it properly and why it is only a small part of the picture.

To rotate User-Agents in Python, build a small pool of current, real browser strings and pick one per request (or per session) with random.choice, sending it as an explicit header in requests or httpx. The important nuance is that a User-Agent never travels alone: real browsers send a consistent set of companion headers, so you must pair each UA with matching Accept, Accept-Language, and — for Chrome — Sec-CH-UA client-hint headers. On its own, UA rotation is weak, because modern anti-bot systems fingerprint your TLS handshake and header order, not just the UA text.

Building a Realistic User-Agent Pool

The most common mistake is inventing User-Agent strings or copying stale ones. Anti-bot vendors maintain databases of real UA strings and their expected companion headers; a UA claiming to be Chrome 40 (years out of date) or one with impossible version combinations is an instant tell. Collect a handful of genuine, current strings from real browsers on real platforms, and refresh them periodically as browsers update. Keep the pool small and plausible rather than large and random — a hundred exotic UAs look stranger than five common ones.

Rotate at the right granularity, too. Switching UA on every single request within one session is itself suspicious: a real user does not change browsers mid-visit. Assign a UA per session or per logical identity, so a sequence of requests that shares cookies also shares a consistent UA. This ties into the session handling covered in Managing Cookies and Sessions, where the browser identity should stay stable across a login flow.

Rotating User-Agents in requests and httpx

The mechanics are simple. Store your pool alongside the companion headers each UA implies, choose one, and send the whole coherent set. The helper below returns a full header dict — not just a UA — so Accept-Language and client hints line up with the browser you are claiming to be. This coherence is what separates convincing rotation from a header that contradicts itself.

import random
import requests

UA_PROFILES: list[dict[str, str]] = [
    {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Sec-CH-UA": '"Chromium";v="125", "Not.A/Brand";v="24", "Google Chrome";v="125"',
        "Sec-CH-UA-Platform": '"Windows"',
    },
    {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
    },
    {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.5",
    },
]


def pick_profile() -> dict[str, str]:
    return random.choice(UA_PROFILES)


def fetch(url: str) -> requests.Response:
    headers = pick_profile()
    resp = requests.get(url, headers=headers, timeout=15)
    resp.raise_for_status()
    return resp


if __name__ == "__main__":
    page = fetch("https://httpbin.org/headers")
    print(page.json()["headers"]["User-Agent"])

The same idea works with httpx, and here rotation is per session: each Client keeps one identity for its lifetime, which is more realistic than reshuffling on every call. If you scrape asynchronously, the same principle applies to each AsyncClient in your pool, following the concurrency patterns in Asynchronous Scraping with asyncio and httpx.

import random
import httpx

UA_PROFILES: list[dict[str, str]] = [
    {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9",
        "Sec-CH-UA-Platform": '"Windows"',
    },
    {
        "User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 17_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Mobile/15E148 Safari/604.1",
        "Accept-Language": "en-US,en;q=0.9",
    },
]


def make_session() -> httpx.Client:
    profile = random.choice(UA_PROFILES)
    base_headers = {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        **profile,
    }
    return httpx.Client(headers=base_headers, http2=True, timeout=15.0)


def fetch(client: httpx.Client, url: str) -> httpx.Response:
    resp = client.get(url)
    resp.raise_for_status()
    return resp


if __name__ == "__main__":
    with make_session() as client:
        r1 = fetch(client, "https://httpbin.org/headers")
        r2 = fetch(client, "https://httpbin.org/user-agent")
        print(r1.json()["headers"]["User-Agent"])
        print(r2.json()["user-agent"])

Why User-Agent Rotation Alone Is Weak

Here is the uncomfortable truth: for any serious anti-bot system, rotating the UA changes almost nothing. Two deeper signals give you away regardless of the header. The first is your TLS fingerprint — the exact cipher suites, extensions, and ordering your HTTP library sends during the handshake. A Python client claiming to be Chrome but presenting an OpenSSL/urllib3 TLS signature is trivially caught; defeating that requires the impersonation techniques in TLS and JA3 Fingerprint Evasion. The second is IP reputation: a thousand requests from one datacenter IP look automated no matter how the UA varies, which is why UA rotation only becomes meaningful alongside the proxy rotation covered across Advanced Scraping Techniques and Anti-Bot Evasion.

Treat UA rotation as basic hygiene, not a defense. It prevents the dumbest kind of blocking — a filter that rejects the literal python-requests/2.x default UA — but it will not fool anything that inspects the connection itself. Spend your effort where the real signals live: coherent headers, a browser-like TLS stack, and clean, rotating IPs.

Edge Cases and Caveats

• Stale strings are worse than none. A UA advertising a browser version that no longer exists flags you immediately. Refresh the pool as browsers update.
• Client hints must match the UA. If you claim Chrome, send consistent Sec-CH-UA values. A Chrome UA with Firefox-style or missing client hints is contradictory.
• Mobile UAs imply mobile behavior. Sending an iPhone UA to a desktop site can trigger different responses or extra checks; only use a mobile UA if you also accept the mobile experience.
• Do not rotate mid-session. Changing UA between requests that share cookies looks impossible for a real user. Rotate per session or per identity, not per request.
• The default UA is a red flag. Never ship the library default like python-httpx/0.x or python-requests/2.x; many sites block those outright.
• Header order and casing matter. Some detectors check the order of headers, not just their values. Libraries that let you preserve a browser-like ordering are more convincing than ones that reorder alphabetically.

Frequently Asked Questions

How do I rotate User-Agent strings in Python? Build a small list of real, current browser User-Agent strings, then select one per request or per session with random.choice and pass it in the headers argument of requests or httpx. Crucially, store companion headers (Accept, Accept-Language, client hints) alongside each UA and send them together so the identity is coherent.

Is rotating the User-Agent enough to avoid getting blocked? No. On its own it only defeats the crudest filters. Serious anti-bot systems fingerprint your TLS handshake and check IP reputation, neither of which the UA header changes. Effective evasion combines coherent headers with a browser-like TLS stack and rotating proxies.

Should I change the User-Agent on every request? Usually not within a single session. A real user keeps one browser for the whole visit, so switching UA between requests that share cookies looks unnatural. Assign a UA per session or per logical identity and keep it stable for that session's lifetime.

Where do I get realistic User-Agent strings? Copy them from real, up-to-date browsers on the platforms you want to mimic, or use a maintained library that tracks current strings. Avoid inventing them or scraping ancient lists — an implausible or outdated UA is a stronger bot signal than the library default.