Understanding HTTP Requests and Responses

Part of The Complete Guide to Python Web Scraping, this guide covers the protocol layer that sits under every extraction script you will ever write. The scope is the request you send and the response you get back: what the bytes on the wire contain, how servers decide whether you look like a browser, and how a scraper should branch on what comes back instead of assuming success.

Almost every scraping failure that looks mysterious is an HTTP fact you did not check. A parse returning zero rows is usually a 403 you never inspected. A script that "hangs forever" is a request without a timeout. Silently wrong data is often a 302 to a login page whose HTML parsed fine and contained nothing. Treating the response as an object with status, headers and encoding — rather than as a string of HTML — eliminates that whole class of bug.

HTTP is stateless by design: the server keeps no memory of your previous request unless you carry that memory yourself in a cookie, a header or a token. Everything that feels like continuity on the web — being logged in, a shopping basket, a paginated result set that remembers your filters — is state your client re-sends on every single round trip. A scraper that ignores this gets served the anonymous, logged-out, unfiltered version of every page and cannot tell the difference, because the response still arrives with a 200.

The unit of work is therefore a pair: a request that fully describes what you want and who you appear to be, and a response whose envelope — status line and headers — tells you how to interpret the body. Reading only the body is like reading a letter and throwing away the envelope, then wondering why you cannot tell whether it was addressed to you.

When to Use requests and When to Reach for Something Else

requests is the right default for the overwhelming majority of scraping. It is synchronous, has connection pooling, transparent decompression and cookie handling, and its API is stable. But it is not always the correct tool, and the failure mode of using it past its limits is slow crawls or unexplained blocks.

The decision to skip HTML entirely is worth making early. If the page you want is populated by an XHR call returning JSON, fetching that endpoint directly is faster, more stable and easier to parse than any amount of selector work — see Reverse-Engineering Private APIs.

Prerequisites

Python 3.10 or newer with a working virtual environment, set up as described in Setting Up Your Python Scraping Environment. The examples need an HTTP client and, from step 4 onwards, a parser.

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install "requests>=2.31" "beautifulsoup4>=4.12" "lxml>=5.1" "tenacity>=8.2"

No system dependencies are required; requests pulls in urllib3, certifi, idna and charset-normalizer automatically. Modern requests handles gzip and deflate out of the box and adds Brotli (br) when brotli or brotlicffi is installed — worth adding, because advertising br in Accept-Encoding and then failing to decode it is a giveaway that you are not the browser you claim to be.

Step-by-Step: Making a Request a Server Will Answer

1. Send headers that match a real client

A default requests call announces User-Agent: python-requests/2.32.3 and offers almost no Accept preferences. That combination is trivially detectable and is the single most common cause of a 403 on a first attempt. Send a coherent set of headers rather than only a User-Agent: real browsers always send Accept, Accept-Language and Accept-Encoding together, and a request with a Chrome User-Agent but no Accept-Language is more suspicious than one with no User-Agent at all.

import requests

BROWSER_HEADERS: dict[str, str] = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1",
}

response = requests.get(
    "https://books.toscrape.com/catalogue/page-2.html",
    headers=BROWSER_HEADERS,
    timeout=(5, 20),
)
print(response.status_code, response.headers.get("Content-Type"))

The timeout=(5, 20) tuple is a connect timeout and a read timeout. Without it, requests waits indefinitely — a socket that never sends data will hang a scheduled job until something external kills it. Rotating the User-Agent across a crawl adds a further layer; the mechanics are in How to Rotate User Agents in Python.

2. Branch on the status code before touching the body

response.text always returns something. On a 403 it returns the block page; on a 429 it returns a rate-limit notice; on a 500 it returns a stack trace. Parsing any of those produces zero rows and no error, which is how a broken crawl runs for hours looking healthy. Route on the class first.

import time
import requests


def fetch(session: requests.Session, url: str) -> str | None:
    """Return HTML only when the response is genuinely the requested resource."""
    response = session.get(url, timeout=(5, 20))
    code = response.status_code

    if code == 200:
        return response.text
    if code in (301, 302, 303, 307, 308):
        return None  # requests follows these already; landing here means allow_redirects=False
    if code == 429:
        wait = float(response.headers.get("Retry-After", "30"))
        time.sleep(min(wait, 300))
        return None
    if code in (403, 404, 410):
        return None  # retrying an unchanged request cannot help
    if 500 <= code < 600:
        return None  # transient; let the retry layer decide
    return None

3. Verify where you actually landed

requests follows redirects automatically for every method except HEAD. That is usually what you want, but it means a 200 does not prove you reached the URL you asked for. Sites redirect logged-out clients to a sign-in page, geo-redirect to a country subsite, and bounce to a consent interstitial — all returning 200 with parseable HTML that contains none of your data.

import requests

response = requests.get(
    "https://httpbin.org/redirect/2",
    headers={"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) Chrome/136.0.0.0 Safari/537.36"},
    timeout=(5, 20),
)

print("final url :", response.url)
print("hops      :", [(r.status_code, r.headers.get("Location")) for r in response.history])

if not response.url.startswith("https://httpbin.org/get"):
    raise RuntimeError(f"redirected away from the target: {response.url}")

response.history holds the chain of intermediate responses in order. Asserting on response.url after every fetch is a two-line check that catches an entire family of silent failures.

4. Decode the body with the right encoding

response.content is bytes; response.text is those bytes decoded. The decoding is a guess unless the server declared a charset. When Content-Type includes charset=, requests uses it. When it does not, requests falls back to charset-normalizer's detection, which is good but not infallible on short documents. HTML pages frequently declare their real encoding only in a <meta charset> tag, which the HTTP layer never sees.

import re
import requests

CHARSET_META = re.compile(rb'<meta[^>]+charset=["\']?\s*([\w-]+)', re.I)


def decode_html(response: requests.Response) -> str:
    """Prefer the HTTP charset, fall back to the meta tag, then to UTF-8."""
    declared = response.headers.get("Content-Type", "")
    if "charset=" in declared.lower():
        return response.content.decode(response.encoding or "utf-8", errors="replace")

    match = CHARSET_META.search(response.content[:2048])
    if match:
        return response.content.decode(match.group(1).decode("ascii"), errors="replace")
    return response.content.decode("utf-8", errors="replace")


page = requests.get(
    "https://books.toscrape.com/",
    headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/136.0.0.0 Safari/537.36"},
    timeout=(5, 20),
)
print(decode_html(page)[:120])

errors="replace" substitutes U+FFFD rather than raising, which keeps a crawl running and leaves a visible marker in the data. Deeper diagnosis of mojibake and double-encoding lives in Fixing Common Unicode Errors in Python Scraping.

5. Check the content type before choosing a parser

A Content-Type of application/json means response.json(); text/html means an HTML parser; application/pdf or image/jpeg means neither. Feeding binary into an HTML parser produces a tree of nonsense rather than an exception, so the check has to be explicit.

import json
import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/136.0.0.0 Safari/537.36"}


def extract(url: str) -> dict[str, object] | list[dict[str, str]]:
    response = requests.get(url, headers=HEADERS, timeout=(5, 20))
    response.raise_for_status()
    mime = response.headers.get("Content-Type", "").split(";")[0].strip().lower()

    if mime == "application/json":
        return response.json()
    if mime in ("text/html", "application/xhtml+xml"):
        soup = BeautifulSoup(response.text, "lxml")
        return [{"title": a.get("title", "")} for a in soup.select("h3 > a")]
    raise ValueError(f"unhandled content type: {mime!r}")


print(json.dumps(extract("https://httpbin.org/json"), indent=2)[:200])

Once the body is known-good HTML, the next stage is structural extraction, covered in Parsing HTML with BeautifulSoup and, for tabular pages, in Step-by-Step Guide to Extracting Tables from HTML.

6. Send parameters and bodies correctly

Hand-concatenating query strings breaks on the first value containing a space, an ampersand or a non-ASCII character. Pass params as a dict and let the library encode it. For POST, the distinction between data= and json= is not cosmetic: data= sends application/x-www-form-urlencoded, which is what an HTML form does, while json= sends application/json, which is what an API expects. Sending the wrong one is a reliable 400.

import requests

HEADERS = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) Chrome/136.0.0.0 Safari/537.36"}

form = requests.post(
    "https://httpbin.org/post",
    data={"q": "python web scraping", "page": "2"},
    headers=HEADERS,
    timeout=(5, 20),
)
api = requests.post(
    "https://httpbin.org/post",
    json={"query": "python web scraping", "page": 2},
    headers=HEADERS,
    timeout=(5, 20),
)

print(form.json()["form"])
print(api.json()["json"])

Form submission, hidden fields and CSRF tokens are their own subject, handled in Handling Forms and Authentication.

7. Retry only what is worth retrying

Retrying a 403 wastes requests and accelerates a ban. Retrying a 503 almost always succeeds. Encode that distinction once, with exponential backoff and jitter so a fleet of workers does not synchronise into a thundering herd.

import requests
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter


class TransientHTTPError(Exception):
    """Raised for statuses where waiting and repeating is reasonable."""


@retry(
    retry=retry_if_exception_type((TransientHTTPError, requests.exceptions.ConnectionError)),
    wait=wait_exponential_jitter(initial=1, max=60),
    stop=stop_after_attempt(5),
    reraise=True,
)
def get_with_retry(session: requests.Session, url: str) -> requests.Response:
    response = session.get(url, timeout=(5, 20))
    if response.status_code in (429, 500, 502, 503, 504):
        raise TransientHTTPError(f"{response.status_code} for {url}")
    response.raise_for_status()
    return response


with requests.Session() as s:
    s.headers.update({"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/136.0.0.0 Safari/537.36"})
    print(get_with_retry(s, "https://books.toscrape.com/").status_code)

The retry policy options and their trade-offs are expanded in Retrying Failed Requests with Tenacity.

8. Log the envelope, not the body

When a crawl misbehaves at three in the morning, the body is far too large to keep and far too noisy to read. What you need is the envelope for every request: final URL, status, content type, byte count, elapsed time and the handful of headers that reveal rate limiting or caching. That is four hundred bytes per request and it answers most post-mortems on its own.

import logging
import requests

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
log = logging.getLogger("fetch")

INTERESTING = ("Content-Type", "Content-Length", "Retry-After", "X-RateLimit-Remaining", "Age", "ETag")


def log_envelope(response: requests.Response) -> None:
    meta = {h: response.headers[h] for h in INTERESTING if h in response.headers}
    log.info(
        "%s %s -> %s in %.0f ms %s",
        response.request.method,
        response.url,
        response.status_code,
        response.elapsed.total_seconds() * 1000,
        meta,
    )


with requests.Session() as s:
    s.headers.update({"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) Chrome/136.0.0.0 Safari/537.36"})
    log_envelope(s.get("https://books.toscrape.com/", timeout=(5, 20)))

response.elapsed measures from sending the request to finishing the headers, so it isolates server latency from your own parsing time. Watching that number climb is the earliest signal that a host has started throttling you, usually well before the first 429 arrives. Turning these lines into queryable events is covered in Structured Logging for Python Scrapers.

Performance and Scaling Considerations

Handshakes dominate small responses. A cold HTTPS request costs a DNS lookup, a TCP handshake and a TLS handshake — typically 80–250 ms of round trips before the server even reads your request line. A 20 KB HTML page then transfers in a few milliseconds. Using a Session keeps the connection alive and amortises that setup to nothing, which on 1,000 sequential requests to one host is the difference between roughly four minutes and roughly forty seconds of pure overhead. This is covered in practical detail in Managing Cookies and Sessions.

The default pool is ten connections per host. requests mounts an HTTPAdapter with pool_connections=10, pool_maxsize=10. Run twenty threads against one domain and ten of them will silently queue, or emit Connection pool is full warnings and discard connections. Mount a wider adapter when you raise concurrency.

import requests
from requests.adapters import HTTPAdapter

session = requests.Session()
session.mount("https://", HTTPAdapter(pool_connections=32, pool_maxsize=32))

Synchronous throughput is bounded by latency, not bandwidth. At 300 ms per request, one thread manages roughly three pages per second regardless of how fast your connection is. Sixteen concurrent workers reach around 50 pages per second against a tolerant host — but that is often far more than the target will accept, and the correct limit is usually set by politeness rather than by hardware. A single global semaphore is the cleanest way to enforce it; see Limiting Concurrency with Semaphores.

Compression is worth ~70% of the bytes and costs almost nothing. HTML compresses extremely well; a 240 KB page typically arrives as 30–40 KB under gzip and slightly less under Brotli. Decompression costs single-digit milliseconds. Never disable it to "simplify" a debugging session — use response.content if you need the decoded bytes.

Stream large bodies instead of materialising them. requests.get(url) reads the whole body into memory. For a file of any size, pass stream=True and iterate, which caps memory at the chunk size rather than the file size.

import requests

with requests.get(
    "https://httpbin.org/bytes/1048576",
    headers={"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) Chrome/136.0.0.0 Safari/537.36"},
    stream=True,
    timeout=(5, 60),
) as r:
    r.raise_for_status()
    total = 0
    for chunk in r.iter_content(chunk_size=65_536):
        total += len(chunk)
print(f"streamed {total} bytes")

Repeat fetches are usually avoidable. During development you will request the same page dozens of times. Caching responses locally removes that load from the target entirely and makes iteration instant — the patterns are in Caching and Incremental Crawling.

Common Errors and Fixes

requests.exceptions.HTTPError: 403 Client Error: Forbidden. The server refused the request as sent. Cause is almost always the header block — a missing or default User-Agent, no Accept-Language, or a Referer that does not match a plausible navigation path. Fix by sending a coherent browser header set and, where the site checks it, a Referer from the same origin.

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
    "Referer": "https://books.toscrape.com/",
}
r = requests.get("https://books.toscrape.com/catalogue/page-3.html", headers=headers, timeout=(5, 20))
r.raise_for_status()

requests.exceptions.ConnectTimeout or ReadTimeout. A connect timeout means the TCP handshake never completed — DNS, firewall, or a dead proxy. A read timeout means the connection opened but the server stopped sending. Distinguish them by using the tuple form and handling the two exceptions separately, because only the read case is worth retrying immediately.

import requests

try:
    r = requests.get("https://httpbin.org/delay/10", timeout=(3, 5),
                     headers={"User-Agent": "Mozilla/5.0 Chrome/136.0.0.0 Safari/537.36"})
except requests.exceptions.ConnectTimeout:
    print("network path or DNS problem — do not retry blindly")
except requests.exceptions.ReadTimeout:
    print("server is slow — retry with a longer read timeout")

requests.exceptions.TooManyRedirects: Exceeded 30 redirects. A redirect loop, usually because a cookie the server expects is missing, so it bounces you between a page and a consent or login endpoint forever. Use a Session so the cookie is retained, and inspect the loop rather than raising the limit.

import requests

with requests.Session() as s:
    s.headers.update({"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) Chrome/136.0.0.0 Safari/537.36"})
    r = s.get("https://httpbin.org/cookies/set?consent=1", allow_redirects=False, timeout=(5, 20))
    print(r.status_code, r.headers.get("Location"), s.cookies.get_dict())

json.decoder.JSONDecodeError: Expecting value: line 1 column 1.response.json() was called on something that is not JSON — nearly always an HTML error or challenge page returned with a 200. Check Content-Type first and log the first 200 characters of the body when it fails; that string usually names the anti-bot product you have hit.

urllib3.exceptions.MaxRetryError: … Caused by ProxyError. The proxy refused, timed out or returned a 407. Verify credentials are URL-encoded (a @ or : in a password must be percent-encoded) and that the scheme matches the target. Proxy selection and rotation are covered in Rotating Proxies and Managing IP Blocks.

A 200 with an empty result set and no exception. The most dangerous failure, because nothing raises. Add an assertion on the shape of the data, not just the status: if a listing page is expected to yield 20 items and yields 0, that is a hard error worth alerting on. See Detecting Silent Scraper Failures.

Frequently Asked Questions

Why do I get a 403 in Python but the page loads fine in my browser? The browser sends fifteen or so headers, reuses a warm TLS session, and carries cookies from earlier visits; a bare requests call sends four headers and identifies itself as a script. Copy a full header block from the browser's network panel first, and if that still fails the block is likely at the TLS fingerprint level rather than the header level.

Should I use response.text or response.content? Use response.text when you want a string and trust the declared encoding, and response.content when you need the raw bytes — for binary files, for hashing, or when you intend to decode with an encoding you determined yourself. Passing response.content to a parser is often safer, because parsers read the document's own charset declaration.

What timeout value should I use? Use a tuple, not a single number: around 5 seconds to connect and 15–30 seconds to read is a sensible default for HTML. A single scalar applies to both phases and forces you to choose one budget for two very different failure modes. Never omit the timeout entirely.

Does raise_for_status() cover everything I need to check? No. It raises on 4xx and 5xx, which is useful, but it says nothing about redirects that landed you somewhere else, about a 200 carrying a challenge page, or about a JSON endpoint that returned HTML. Treat it as one of three checks alongside response.url and Content-Type.

How many requests per second is acceptable? There is no universal number, but one request every 1–3 seconds per domain is a defensible starting point for a site with no published policy, and you should honour any Crawl-delay or rate limit the site states. Concurrency should be capped per domain rather than globally, so one slow host cannot consume the whole budget.

Related

• The Complete Guide to Python Web Scraping — the parent guide this page belongs to.
• Setting Up Your Python Scraping Environment — the interpreter, packages and TLS stack these requests run on.
• Step-by-Step Guide to Extracting Tables from HTML — turning a fetched page into rows.
• Selecting Elements with XPath and CSS Selectors — locating nodes once the body is parsed.
• Handling Pagination and Infinite Scroll — repeating these requests across a whole result set.