Handling Forms and Authentication

A large share of the data worth collecting sits behind a sign-in page, and this guide — part of The Complete Guide to Python Web Scraping — covers how a Python client gets through that page and stays signed in for the rest of a run. The scope is narrow and deliberate: authenticating as yourself, with your own account, to a service you are entitled to use, and where automated access is permitted by the terms you agreed to. Nothing here is about reaching an account you do not hold.

The single most common mistake is to treat a login as one request. It is at least three, and usually four. You fetch the page that contains the form, because that response is what tells you the field names and hands you the first cookie. You build a payload from the markup you just received. You post that payload to the URL the form itself names. Then you check that the result is actually an authenticated session rather than the login page rendered a second time with an error banner. Skip any of those steps and you get a scraper that appears to work, returns status 200 for every request, and silently collects the logged-out version of every page.

When to Use Each Authentication Approach

Sites hand out sessions in several different ways, and the mechanism dictates the code you write. Open the network panel in your browser, sign in by hand once, and look at what the submit request actually is before choosing.

Two of those rows deserve emphasis. If the site exposes a token endpoint, it almost certainly has a documented API, and an API contract is far more stable than a page layout — that route is described in Reverse-Engineering Private APIs. And if a second factor is involved, the honest answer is that a scripted login is the wrong tool; look for a personal access token in the account settings instead.

Prerequisites

Python 3.10 or newer. The HTTP client and the parser do all the work here — you need the parser because reading a form correctly means parsing HTML, not pattern-matching it.

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install requests beautifulsoup4 lxml httpx python-dotenv

requests provides the session object, beautifulsoup4 with the lxml parser reads the form, httpx is there for the token-based flows, and python-dotenv loads credentials from a local .env file that is never committed. If you have not set up an isolated environment yet, start with Setting Up Your Python Scraping Environment.

Add .env to .gitignore before you write a single line of login code. A credential that reaches a commit is a credential you have to rotate, and history rewrites rarely catch every copy.

Step-by-Step: Getting a Python Client Signed In

1. Read the form rather than guessing its fields

Field names are not conventions, they are choices. username, user, login, email, identity, session[email] and _username are all in wide use, and a payload with the wrong key produces a perfectly normal-looking 200 with an error message inside it. Fetch the page and print what is really there.

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-GB,en;q=0.9",
}


def describe_forms(url: str) -> None:
    response = requests.get(url, headers=HEADERS, timeout=20)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "lxml")

    for index, form in enumerate(soup.find_all("form")):
        print(f"form[{index}] action={form.get('action')!r} method={form.get('method')!r}")
        for field in form.find_all(["input", "select", "textarea"]):
            print(
                f"    {field.name}"
                f" type={field.get('type')!r}"
                f" name={field.get('name')!r}"
                f" value={field.get('value')!r}"
            )


describe_forms("https://httpbin.org/forms/post")

Run this against your target once, by hand, before writing anything else. Two things usually surprise people. The first form on a page is very often the site search box, not the login. And the login form typically contains fields you never saw in the browser — a token, a next redirect target, a form build identifier, sometimes a honeypot input that must stay empty.

If the listing comes back empty, you did not receive the login page. The usual culprits are a consent interstitial, a geographic redirect, or a protection layer serving a challenge instead of the document. Print response.url, len(response.text) and response.headers.get("content-type") before concluding that a site has no form; a 4 KB body where you expected 60 KB is a block page, not a login page. Sending a realistic Accept, Accept-Language and Accept-Encoding set alongside the User-Agent removes a surprising share of these, because a request advertising only */* looks nothing like a browser.

2. Build the payload from the markup, not from memory

Start from every named input the form declares, keep the server's own default values, and overwrite only the two fields that are genuinely yours. That single habit removes an entire family of bugs, because any hidden value the site adds next month is carried automatically.

from bs4 import BeautifulSoup
from bs4.element import Tag


def form_payload(form: Tag) -> dict[str, str]:
    """Every named field the form declares, with its default value."""
    payload: dict[str, str] = {}

    for field in form.find_all("input"):
        name = field.get("name")
        if not name or field.get("type") in {"submit", "button", "image", "reset"}:
            continue
        if field.get("type") in {"checkbox", "radio"} and not field.has_attr("checked"):
            continue
        payload[name] = field.get("value", "")

    for select in form.find_all("select"):
        name = select.get("name")
        if not name:
            continue
        chosen = select.find("option", selected=True) or select.find("option")
        payload[name] = chosen.get("value", chosen.get_text(strip=True)) if chosen else ""

    for textarea in form.find_all("textarea"):
        name = textarea.get("name")
        if name:
            payload[name] = textarea.get_text()

    return payload


html = """
<form action="/session" method="post">
  <input type="hidden" name="csrf_token" value="c9f1e7">
  <input type="hidden" name="next" value="/dashboard">
  <input type="text" name="identity" value="">
  <input type="password" name="password" value="">
  <input type="checkbox" name="remember" value="1" checked>
  <input type="submit" name="commit" value="Sign in">
</form>
"""
form = BeautifulSoup(html, "lxml").find("form")
print(form_payload(form))

Note what is excluded and why. Submit buttons are skipped by default because a browser only sends the one that was clicked, and sending all of them can confuse a server that branches on commit. Unchecked checkboxes are skipped because a browser omits them entirely — sending remember="" is not the same as not sending remember, and some frameworks treat the presence of the key as truth.

Locating the right form is its own small problem. The most reliable rule is structural rather than positional: pick the form that contains a password input. That is a single CSS selector, and the technique generalises — the reasoning behind choosing selectors that survive markup churn is set out in Selecting Elements with XPath and CSS Selectors.

3. Use one session for the whole flow

Cookies are the entire point. The GET that delivers the form usually also sets an anonymous session cookie, and many frameworks bind the form's token to that specific cookie. Fetch with requests.get and post with requests.post as separate calls and the two are strangers: the token belongs to a session the server has never seen again, and you get a 403 that looks like a permissions problem but is really a state problem.

import os
import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

session = requests.Session()
session.headers.update(HEADERS)

page = session.get("https://httpbin.org/forms/post", timeout=20)
form = BeautifulSoup(page.text, "lxml").find("form")

payload = {"custname": os.environ.get("DEMO_NAME", "example"), "comments": "hello"}
result = session.post("https://httpbin.org/post", data=payload, timeout=20)
print(result.status_code, sorted(session.cookies.get_dict()))

The session also gives you connection pooling, a shared header set and a single place to attach retries or proxies. The mechanics of the cookie jar — scope, expiry, why a cookie set on www.example.com is not sent to api.example.com — are covered in Managing Cookies and Sessions.

4. Post to the URL the form names

An action attribute may be absolute, relative, or absent. Absent means "post back to the current URL", which is not the same as posting to the site root. Resolve it with urljoin against the URL of the page you fetched — and use the final URL after redirects, not the one you asked for.

from urllib.parse import urljoin


def resolve_action(page_url: str, action: str | None) -> str:
    """Turn a form action attribute into an absolute URL."""
    if not action:
        return page_url
    return urljoin(page_url, action)


print(resolve_action("https://example.com/u/login", "../session"))
print(resolve_action("https://example.com/u/login", None))
print(resolve_action("https://example.com/u/login", "https://auth.example.com/in"))

Watch the third case. A cross-host action means the credential goes to a different origin, and the session cookie you get back is scoped to that other host. If the site does that, you will need to follow the redirect chain back to the original host before the main-site cookie appears — requests does this automatically, and response.history shows you every hop.

Send the body in the encoding the form declares. enctype="application/x-www-form-urlencoded" is the default and maps to data=; multipart/form-data maps to files=; a JSON login endpoint wants json=. Passing a dict to data= when the server expects JSON produces a 400 with a body that rarely explains itself.

5. Verify the login, because the status code will not

This is the step people skip, and it is the one that determines whether your dataset is real. A failed login almost always returns 200 OK with the form re-rendered. A successful one usually looks different in three independent ways, and checking more than one is what makes the check robust.

import requests
from bs4 import BeautifulSoup


class LoginError(RuntimeError):
    """Raised when a login attempt did not produce an authenticated session."""


def assert_logged_in(response: requests.Response, marker: str, cookie_name: str) -> None:
    redirected = bool(response.history)
    has_cookie = cookie_name in response.cookies or cookie_name in getattr(
        response.request, "_cookies", {}
    )
    soup = BeautifulSoup(response.text, "lxml")
    has_marker = soup.select_one(marker) is not None

    if not (redirected or has_cookie or has_marker):
        raise LoginError(
            f"login failed: status={response.status_code} url={response.url} "
            f"redirected={redirected} cookie={has_cookie} marker={has_marker}"
        )

The three signals, in order of reliability:

• A redirect away from the login path. response.history is non-empty and response.url no longer contains /login. This is the strongest signal on server-rendered sites because the 302 is the framework's own success path.
• A session cookie that changed. Most frameworks rotate the session identifier on successful authentication specifically to prevent session fixation. Snapshot session.cookies.get_dict() before the post and compare after; a changed value is near-proof.
• A logged-in element in the HTML. A sign-out link, an account menu, the user's display name. Cheap to check and easy to read, but it breaks on a redesign, so treat it as confirmation rather than as the only test.

Failing loudly matters more than it sounds. A scraper that quietly continues with an anonymous session will run to completion, produce plausible row counts, and fill a database with public placeholder content that nobody notices for weeks.

The same check is worth repeating mid-run, not only at login. Sessions expire, and a long crawl can cross that boundary halfway through. A cheap way to notice is to assert the logged-in marker on every response rather than only on the first: one select_one call per page costs microseconds and converts a silent data-quality failure into an immediate, actionable exception. When it fires, re-authenticate once and retry that single request; if it fires again, stop the run rather than continuing to collect anonymous pages.

6. Keep credentials out of the code

Credentials belong in environment variables, loaded at process start, never assigned as literals and never written into a notebook cell that gets shared.

import os
import sys
from dotenv import load_dotenv

load_dotenv()  # reads .env in the working directory; .env stays in .gitignore


def credential(name: str) -> str:
    value = os.environ.get(name)
    if not value:
        sys.exit(f"missing required environment variable: {name}")
    return value


USERNAME = credential("SITE_USERNAME")
PASSWORD = credential("SITE_PASSWORD")
print(f"loaded credentials for {USERNAME[:2]}***")

Three rules worth following without exception. Never log the password, and never log a whole request body during debugging — masking after the fact does not remove it from yesterday's log file. Never commit .env; add it to .gitignore in the first commit of the project. And when a scraper runs on a schedule, put the values in the runner's secret store rather than in the repository, as described in Scheduling Scrapers with Cron and GitHub Actions.

Read the site's terms before you automate a login at all. Many services restrict automated access to an account, some restrict it only above a request rate, and a few offer an official API precisely so you do not have to drive the HTML. Checking is a five-minute job that decides whether the project is viable.

7. Know when the login needs a real browser

An HTTP client can only send what exists in the response it received. When the submitted payload contains a value computed by JavaScript at click time — a device fingerprint, a signed nonce, a proof-of-work result, an encrypted password blob — there is nothing in the static HTML to copy, and no amount of header tuning will produce it.

The signs are consistent: the form has no action and no method, the submit handler is bound in a bundle, the network panel shows a JSON request whose body contains fields with no matching inputs, or the payload includes a long opaque string that changes on every page load. In those cases drive a real engine, log in once, and export the resulting cookies to a plain HTTP client for the bulk of the work — a hybrid that keeps the browser cost to a single page load. That handover is covered in Using Playwright for Modern Web Automation.

The economics favour the hybrid heavily. A browser context costs roughly 80–150 MB of resident memory and one to three seconds per page navigation; a pooled HTTP request costs a few kilobytes and one round trip. Paying the browser price once for authentication and then running ten thousand pages through requests is typically two orders of magnitude cheaper than driving the whole crawl through the browser. The handover is mechanical: read the cookies out of the browser context after login and set each one on a session, matching name, value, domain and path. What does not transfer is anything the site derives from live JavaScript — if pages are also signed per request, the browser has to stay in the loop.

Performance and Scaling Considerations

• Log in once per run, not once per page. A login is typically two round trips plus a password hash on the server, which is deliberately slow — bcrypt and argon2 are tuned to take 50–250 ms of CPU. Re-authenticating in a loop is the most expensive mistake available and the one most likely to trip a rate limiter.
• Persist the cookie jar across runs. A scraper that runs every fifteen minutes does not need a fresh login each time. Serialise the jar, reload it, and only re-authenticate when a request comes back unauthenticated. The mechanics are in Persisting a Session Between Runs.
• One session per identity, not per thread. requests.Session is not documented as thread-safe. Under concurrency, either serialise access or give each worker its own session seeded from the same cookie jar. For async work, httpx.AsyncClient holds a cookie jar with the same semantics — see Asynchronous Scraping with Asyncio and HTTPX.
• Do not spread one account over many proxies. Sessions are frequently bound to an address range, and an account whose requests arrive from four continents in a minute is an obvious anomaly. If you rotate exits, keep one session pinned to one exit for its lifetime; the trade-offs are in Rotating Proxies and Managing IP Blocks.
• Budget for re-authentication. Sessions expire on a timer, on inactivity, and sometimes on a deploy. Wrap the fetch layer so that a single unauthenticated response triggers one re-login and one retry, then gives up. Unbounded retry on a 401 is an infinite loop with a rate limit attached.

Common Errors and Fixes

A 200 OK that is really a failure. The most frequent symptom of all. The response body is the login page again, often with a small error element you never look at. Fix it by asserting on a positive marker rather than on response.ok, as in step 5.

403 Forbidden on the POST, with a valid-looking session. Nearly always a token problem: not sent, sent stale, or sent in the body when the server wants it in a header. Some frameworks additionally require Referer and Origin to match the site. The full diagnosis is in Handling CSRF Tokens When Scraping.

KeyError or AttributeError: 'NoneType' object has no attribute 'get'. Your form lookup returned nothing, because the markup differs from what you expected or the page was a block page rather than the login page. Guard the lookup and print len(response.text) plus the first 300 characters when it fails.

The POST succeeds but the next request is anonymous. Two usual causes. You used requests.post instead of session.post, so the cookie went nowhere. Or the cookie is scoped to a different host than the one you queried next — print session.cookies and check the domain attribute of each entry.

requests.exceptions.TooManyRedirects. A redirect loop between the login page and the target, which almost always means the session cookie is not being accepted at all. Check that you are not passing cookies={} per request, which replaces the jar for that call.

A 302 to /login?next=/account after a successful-looking POST. The server accepted the credentials but rejected the session, typically because the cookie was set on a host you then abandoned. Follow response.history hop by hop and note which host issued which cookie; the header-level view is explained in Understanding HTTP Requests and Responses.

401 Unauthorized on an API call that worked in the browser. The browser was sending a bearer token from memory, not a cookie. Look for an Authorization header in the network panel rather than assuming cookie auth.

400 Bad Request from the login endpoint. Usually an encoding mismatch: a JSON endpoint received a form-encoded body, or a form endpoint received JSON. Match the enctype the form declares — data= for application/x-www-form-urlencoded, files= for multipart/form-data, json= for an API login — and check Content-Type on the request you actually sent, not the one you intended.

UnicodeEncodeError or a rejected password containing accented characters. requests encodes the body as UTF-8, which is right almost everywhere. A small number of older applications expect latin-1. If an ASCII password succeeds and a non-ASCII one fails, encode the body yourself and set Content-Type with an explicit charset.

The account gets locked after a few runs. Automatic retry on a rejected credential is the cause more often than anything else. A wrong password is still wrong on the second attempt, and repeated failures trip lockout policies. Retry transient network errors and 5xx responses; never retry an authentication rejection.

Frequently Asked Questions

Why does my login return 200 but the pages are still logged out? Because a failed login is usually rendered as a normal page rather than as an error status. The server returns the login form again with a message inside the body, so response.ok is true and raise_for_status() stays silent. Verify with a positive signal instead — a redirect away from the login path, a changed session cookie, or an element that only exists when signed in.

Do I need to send the hidden fields if the site works without them? Send them. They cost nothing, and a hidden field that is optional today becomes mandatory the moment the site adds token rotation or a form version identifier. Building the payload from the parsed form rather than from a hard-coded dict means those changes never reach your code.

Where should the username and password live? In environment variables, loaded from a .env file that is listed in .gitignore, or in the secret store of whatever runs the job. Never in the source, never in a committed configuration file, and never printed in a log line. Check the site's terms as well — plenty of services restrict automated access to an account regardless of how the credentials are stored.

When do I have to use a browser instead of requests? When the submitted payload contains a value that only exists after JavaScript runs: a computed signature, a device fingerprint, an encrypted password field, or a challenge response. If the network panel shows fields with no matching input elements in the HTML, an HTTP client cannot reproduce them. Log in with a browser engine once, then export the cookies to a fast HTTP client for the rest of the run.

How do I keep a session alive across scheduled runs? Serialise the cookie jar at the end of a run and load it at the start of the next, then treat an unauthenticated response as the trigger to re-authenticate once. This turns a login from a per-run cost into an occasional one and greatly reduces how often you send credentials over the wire.

Related

• The Complete Guide to Python Web Scraping — the parent guide to this section
• Logging In with Python Requests — the complete runnable login script
• Handling CSRF Tokens When Scraping — where tokens hide and why a 403 appears
• Scraping Behind Bearer Tokens and OAuth — token auth, refresh flows and expiry
• Managing Cookies and Sessions — how the cookie jar behaves once you are signed in