Logging In with Python Requests

This page gives the end-to-end version of the flow introduced in Handling Forms and Authentication: one function that signs a requests.Session in, raises if it did not work, and hands back a client every later request can use.

The short version: create a session, GET the login page with it, find the form that contains the password input, copy every named field out of that form, overwrite the two you own with values from the environment, POST the result to the resolved action URL, and assert on a positive marker before returning. Everything after that is an ordinary request against an authenticated session. The function below does exactly that in about forty lines and raises LoginError rather than returning something ambiguous.

Why the GET Before the POST Is Not Optional

It is tempting to skip straight to the POST — you know the URL, you know the two field names, why fetch a page you are not going to read? Because that first response does two things you cannot get any other way.

It plants the pre-authentication cookie. Django, Rails, Laravel and most other server-rendered frameworks issue an anonymous session on first contact and bind the form's token to it. A POST that arrives without that cookie is not a request with a bad token; it is a request from a client the server has never met, and the usual answer is 403.

It tells you the field names and the hidden values. Those are per-deployment choices, and they change. A payload assembled from the live markup keeps working when the site renames username to session[identity] or adds a form_build_id; a hard-coded dict does not, and it fails quietly, with a 200.

There is a third, smaller benefit: the response URL after redirects is the correct base for resolving the form's action. If /login redirected to /en/accounts/login/, resolving against the original path produces the wrong target.

How the Action URL Is Resolved

The action attribute is one of four things, and each needs different handling.

An absolute URL is used as-is. A root-relative path (/session) replaces the path of the page URL. A document-relative path (../session) is resolved against the page's directory. And a missing or empty action means "submit to the current URL" — which is the page URL after redirects, not the site root. urllib.parse.urljoin implements all four rules correctly, so there is no reason to do string surgery.

The method attribute matters too, though less often. It defaults to get in HTML, but a login form that omits method is a broken login form; in practice you will see method="post" on everything that carries a password, and if you genuinely find one that does not, read it as a signal that the real submission is happening in JavaScript.

A Complete Login Function

This runs as written. It targets httpbin.org, which echoes what it receives, so you can see exactly what your payload looks like on the wire before pointing it at a service you have an account with.

import os
import sys
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup
from bs4.element import Tag

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
}


class LoginError(RuntimeError):
    """Raised when a login attempt did not produce an authenticated session."""


def find_login_form(soup: BeautifulSoup) -> Tag:
    """The form holding a password input — never simply the first form."""
    for form in soup.find_all("form"):
        if form.find("input", attrs={"type": "password"}):
            return form
    forms = soup.find_all("form")
    if len(forms) == 1:
        return forms[0]
    raise LoginError(f"no password form found among {len(forms)} forms")


def collect_fields(form: Tag) -> dict[str, str]:
    """Every named field the form declares, carrying its default value."""
    fields: dict[str, str] = {}
    for field in form.find_all("input"):
        name = field.get("name")
        kind = (field.get("type") or "text").lower()
        if not name or kind in {"submit", "button", "image", "reset"}:
            continue
        if kind in {"checkbox", "radio"} and not field.has_attr("checked"):
            continue
        fields[name] = field.get("value", "")
    for select in form.find_all("select"):
        name = select.get("name")
        option = select.find("option", selected=True) or select.find("option")
        if name and option is not None:
            fields[name] = option.get("value", option.get_text(strip=True))
    return fields


def login(
    session: requests.Session,
    login_url: str,
    user_field: str,
    pass_field: str,
    username: str,
    password: str,
    success_selector: str,
) -> requests.Response:
    """Sign the session in, or raise LoginError explaining what failed."""
    page = session.get(login_url, timeout=20)
    page.raise_for_status()

    soup = BeautifulSoup(page.text, "lxml")
    form = find_login_form(soup)

    payload = collect_fields(form)
    payload[user_field] = username
    payload[pass_field] = password

    action = urljoin(page.url, form.get("action") or "")
    method = (form.get("method") or "post").lower()
    before = session.cookies.get_dict()

    response = session.request(method, action, data=payload, timeout=20)
    response.raise_for_status()

    after = session.cookies.get_dict()
    redirected = bool(response.history)
    cookie_changed = after != before
    marker_found = BeautifulSoup(response.text, "lxml").select_one(success_selector) is not None

    if not (redirected or cookie_changed or marker_found):
        raise LoginError(
            f"login rejected: status={response.status_code} url={response.url} "
            f"redirected={redirected} cookies_changed={cookie_changed} "
            f"marker={success_selector!r} found={marker_found}"
        )
    return response


def main() -> int:
    username = os.environ.get("SITE_USERNAME", "demo-user")
    password = os.environ.get("SITE_PASSWORD", "demo-secret")

    session = requests.Session()
    session.headers.update(HEADERS)
    try:
        result = login(
            session,
            login_url="https://httpbin.org/forms/post",
            user_field="custname",
            pass_field="custtel",
            username=username,
            password=password,
            success_selector="pre",
        )
    except LoginError as error:
        print(f"could not sign in: {error}", file=sys.stderr)
        return 1

    print("signed in, landed on", result.url)
    print("cookies held:", sorted(session.cookies.get_dict()))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Four details are doing most of the work. find_login_form keys on the password input, which is structural rather than positional, so it survives a new search box appearing above the login. collect_fields preserves hidden values, so a token or a next parameter is carried without you naming it. urljoin(page.url, ...) uses the post-redirect URL. And the verification compares the cookie jar before and after, which catches the very common case where a framework rotates the session identifier on success — a signal that is independent of the HTML and therefore does not break on a redesign.

Two guards are worth adding for real targets. session.max_redirects = 5 turns a redirect loop into a fast exception instead of a slow one, and a timeout on every call — already present above — stops a hung connection from stalling the run.

The credentials arrive through os.environ and never appear as literals. Keep them that way: a .env file listed in .gitignore for local work, and the runner's secret store for anything scheduled. And read the site's terms first, because a service that prohibits automated access to an account does not become permissive because the script is well written.

Diagnosing a login that will not take

When the function raises, the message already contains the four facts that matter, but the payload itself is what usually gives the answer away. Print the keys you are about to submit next to the input names the page declares, and the mismatch is normally visible immediately.

from bs4 import BeautifulSoup

html = """
<form action="/session" method="post">
  <input type="hidden" name="authenticity_token" value="Vd91">
  <input type="email" name="session[email]">
  <input type="password" name="session[password]">
  <input type="checkbox" name="session[remember]" value="1">
</form>
"""
form = BeautifulSoup(html, "lxml").find("form")
declared = {i.get("name") for i in form.find_all("input") if i.get("name")}
sending = {"username", "password", "authenticity_token"}

print("declared but not sent:", sorted(declared - sending))
print("sent but not declared:", sorted(sending - declared))

Anything in the second list is a field the server never asked for, and anything in the first is a field you are dropping. Bracketed names such as session[email] are the single most common cause of a silent rejection, because they look like a nested structure and are in fact one flat key — pass the whole string, brackets included, exactly as the attribute spells it.

Print the resolved action URL too. A POST to /login when the form named /sessions returns the login page with a 200, which is indistinguishable from a wrong password unless you look.

Reusing the Session Afterwards

Once login() returns, the session is the only object your scraper needs. It carries the cookie jar, the default headers and a pooled TLS connection to the host, so subsequent pages cost one round trip each rather than a fresh handshake.

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": (
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
    )
})

# Stand-in for the authenticated session returned by login().
session.cookies.set("session_demo", "ok", domain="httpbin.org", path="/")

for page in range(1, 4):
    response = session.get(
        "https://httpbin.org/cookies", params={"page": page}, timeout=20
    )
    response.raise_for_status()
    print(page, response.json()["cookies"])

Pass the session into your fetch functions rather than creating one inside them. A helper that instantiates requests.Session() per call throws away both the cookies and the connection pool, which is a correctness bug and a performance bug at the same time. If a request comes back looking anonymous mid-run, call login() once more and retry that single request — do not loop.

Edge Cases and Caveats

• The password field is not on the first page. Identifier-first flows ask for the email, then serve the password form at a second URL, often with a fresh token. Treat it as two logins in sequence, parsing each page in turn.
• The form posts to a different host. A cross-host action means the credential goes to an identity provider, and the main-site cookie only appears after the redirect chain returns. response.history shows every hop and which host set what.
• Unchecked boxes must be omitted, not blanked. Browsers do not send unchecked checkboxes at all. Sending remember="" can be read as a present-and-false value by one framework and as present-and-true by another.
• Honeypot inputs must stay empty. Some forms include a hidden text field with a plausible name that real users never fill. Copying the default empty value is correct; inventing a value is what gets the request rejected.
• Non-ASCII credentials. requests encodes the body as UTF-8 by default, but a few older applications expect latin-1. If a password with an accented character fails while an ASCII one succeeds, encode the body yourself and set Content-Type explicitly.
• Referer and Origin checks. Some servers reject a POST whose Referer is missing. Setting Referer to page.url and Origin to the scheme-plus-host of the login page costs nothing and removes a whole class of 403s.
• Rate limits apply to logins too. Repeated authentication attempts are exactly the pattern abuse systems watch for. Log in once per run, keep the session, and never retry a rejected credential automatically — a wrong password will not become right on the second try, and repeated attempts can lock the account.

Frequently Asked Questions

Should I use session.post or requests.post for the login? Always session.post. A bare requests.post creates a throwaway client: it does not send the cookie you received from the login page, and it discards the cookie the server sets on success. The whole point of the session object is that those two things are carried for you.

How do I find the right field names without opening DevTools? Fetch the login page and print every input element's name, type and value. That listing is the authoritative answer for that deployment, and it takes five lines. Guessing from convention fails often enough that it is not worth the time saved.

Why does my login work in the browser but return the login page in Python? The three usual causes are a missing hidden field, a token bound to a cookie you did not keep, and a payload posted to the wrong URL because the action was relative. Print the resolved action, the payload keys, and the cookie jar before and after — one of the three will be visibly wrong.

Is it safe to store the session cookie on disk between runs? It is a credential, so treat it like one: restrictive file permissions, outside the repository, and never in a commit. Reusing it is good practice — it means fewer password submissions over the wire — but a leaked session cookie grants the same access a leaked password does until it expires.

Related

• Handling Forms and Authentication — the parent topic for this article
• Handling CSRF Tokens When Scraping — when the hidden field is a token that rotates
• Scraping Behind Bearer Tokens and OAuth — logins that return a token instead of a cookie
• Persisting a Session Between Runs — saving the authenticated jar to disk