Fixing Common Unicode Errors in Python Scraping

Pattern matching only works on text you decoded correctly, so before any of the techniques in Extracting Data with Regular Expressions can be trusted, the bytes coming off the wire have to become the right characters.

Work from response.content (bytes), not response.text, and choose the codec deliberately: a byte order mark first, then the HTTP Content-Type charset, then the document's own <meta charset>, then statistical detection, and only then a fallback. Decoding with the wrong codec usually does not raise — it produces mojibake that silently reaches your database — so the absence of an exception is not evidence that the decode was correct.

The Three Distinct Failures

Encoding bugs in a scraper come in three shapes, and confusing them wastes hours because they need opposite fixes.

UnicodeDecodeError is raised when bytes cannot be interpreted under the codec you asked for. The message names the offending position: 'utf-8' codec can't decode byte 0xe9 in position 214: invalid continuation byte. A lone 0xe9 is legal Latin-1 and illegal UTF-8, so this almost always means the page is in a single-byte legacy encoding.

UnicodeEncodeError is the mirror image: a decoded string contains characters your output target cannot represent. 'charmap' codec can't encode character '—' in position 12 is the classic Windows console failure, where the em dash exists in your string but not in the terminal's code page. Nothing is wrong with the scraped data — the writing side is the problem.

Mojibake is the third and worst, because there is no exception at all. Bytes get decoded under a codec in which every byte happens to be legal but means something else. UTF-8 c3 a9 (é) read as cp1252 becomes é, two perfectly valid characters. The scraper reports success and the corruption is discovered weeks later in a report.

The practical consequence: never treat "no traceback" as "correct encoding". Assert on the output instead — a page that should contain accented characters and instead contains Ã, ’ or  has been double-decoded.

Encoding Precedence and Why requests Gets It Wrong

requests populates response.encoding from the HTTP Content-Type header. If that header names a charset, response.text uses it. If the header is text/html with no charset, requests leaves encoding as None and response.text falls back to charset-normalizer's detection — which is decent but probabilistic, and needs a reasonable amount of text to be confident.

Crucially, requests does not read the document's <meta charset> tag. A server that sends a bare Content-Type: text/html while the HTML declares <meta charset="windows-1874"> will be decoded by the header rules alone, and the declaration inside the body is ignored.

apparent_encoding runs the detector and is what you compare against when the header looks suspicious. Note that it re-scans the whole body each time you touch it, so cache it rather than referencing it in a loop.

import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

response = requests.get("https://httpbin.org/encoding/utf8", headers=HEADERS, timeout=20)
print(f"header says   : {response.encoding}")
print(f"detector says : {response.apparent_encoding}")
print(f"first bytes   : {response.content[:16]!r}")

When those two disagree, trust the detector for legacy pages and the header for anything modern and well maintained. When they agree, you are almost certainly fine.

A Decoder That Resolves the Precedence Explicitly

The function below implements the ladder in the figure above and returns the codec it used, so you can log it and spot a site that changes encoding mid-crawl.

"""Decode response bytes using an explicit source-of-truth ladder."""
import codecs
import re

import requests

_META_CHARSET = re.compile(rb'<meta[^>]+charset=["\']?\s*([\w\-]+)', re.IGNORECASE)

_BOMS: tuple[tuple[bytes, str], ...] = (
    (codecs.BOM_UTF8, "utf-8-sig"),
    (codecs.BOM_UTF32_LE, "utf-32"),
    (codecs.BOM_UTF32_BE, "utf-32"),
    (codecs.BOM_UTF16_LE, "utf-16"),
    (codecs.BOM_UTF16_BE, "utf-16"),
)


def decode_response(response: requests.Response) -> tuple[str, str]:
    """Return (text, codec_used) using BOM, header, meta, detector, fallback."""
    raw = response.content

    for bom, codec in _BOMS:
        if raw.startswith(bom):
            return raw.decode(codec), codec

    candidates: list[str] = []
    declared = response.headers.get("Content-Type", "")
    if "charset=" in declared.lower():
        candidates.append(declared.lower().split("charset=")[-1].split(";")[0].strip())

    match = _META_CHARSET.search(raw[:4096])
    if match:
        candidates.append(match.group(1).decode("ascii", "ignore").lower())

    if response.apparent_encoding:
        candidates.append(response.apparent_encoding.lower())

    for codec in candidates:
        # ISO-8859-1 in a header is usually the HTTP default, not a real claim.
        if codec in {"iso-8859-1", "latin-1", "latin1"} and len(candidates) > 1:
            continue
        try:
            return raw.decode(codec), codec
        except (UnicodeDecodeError, LookupError):
            continue

    return raw.decode("cp1252", errors="replace"), "cp1252/replace"


if __name__ == "__main__":
    resp = requests.get(
        "https://books.toscrape.com/",
        headers={"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)"},
        timeout=20,
    )
    text, used = decode_response(resp)
    print(f"decoded with {used}, {len(text)} characters")

Three details make this correct rather than merely plausible. utf-8-sig strips the BOM from the resulting string — plain utf-8 leaves a U+FEFF at position zero, which then appears as an invisible first character in your first column and breaks equality checks and CSV headers. LookupError is caught alongside UnicodeDecodeError because pages declare charsets Python has never heard of, such as utf8mb4 or a typo. And the iso-8859-1 skip exists because RFC 2616 made it the default charset for text/*, so servers that set nothing at all often appear to be declaring it.

Feeding raw bytes straight to your parser is an equally valid route and often the better one — both lxml and BeautifulSoup read the meta declaration themselves, which is one of the reasons BeautifulSoup vs lxml: Which Parser Is Faster recommends passing response.content rather than response.text.

Normalising Text Before It Reaches Storage

A correct decode is not yet clean text. Scraped strings routinely carry non-breaking spaces, soft hyphens, zero-width joiners, and the same visible character encoded two different ways. é can be one code point (U+00E9) or two (U+0065 U+0301); they render identically, compare unequal, and will produce duplicate rows in any deduplication pass.

import re
import unicodedata

# Escapes, not literals: invisible characters pasted into source are unreviewable.
_INVISIBLE = re.compile(
    "[\u200b-\u200f\u202a-\u202e\u2060-\u2064\ufeff\u00ad]"  # zero-width, bidi, soft hyphen
    "|[\x00-\x08\x0b\x0c\x0e-\x1f\x7f-\x9f]"  # control characters
)
_REPLACEMENTS = {
    "\u00a0": " ",  # non-breaking space
    "\u2019": "'",  # right single quotation mark
    "\u2013": "-",  # en dash
    "\u2014": "-",  # em dash
}


def clean_text(raw: str) -> str:
    """Normalise, drop invisible characters, and collapse whitespace."""
    text = unicodedata.normalize("NFC", raw)
    text = _INVISIBLE.sub("", text)
    for source, target in _REPLACEMENTS.items():
        text = text.replace(source, target)
    return re.sub(r"\s+", " ", text).strip()


if __name__ == "__main__":
    messy = "Caf\u00e9\u00a0 de\u200b luxe\u2019s\u00ad menu "
    print(repr(clean_text(messy)))

Use NFC for storage and display, because it is the shortest form and what almost every other system expects. Use NFKC only when you deliberately want compatibility folding — it rewrites ① to 1, fi to fi, and full-width Latin to ASCII, which is useful for search keys and destructive for anything you plan to show a user. Keep both if you need them: the normalised value for matching, the original for display. Broader field-level rules belong with the rest of Cleaning and Validating Scraped Data.

Repairing Text That Was Already Corrupted

Sometimes the damage is upstream and you cannot re-fetch. Mojibake is reversible when the corruption was a clean single mis-decode: encode the mangled string back to bytes under the codec that produced it, then decode those bytes under the codec that should have been used.

def unmojibake(text: str) -> str:
    """Undo one round of UTF-8 bytes wrongly decoded as cp1252."""
    try:
        return text.encode("cp1252").decode("utf-8")
    except (UnicodeEncodeError, UnicodeDecodeError):
        return text


print(unmojibake("Café déjà  vu"))  # -> Café déjà vu

The try block matters because the round trip fails on text that was never mojibake, and you want to leave those strings untouched rather than crash the batch. This only works for a single mis-decode; text that was mangled twice needs the operation applied twice, and text that passed through a lossy step such as errors="ignore" cannot be recovered at all because the bytes are gone. The ftfy package automates the detection and handles the multi-round cases, which is worth the dependency when you are cleaning an archive rather than fixing a live scraper.

The better fix is always upstream. Log the codec your decoder chose alongside every record, and a site that flips from cp1252 to UTF-8 during a redesign shows up as a change in that field rather than as a slow drip of broken rows.

Edge Cases and Caveats

• UnicodeEncodeError on print, not on scrape. The Windows console defaults to a legacy code page. Set PYTHONUTF8=1 or PYTHONIOENCODING=utf-8, or on Python 3.7+ call sys.stdout.reconfigure(encoding="utf-8"). Do not "fix" it by stripping characters from your data.
• Excel and the CSV BOM. Excel reads a plain UTF-8 CSV as the system code page. Write with encoding="utf-8-sig" so it sees the BOM and switches; every other tool ignores it.
• errors="ignore" silently deletes data. Prefer errors="replace", which leaves a visible � you can count and alert on. A rising replacement-character rate is one of the cheapest signals for Detecting Silent Scraper Failures.
• Surrogates from errors="surrogateescape". Strings carrying lone surrogates raise UnicodeEncodeError on JSON serialisation and on most database drivers. Re-encode with text.encode("utf-8", "replace").decode("utf-8") before storing.
• Gzip and Brotli. requests decompresses transparently, so response.content is already plaintext bytes. If you use stream=True and read the raw socket, you get compressed bytes and every decode fails at position zero.
• PostgreSQL rejects NUL. \x00 is legal in a Python str but illegal in a text column, producing ValueError: A string literal cannot contain NUL (0x00) characters. The control-character strip above removes it; see Saving Scraped Data to PostgreSQL for the rest of the insert path.
• Regex and Unicode. In Python 3 \w and \b are Unicode-aware by default on str patterns but ASCII-only on bytes patterns. Matching on undecoded bytes is a common reason an accented word fails to match.
• JSON APIs. response.json() assumes UTF-8 per RFC 8259 and ignores response.encoding. An API that genuinely serves cp1252 JSON needs json.loads(response.content.decode("cp1252")).

Frequently Asked Questions

Why does my scraper produce é instead of é even though nothing failed? The bytes were UTF-8 but were decoded as cp1252 or latin-1, in which every byte is individually legal, so no exception is raised. Check response.encoding against response.apparent_encoding; if the header claims ISO-8859-1 while the detector says UTF-8, the header is the HTTP default rather than a real declaration and you should override it.

Should I use response.text or response.content? Use response.content and decide the codec yourself, or hand the bytes straight to a parser that reads the document's own declaration. response.text is convenient but applies the header charset, or a statistical guess when the header is silent, and it gives you no signal about which of the two it used.

Is latin-1 a safe fallback? It is a safe way to avoid an exception, not a safe way to get correct text, because latin-1 maps all 256 byte values so nothing can ever fail. Real-world "latin-1" pages are usually windows-1252, which additionally defines the curly quotes and the em dash in the 0x80–0x9F range that latin-1 leaves as control characters, so cp1252 is the better fallback almost every time.

How do I stop the same word appearing twice in my data? Apply unicodedata.normalize("NFC", value) before any comparison or uniqueness constraint, so that a precomposed é and a decomposed e-plus-accent collapse to one representation. Normalise at write time and keep the normalisation form consistent across every writer touching the table, otherwise the duplicates come back through whichever path skipped it.

Related

• Extracting Data with Regular Expressions — the parent topic, and where the decoded text goes next.
• Understanding HTTP Requests and Responses — headers, content negotiation, and what the server actually promised.
• Parsing HTML with BeautifulSoup — passing bytes to the parser instead of a decoded string.
• Validating Scraped Data with Pydantic — catching corrupted fields before they reach storage.