Parsing XML Sitemaps with Python

Most crawlers spend their first thousand requests rediscovering a URL list the site already publishes, and this article — part of Parsing JSON and XML API Responses in Python — covers how to read that list properly, including the discovery chain, the namespace that makes half of all sitemap parsers return nothing, and the lastmod field that turns a full recrawl into an incremental one.

A sitemap is the cheapest crawl frontier available. Four HTTP requests against a site with a hundred thousand pages typically yield every URL the site wants indexed, each with a machine-readable last-modified date, at a total cost of a few megabytes of gzipped XML. Compare that against link-following, which needs one request per page just to discover the next page, and which cannot see anything that is not linked from somewhere you have already visited.

Finding the Sitemap Before Guessing at It

/sitemap.xml is a convention, not a rule. The authoritative location is the Sitemap: directive in robots.txt, which may appear more than once, may point at a different hostname, and is defined to be independent of any User-agent block — it applies globally regardless of where in the file it sits.

Python's urllib.robotparser exposes it through site_maps(), which returns None rather than an empty list when no directive exists, so the call needs a guard.

from urllib.parse import urljoin
from urllib.robotparser import RobotFileParser

import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
    "Accept": "application/xml,text/xml;q=0.9,*/*;q=0.8",
}

def discover_sitemaps(site: str) -> list[str]:
    """Sitemaps declared in robots.txt, falling back to the conventional path."""
    robots_url = urljoin(site, "/robots.txt")
    parser = RobotFileParser()
    try:
        resp = requests.get(robots_url, headers=HEADERS, timeout=10)
        resp.raise_for_status()
        parser.parse(resp.text.splitlines())
    except requests.RequestException:
        return [urljoin(site, "/sitemap.xml")]
    declared = parser.site_maps()
    return list(declared) if declared else [urljoin(site, "/sitemap.xml")]

print(discover_sitemaps("https://books.toscrape.com"))

Fetching robots.txt with requests rather than letting RobotFileParser.read() do it is deliberate: read() uses urllib, which sends a Python-urllib/3.x user agent that a growing number of edge providers reject outright. Feeding the parser text you fetched yourself keeps your headers consistent with the rest of the crawl.

When robots.txt names nothing, /sitemap.xml is worth one speculative request, followed by /sitemap_index.xml and /sitemap-index.xml. Beyond those three, guessing is not productive — a site that hides its sitemap has usually decided not to have one.

Index or Urlset: Two Documents, One Extension

Two distinct root elements share the .xml extension and the same namespace. A <urlset> contains <url> entries, each with a <loc> and optionally <lastmod>, <changefreq> and <priority>. A <sitemapindex> contains <sitemap> entries, each with a <loc> pointing at another sitemap file and optionally its own <lastmod>.

The specification permits one level of nesting — an index may not contain another index — but real sites violate this often enough that a parser should recurse defensively rather than assume a depth of two. A recursion guard and a seen set cost nothing and prevent a self-referential index from looping forever.

Deciding which document you have means looking at the root tag, not at the filename. products.xml may be either; sitemap_index.xml occasionally turns out to be a plain urlset.

The Namespace That Makes Everything Return Nothing

Every sitemap declares xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" on its root element. That is a default namespace, which means it applies to the root and to every descendant that does not override it, and it applies whether or not any prefix appears in the document text.

To an XML parser, the element written as <loc> is really named {http://www.sitemaps.org/schemas/sitemap/0.9}loc. A query for loc asks for an element in no namespace, which does not exist in the document, so the result is an empty list. Nothing raises, nothing warns, and the scraper reports zero URLs against a file that visibly contains fifty thousand.

There are three ways to handle it, and they differ in how brittle they are.

Registering a prefix map is the standard approach and works in both xml.etree.ElementTree and lxml. Passing namespaces={"sm": SITEMAP_NS} and querying .//sm:loc is explicit and fast. Its weakness is that a handful of sites — usually ones generating XML with string templates — omit the xmlns entirely, and then the prefixed query returns nothing for the opposite reason.

Matching on the local name only, with .//{*}loc, is supported by ElementTree on Python 3.8+ and by lxml. It handles both the namespaced and the bare case, and it also survives a site that declares a slightly different URI, such as the old 0.84 schema still served by some legacy CMS installations. It is marginally slower and cannot distinguish sm:loc from a loc in a different vocabulary, which in practice never happens inside a sitemap.

Stripping namespaces from the tree before querying works, but it means walking every element to rewrite its tag, which on a fifty-thousand-URL document is real time spent for no benefit over {*}.

The examples below use {*} for robustness, since sitemap XML in the wild is not consistently well formed.

A Recursive Fetcher That Yields Every URL

The following is complete and runnable. It handles gzipped sitemaps, distinguishes an index from a urlset, recurses with a depth limit and a visited set, and yields each URL together with its parsed lastmod.

import gzip
import datetime as dt
from collections.abc import Iterator
from xml.etree import ElementTree as ET

import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
    "Accept": "application/xml,text/xml;q=0.9,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate",
}

def _decompress(body: bytes, url: str, content_type: str) -> bytes:
    """Handle .xml.gz served without Content-Encoding, which requests will not unwrap."""
    if body[:2] == b"\x1f\x8b":                     # gzip magic number
        return gzip.decompress(body)
    if url.endswith(".gz") or "gzip" in content_type:
        try:
            return gzip.decompress(body)
        except (OSError, gzip.BadGzipFile):
            return body
    return body

def _parse_lastmod(raw: str | None) -> dt.datetime | None:
    if not raw:
        return None
    text = raw.strip().replace("Z", "+00:00")
    try:
        parsed = dt.datetime.fromisoformat(text)
    except ValueError:
        return None
    if parsed.tzinfo is None:                        # a bare date, e.g. 2026-07-14
        parsed = parsed.replace(tzinfo=dt.timezone.utc)
    return parsed

def iter_sitemap(
    url: str,
    session: requests.Session | None = None,
    seen: set[str] | None = None,
    depth: int = 0,
    max_depth: int = 4,
) -> Iterator[tuple[str, dt.datetime | None]]:
    """Yield (loc, lastmod) for every URL reachable from a sitemap or sitemap index."""
    session = session or requests.Session()
    seen = seen if seen is not None else set()
    if url in seen or depth > max_depth:
        return
    seen.add(url)

    resp = session.get(url, headers=HEADERS, timeout=30)
    resp.raise_for_status()
    body = _decompress(resp.content, url, resp.headers.get("Content-Type", ""))

    try:
        root = ET.fromstring(body)
    except ET.ParseError as exc:
        print(f"skipping unparseable sitemap {url}: {exc}")
        return

    if root.tag.endswith("sitemapindex"):
        for entry in root.findall(".//{*}sitemap"):
            child = entry.findtext("{*}loc")
            if child:
                yield from iter_sitemap(child.strip(), session, seen, depth + 1, max_depth)
        return

    for entry in root.findall(".//{*}url"):
        loc = entry.findtext("{*}loc")
        if not loc:
            continue
        yield loc.strip(), _parse_lastmod(entry.findtext("{*}lastmod"))

if __name__ == "__main__":
    count = 0
    for loc, lastmod in iter_sitemap("https://www.python.org/sitemap.xml"):
        count += 1
        if count <= 5:
            print(f"{lastmod}  {loc}")
    print(f"{count} URLs in the frontier")

Three details deserve attention. The gzip check tests for the magic bytes first rather than trusting the extension, because a surprising number of servers send .xml.gz with Content-Encoding: gzip — in which case requests has already decompressed it and a second gzip.decompress would fail on plain XML. Checking the bytes covers both cases with no configuration.

_parse_lastmod accepts both forms the specification allows. lastmod is a W3C datetime, which means it may be a full timestamp with an offset or a bare YYYY-MM-DD. datetime.fromisoformat handles both on Python 3.11+, and normalising a naive value to UTC prevents a comparison against an aware timestamp from raising TypeError later.

Finally, the function is a generator. A sitemap index covering several million URLs would be a substantial list; yielding pairs means the consumer can filter, deduplicate, or enqueue them without the whole frontier existing at once.

Size Limits and How Big Sites Split Their Sitemaps

The specification caps a single sitemap file at 50,000 URLs and 50 MB uncompressed. Both limits are real and both are enforced by the search engines that consume them, so any site above that threshold splits its sitemap and publishes an index.

That split is informative. Sites usually partition by content type — sitemap-products-1.xml, sitemap-categories.xml, sitemap-blog.xml — which lets a scraper skip entire sections it does not want without fetching them. Where the split is purely numeric, the ordering is often insertion order, so the highest-numbered file holds the newest content.

An index file is itself capped at 50,000 entries, which puts a theoretical ceiling of 2.5 billion URLs on a two-level structure. Sites that exceed that nest indexes anyway, which is why the recursion guard above matters.

The gzip convention is worth understanding as a bandwidth question. Sitemap XML compresses to roughly 5% of its size because it is thousands of near-identical entries differing in a few characters. A 45 MB sitemap becomes about 2 MB gzipped, which is the difference between a fetch that is worth doing on every run and one that is not.

Incremental Crawling with lastmod

lastmod is the field that changes what a sitemap is for. Fetching the sitemap on every run and comparing each entry's timestamp against the time you last fetched that URL reduces a full recrawl to just the pages that changed.

import datetime as dt
import sqlite3

def changed_since_last_crawl(sitemap_url: str, db_path: str = "frontier.db") -> list[str]:
    conn = sqlite3.connect(db_path)
    conn.execute(
        "CREATE TABLE IF NOT EXISTS pages ("
        " url TEXT PRIMARY KEY, lastmod TEXT, fetched_at TEXT)"
    )
    known = {row[0]: row[1] for row in conn.execute("SELECT url, lastmod FROM pages")}

    todo: list[str] = []
    for loc, lastmod in iter_sitemap(sitemap_url):
        stamp = lastmod.isoformat() if lastmod else None
        previous = known.get(loc, "__absent__")
        if previous == "__absent__" or stamp is None or stamp > (previous or ""):
            todo.append(loc)
            conn.execute(
                "INSERT INTO pages (url, lastmod, fetched_at) VALUES (?, ?, ?) "
                "ON CONFLICT(url) DO UPDATE SET lastmod = excluded.lastmod, "
                "fetched_at = excluded.fetched_at",
                (loc, stamp, dt.datetime.now(dt.timezone.utc).isoformat()),
            )
    conn.commit()
    conn.close()
    return todo

fresh = changed_since_last_crawl("https://www.python.org/sitemap.xml")
print(f"{len(fresh)} pages to refetch")

Note the treatment of a missing lastmod: the URL goes into the refetch list. Absent means unknown, and treating unknown as unchanged means never fetching a page whose sitemap entry never carried a date.

The larger caveat is that lastmod is a claim, not a fact. Many content management systems regenerate their sitemaps nightly and stamp every entry with the generation time, which makes the entire frontier look freshly modified. Others never update it at all, so a page rewritten last week still shows a date from 2019. A cheap way to tell the difference is to look at the distribution of lastmod values across the file: a healthy sitemap has dates spread over months or years, while a useless one has fifty thousand entries sharing a single timestamp. Where lastmod cannot be trusted, the fallback is conditional requests — see Incremental Crawls with ETag and Last-Modified, which asks the server directly rather than believing the index.

Edge Cases and Caveats

• A sitemap is not a list of everything. It contains what the site chose to advertise, which typically excludes paginated listings, filtered views, and anything behind a login. Sitemaps and pagination crawling are complements, not substitutes.
• Entries may point at other hostnames. Cross-submission is legal when both hosts are verified together, so a sitemap on www.example.com may list URLs on shop.example.com. Filter by the hostnames you intend to crawl rather than trusting the file.
• Some sitemaps are plain text. The specification allows a .txt file with one URL per line and no XML at all. ET.fromstring raises ParseError on those; the fix is a fallback that splits on newlines when parsing fails and the body contains no <.
• Extension namespaces carry extra data. News, image, and video sitemaps add elements in their own namespaces — {...image/1.1}image and so on — which the {*}url query above ignores. If you want image URLs, query .//{*}image/{*}loc inside each entry.
• Very large sitemaps deserve iterparse. ET.fromstring builds the whole tree, which for a 50 MB document costs several hundred megabytes of Python objects. lxml.etree.iterparse(fh, tag="{*}url") with an element.clear() after each entry keeps memory flat at the price of a slightly less convenient loop.
• Deduplicate before enqueuing. Indexes overlap more often than they should, and the same URL can appear in several child sitemaps. A set is fine up to a few million URLs; beyond that see Deduplicating URLs with Bloom Filters.
• Scrapy has this built in. scrapy.spiders.SitemapSpider handles discovery, gzip, indexes and namespaces, and matches URLs to callbacks with sitemap_rules. Rolling your own is worth it outside a Scrapy project or when you need the lastmod values themselves, which the built-in spider does not expose.

Frequently Asked Questions

Why does my XPath return an empty list on a sitemap that clearly has URLs? Because of the default namespace on the root element. Query .//{*}loc to match on the local name regardless of namespace, or register a prefix map and query .//sm:loc. An unqualified .//loc matches elements in no namespace, which a valid sitemap never contains.

Should I trust lastmod for incremental crawling? Trust it after checking it. Look at the spread of values across one sitemap file: widely distributed dates suggest a generator that tracks real edits, while a single shared timestamp means the file is regenerated wholesale and the field carries no information. Where it is unreliable, fall back to conditional requests with If-Modified-Since.

How do I handle a sitemap with more than 50,000 URLs? You will not encounter one from a well-behaved site, because the limit is part of the specification and generators enforce it — you will encounter an index pointing at several files instead. If a non-conforming site does serve one, nothing breaks except memory, so switch to streaming with iterparse.

Is fetching a sitemap subject to robots.txt rules? The sitemap itself is normally allowed, and by convention robots.txt is where its location is published, so blocking it would be self-defeating. The URLs inside it are a different matter: an entry appearing in a sitemap does not override a Disallow rule covering that path, and a crawler should check each URL against the rules before requesting it.

Related

• Parsing JSON and XML API Responses in Python — the parent topic covering the parsers and namespace handling in general.
• Caching and Incremental Crawling — the wider strategy that a sitemap frontier feeds into.
• Incremental Crawls with ETag and Last-Modified — asking the server what changed when lastmod cannot be trusted.
• Web Scraping with Scrapy — the framework whose SitemapSpider automates most of this.