Step-by-Step Guide to Extracting Tables from HTML

An HTML table is the one structure on the web that already has the shape you want, which is why the extraction step in Understanding HTTP Requests and Responses is usually short — right up until a cell spans two rows.

If the <table> element is present in the bytes you fetched and its rows all have the same number of cells, pandas.read_html gets you a typed DataFrame in one call and you should use it. Write a manual parser only when the table uses colspan or rowspan, mixes header cells into the body, or carries footnote markers you need to strip. If the table is absent from response.text and only appears in the browser, stop parsing HTML entirely and call the JSON endpoint that populates it.

Deciding Which of the Three Methods Applies

The first check costs one command and saves an hour. Fetch the page and search the raw text for a distinctive cell value:

python -c "import requests, sys; r = requests.get(sys.argv[1], headers={'User-Agent': 'Mozilla/5.0'}, timeout=20); print('found' if 'Nectar' in r.text else 'missing')" https://books.toscrape.com/

If the value is missing, the table is rendered client-side and no HTML parser will ever see it. That case is covered in Finding Hidden API Endpoints in Network Traffic; the payload is almost always JSON, which is a better source than the rendered table anyway because it carries raw numeric types instead of formatted strings.

The second check is whether the grid is rectangular. Count cells per row and compare against the header count; any disagreement means spans, nested tables, or a section-divider row masquerading as data.

Isolating the Right Table Among Many

A content page routinely contains five to fifteen <table> elements once you include layout tables, infoboxes and footer grids, and index-based selection (tables[3]) breaks the first time an editor adds one above yours. Anchor on something the page's own authors control.

In order of durability: an id attribute, then a semantic class such as wikitable or data-table, then a caption or a preceding heading, and only as a last resort position. The match= parameter of read_html takes a regex tested against the table's text, which is often the sturdiest of all — it survives class renames because it keys on the content:

frames = pd.read_html(io.StringIO(response.text), match=r"Product Type")

When you need the ancestor-then-descendant form ("the table that follows the heading Specifications"), CSS cannot express it and you need the axes described in Selecting Elements with XPath and CSS Selectors:

from lxml import html as lxml_html

tree = lxml_html.fromstring(response.content)
table = tree.xpath('//h2[contains(., "Specifications")]/following::table[1]')[0]

Whatever anchor you pick, assert on the result before parsing it. A single line — assert list(frame.columns) == EXPECTED_COLUMNS — converts a silent schema change into an immediate, obvious failure instead of a month of subtly wrong rows.

How read_html Actually Works

pandas.read_html is a thin coordinator over a parsing backend. It selects lxml first, then falls back to html5lib with beautifulsoup4; you can force one with flavor="bs4". It returns a list of every table it could parse, in document order, so read_html(...)[0] is only correct if you have verified there is nothing above your target — navigation and layout tables count.

Three behaviours surprise people. It expands colspan and rowspan itself, filling repeated values across the grid, which means it handles the merged-cell case that manual find_all('td') loops get wrong. It infers dtypes per column, so a column of 1,234 strings with thousands separators stays object rather than becoming int64. And since pandas 2.1 passing a URL string directly is deprecated in favour of passing markup or a file-like object — you now fetch the page yourself and hand over the text, which is better practice regardless because it lets you set headers and a timeout.

import io

import pandas as pd
import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}


def fetch_tables(url: str) -> list[pd.DataFrame]:
    response = requests.get(url, headers=HEADERS, timeout=20)
    response.raise_for_status()
    return pd.read_html(io.StringIO(response.text))


if __name__ == "__main__":
    tables = fetch_tables("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html")
    print(f"{len(tables)} table(s) found")
    product = tables[0]
    product.columns = ["field", "value"]
    print(product.to_string(index=False))

Wrapping the text in io.StringIO silences the deprecation warning and works across pandas 1.5 through 3.x. If read_html finds nothing it raises ValueError: No tables found, which is the exception to catch — it means either a client-rendered table or an anti-bot page returned in place of the content.

Expanding colspan and rowspan by Hand

When you do need row-by-row control, the mechanism to implement is the same one a browser uses: maintain a set of pending row-spans and consume grid positions rather than list positions.

A rowspan="2" cell appears exactly once in the markup, on the first of the two rows. The second row's <td> list is one item short, and a naive zip(headers, cells) therefore shifts every value one column to the left — silently, with no exception, producing a DataFrame that looks plausible and is wrong. The fix is to keep a dictionary of column index to (value, rows_remaining) and fill from it before consuming the next real cell.

from bs4 import BeautifulSoup, Tag


def table_to_grid(table: Tag) -> list[list[str]]:
    """Expand colspan and rowspan into a rectangular list of rows."""
    grid: list[list[str]] = []
    pending: dict[int, tuple[str, int]] = {}

    for tr in table.find_all("tr"):
        row: list[str] = []
        col = 0
        cells = tr.find_all(["td", "th"])
        cursor = 0

        while cursor < len(cells) or col in pending:
            if col in pending:
                value, remaining = pending[col]
                row.append(value)
                if remaining > 1:
                    pending[col] = (value, remaining - 1)
                else:
                    del pending[col]
                col += 1
                continue

            cell = cells[cursor]
            cursor += 1
            text = cell.get_text(" ", strip=True).replace("\xa0", " ")
            colspan = int(cell.get("colspan", 1))
            rowspan = int(cell.get("rowspan", 1))

            for offset in range(colspan):
                row.append(text)
                if rowspan > 1:
                    pending[col + offset] = (text, rowspan - 1)
            col += colspan

        if row:
            grid.append(row)
    return grid


HTML = """
<table>
  <tr><th>Region</th><th>Q1</th><th>Q2</th></tr>
  <tr><td rowspan="2">EU</td><td>12</td><td>15</td></tr>
  <tr><td>18</td><td>21</td></tr>
  <tr><td>US</td><td>9</td><td>11</td></tr>
</table>
"""

if __name__ == "__main__":
    soup = BeautifulSoup(HTML, "lxml")
    for line in table_to_grid(soup.find("table")):
        print(line)

The output is four rows of three cells each, with EU repeated. Feed grid[1:] to pd.DataFrame(..., columns=grid[0]) and you have the same result read_html would produce, but with a hook where you can strip footnote markers, parse currency, or drop divider rows before the DataFrame exists.

Two details in that code matter. get_text(" ", strip=True) joins nested elements with a space instead of concatenating them, so <td><b>12</b><sup>a</sup></td> becomes 12 a rather than 12a. And the explicit \xa0 replacement handles the non-breaking spaces that appear in almost every hand-authored table; strip=True does not remove them because U+00A0 is not ASCII whitespace.

Cleaning the Grid Before It Becomes a DataFrame

Header text is the usual source of duplicate column names — read_html will happily hand back two columns called Notes, and any later df["Notes"] returns a DataFrame rather than a Series. Deduplicate explicitly, and normalise the numeric columns while the values are still strings:

import re

import pandas as pd


def tidy(rows: list[list[str]]) -> pd.DataFrame:
    header = [h.strip() or f"col_{i}" for i, h in enumerate(rows[0])]
    seen: dict[str, int] = {}
    columns: list[str] = []
    for name in header:
        seen[name] = seen.get(name, 0) + 1
        columns.append(name if seen[name] == 1 else f"{name}_{seen[name]}")

    frame = pd.DataFrame(rows[1:], columns=columns)
    for column in frame.columns:
        cleaned = frame[column].str.replace(r"[^\d.\-]", "", regex=True)
        numeric = pd.to_numeric(cleaned, errors="coerce")
        if numeric.notna().mean() > 0.9:
            frame[column] = numeric
    return frame

The notna().mean() > 0.9 test converts a column only when nearly every value parses as a number, which keeps a Region column of country names as text while turning £53.74 into 53.74. Anything more aggressive silently destroys data. Broader normalisation rules — currencies, dates, unit suffixes — belong in Normalizing Prices, Dates and Units.

For exporting the finished frame, to_csv is fine for a few thousand rows; past that, the type-preserving options in Exporting Scraped Data to CSV and Parquet avoid the round-trip where every column comes back as a string.

On cost: parsing dominates only for large tables. On one laptop run against a synthetic 5,000-row, 8-column table, read_html with the lxml flavour finished in roughly 0.4 s while the bs4/html5lib flavour took about 4 s, and the manual table_to_grid function above sat between them at around 1 s because it builds the same BeautifulSoup tree but skips type inference. Treat those as indicative of the ratio on that setup rather than as portable figures — the shape of the table and the parser version both move them. What does not change is the ordering: lxml fastest, html5lib slowest by roughly an order of magnitude, and the network fetch usually costing more than any of them.

Edge Cases and Caveats

• Nested tables. soup.find("table").find_all("tr") descends into inner tables, mixing their rows into the outer grid. Restrict the search with table.find_all("tr", recursive=False) on the <tbody>, or select rows whose nearest table ancestor is the one you targeted.
• Headers not in <thead>. Many hand-written tables put the header in the first <tr> of <tbody> using <th>. Detect it by checking whether the first row is entirely <th> rather than trusting <thead> to exist.
• <th> inside body rows. Row-header tables use <th scope="row"> for the first cell of every row. A parser that collects only <td> drops that column entirely; collect ["td", "th"] as the code above does.
• Sticky or virtualised tables. Some data grids render only the visible rows into the DOM and swap them on scroll. The HTML then contains 30 rows out of 10,000 no matter how long you wait — the endpoint is the only complete source.
• colspan="0". Legal in HTML 4, meaning "to the end of the column group". int(cell.get("colspan", 1)) yields 0 and the cell vanishes. Clamp with max(1, int(...)) if you scrape older documents.
• Numbers formatted for humans. 1,234, (1,234) for negatives, 1.234,56 in European locales, and trailing footnote letters all defeat naive float(). Decide the locale before parsing, not after.
• Very large tables. A 50,000-row table builds a BeautifulSoup tree of several hundred thousand objects; expect a few hundred megabytes of resident memory. Parsing with lxml directly, or streaming with lxml.etree.iterparse, keeps that bounded.

Frequently Asked Questions

Should I use pandas.read_html or BeautifulSoup for tables? Use read_html whenever it works, because it already expands merged cells and infers column types, and reach for BeautifulSoup when you need to intervene between the markup and the frame — stripping footnote markers, dropping section-divider rows, or keeping a cell's href alongside its text. read_html discards attributes entirely, so any table where the link matters requires a manual pass.

Why does read_html raise ValueError: No tables found on a page with a visible table? Either the table is injected by JavaScript after page load, so it never appears in the HTML you fetched, or the server returned a challenge or error page instead of the content. Print the first 500 characters of response.text to tell the two apart; if the markup is real but the table is absent, the data is coming from an XHR call you can request directly.

How do I keep the links inside table cells? Parse with BeautifulSoup and build the row dictionaries yourself, reading both cell.get_text(strip=True) and cell.find("a")["href"] into separate keys. Resolve the href against the page URL with urllib.parse.urljoin before storing it, because table links are very often relative.

Can I extract a table that is split across paginated pages? Yes — fetch each page, convert each one to a DataFrame, and concatenate with pd.concat(frames, ignore_index=True). Verify that every page's column list is identical first, because a site that changes column order between pages produces a silently misaligned result. The traversal itself is covered in Handling Pagination and Infinite Scroll.

Related

• Understanding HTTP Requests and Responses — the parent topic, covering how to fetch the page the table lives on.
• Parsing HTML with BeautifulSoup — the traversal API behind the manual parser above.
• Selecting Elements with XPath and CSS Selectors — pinning down one table among many.
• Saving Scraped Data to PostgreSQL — writing the finished frame to a database.