How to Parse HTML in Python – Libraries and Patterns

Learn how to parse HTML in Python: choosing a parser, selecting with CSS and XPath, handling broken markup, and extracting clean structured fields.

Parsing HTML means turning markup into a queryable tree, then addressing the parts you want. Python has several libraries for this, and the choice mostly comes down to speed, tolerance of broken markup, and whether XPath is needed.

1. Choosing a Parser

BeautifulSoup with lxml as its backend is a reasonable default: tolerant input handling with fast parsing.

Example

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")
for card in soup.select("article.product"):
    title = card.select_one("h3")
    price = card.select_one(".price")
    print(title.get_text(strip=True), price.get_text(strip=True))

2. Selecting Elements

Two addressing languages cover everything:

• CSS selectors — shorter, sufficient for class, id, attribute, and structural matching.
• XPath — needed when the anchor is text content, or when you must traverse upward to a parent.

A common pattern is CSS to reach a container, then a text-anchored XPath inside it for labelled values.

3. Getting Clean Text

Raw text nodes carry whitespace, non-breaking spaces, and line breaks from the markup's formatting. Normalise on extraction rather than later:

def clean(node):
    if node is None:
        return None
    return " ".join(node.get_text(" ", strip=True).split())

Collapsing runs of whitespace and stripping the result removes a whole class of downstream mismatch, particularly when values become dictionary keys or database columns.

4. Handling Missing Elements

Real pages are inconsistent: a field present on most cards is absent on some. Code that assumes presence raises AttributeError on the first exception.

Prefer select_one with a None check over indexing select(...)[0], and make missing values explicit as None rather than empty strings, so the difference between "absent" and "empty" survives into storage.

5. Prefer Structured Sources Inside the Page

Before writing selectors against rendered markup, check whether the page already carries structured data:

• A <script type="application/ld+json"> block with schema.org data.
• A state blob in a script tag that the page's own JavaScript reads.
• A JSON endpoint the page calls, visible in the network panel.

Parsing JSON is more stable than parsing markup, because layout changes far more often than data contracts.

6. Tables and Repeated Structures

For tabular markup, pandas.read_html converts tables to data frames in one call and handles most well-formed cases. For repeated card layouts, iterate the container elements and build one dictionary per record — never rely on flat document order across records, because a single missing field shifts everything after it.

7. Common Pitfalls

• Parsing before rendering. If content arrives via JavaScript, the fetched HTML will not contain it.
• Anchoring on hashed class names. They change on every deploy.
• Assuming encoding. Decode using the declared charset rather than the platform default.
• Regular expressions instead of a parser. Workable for one attribute, fragile for anything nested.

8. Summary

Parse HTML in Python by choosing a tolerant, fast parser, addressing elements with CSS selectors and XPath where text anchors are needed, and normalising text on the way out.

The largest gains come before selection: check for embedded JSON or an underlying endpoint first, because structured sources survive redesigns that break every selector you would otherwise write.

Potential Related Glossary

• Parsing
• Unstructured Data
• Data Wrangling
• Data Structure