What Is lxml? – Definition and How It Works

Learn what lxml is, why its C-backed parser and full XPath support make it the fast choice, and its trade-offs.

lxml is a Python library binding libxml2 and libxslt. It parses HTML and XML at C speed and supports full XPath 1.0, XSLT, and schema validation — capabilities pure-Python parsers do not have.

1. Two Things It Gives You

Speed. Parsing happens in C, so it is substantially faster than pure-Python parsers on large documents. On a pipeline processing many thousands of pages, parsing stops being a bottleneck.

XPath. Complete XPath 1.0, including axes, functions, and predicates — which CSS selectors cannot express.

from lxml import html

doc = html.fromstring(page_html)

# CSS could do this
titles = doc.cssselect("h2.title")

# CSS cannot: select a parent from a child's text
rows = doc.xpath('//td[contains(text(), "Price")]/parent::tr')

# Nor this: the following sibling of a specific label
value = doc.xpath('//dt[text()="SKU"]/following-sibling::dd[1]/text()')

The last two patterns are the reason lxml keeps its place. Extracting a value from a label-value layout is natural in XPath and awkward or impossible in CSS.

2. HTML Versus XML Mode

from lxml import html, etree

doc  = html.fromstring(broken_html)      # lenient, HTML rules
tree = etree.fromstring(strict_xml)      # strict, raises on malformed input

Use lxml.html for web pages, always. etree in XML mode rejects the unclosed tags and misnested elements that real HTML contains constantly.

For XML feeds, strict mode is correct — a malformed feed should raise rather than silently parse to something wrong.

3. Both Selector Languages

cssselect translates CSS to XPath internally, so both are available and can be mixed. CSS is more readable for straightforward descent; XPath handles the rest.

Practical rule: reach for CSS first, and switch to XPath when you need a parent axis, a sibling relationship, a text predicate, or a positional constraint CSS cannot express.

4. Streaming Large Documents

For documents too large to hold in memory, iterparse processes incrementally:

from lxml import etree

for event, elem in etree.iterparse("feed.xml", events=("end",), tag="item"):
    handle(elem.findtext("title"), elem.findtext("link"))
    elem.clear()                                  # free this element
    while elem.getprevious() is not None:
        del elem.getparent()[0]                   # free earlier siblings

The clearing is not optional. Without it, iterparse retains the whole tree and uses as much memory as a full parse — which defeats the purpose and is the classic mistake with this API.

5. Compared With Other Parsers

BeautifulSoup with lxml as its backend is a common middle ground: lxml's parsing speed with BeautifulSoup's friendlier traversal API, at the cost of an object-construction overhead lxml alone avoids.

6. Installation Reality

lxml is a compiled extension. Wheels exist for common platforms, so pip install lxml usually just works — but on an unusual platform or an older base image it falls back to building, and then needs libxml2 and libxslt development headers.

This is the one genuine friction point, and it is why a pure-Python parser is sometimes chosen for a constrained deployment.

7. Practical Notes

• XPath returns a list, always. Index it, or use a helper that handles the empty case — an unchecked [0] on a missing element is the most common lxml crash.
• text() returns text nodes, not descendant text. Use string() or .text_content() when you want everything inside an element.
• Namespaces matter in XML. Pass a namespace map; a namespaced document will not match unprefixed paths.
• html.fromstring on an empty string raises. Guard it, since an empty response is a normal failure mode.
• Compile repeated XPath expressions with etree.XPath() when running the same query over many documents.

8. Summary

lxml wraps libxml2 to give Python fast C-backed parsing plus full XPath, CSS selectors, XSLT, and schema validation, with a lenient HTML mode for real web pages and a strict XML mode for feeds.

Its advantages are speed and XPath's ability to express parent, sibling, and text-predicate relationships CSS cannot — with a compiled installation as the main cost, and iterparse requiring explicit element clearing to actually save memory.

Potential Related Glossary

• Parsing
• Unstructured Data
• Web Crawling
• Data Structure