What Is Boilerplate Removal? – Definition and How It Works

Informational P3 what is boilerplate removal

Learn what boilerplate removal is, which algorithms work, and why it decides the quality of any text corpus.

Boilerplate removal separates a page's main content from everything surrounding it — navigation, headers, footers, sidebars, cookie notices, related-article rails, comment chrome, advertising.

1. How Much Is Boilerplate

On a typical content page, the main article is a minority of the extracted text. Navigation menus, footer link lists, and recommendation modules frequently exceed the article in word count.

That ratio is why removal matters so much: a naive tag-strip produces a document in which the actual content is outnumbered by furniture, and every downstream use — search, summarisation, training, similarity — degrades accordingly.

2. Approaches

Approach	How	Trade-off
Site-specific selectors	Hand-written per site	Accurate, breaks on redesign, unscalable
Structural heuristics	Text density, link ratios	Generalises, imperfect
Readability algorithms	Score elements, pick the best subtree	Good default
Template detection	Compare pages from one site	Very accurate, needs several pages
Machine-learned models	Trained classifiers	Strong, needs training data

3. Structural Heuristics

The signals that generalise across sites:

Text density — characters of text divided by markup bytes. Content is dense; navigation is not.
Link density — links divided by words. A navigation block is mostly links; a paragraph is mostly words.
Block length. Long blocks are content; short ones are labels and menu items.
Tag context. Text in nav, footer, aside, and header is structural by definition.
Position in document order, since content usually sits between chrome.
Repetition across pages from the same site — the strongest signal of all.

Link density is the single most useful discriminator, and it explains why footer link lists are removed reliably while a short standalone paragraph sometimes is not.

4. Template Detection Is the Strongest Method

Given several pages from one site, whatever text appears identically on all of them is the template.

from collections import Counter

def template_blocks(pages, threshold=0.8):
    counts = Counter()
    for blocks in pages:
        for b in set(blocks):
            counts[b] += 1
    n = len(pages)
    return {b for b, c in counts.items() if c / n >= threshold}

Highly accurate, because it uses evidence rather than heuristics — and it requires a corpus per site, which is exactly what a crawler already has. For any site being collected repeatedly, this is the method to use.

5. Using a Library

import trafilatura

downloaded = trafilatura.fetch_url(url)
text = trafilatura.extract(downloaded, include_comments=False,
                           include_tables=True, favor_precision=True)

favor_precision prefers dropping uncertain blocks over including them, which is usually right for a corpus — a missing paragraph is less damaging than a navigation menu embedded mid-article.

Where recall matters more, the opposite setting exists. Choosing between them is a decision about which error costs you more.

6. What Goes Wrong

Over-removal. A short article, a table-based layout, or a list-heavy page can be misread as navigation and largely discarded. The failure is silent — a page extracts to two sentences and nothing reports a problem.

Under-removal. A sidebar with prose in it survives, contaminating the content.

Structural loss. Aggressive removal can flatten headings, lists, and tables, destroying information the text alone does not carry.

Content in unexpected places. Product descriptions in tabs, specifications in tables, key facts in a sidebar — all legitimately content, all structurally similar to chrome.

7. Validating the Result

Because the failure mode is silent, checks are necessary:

Extracted length against page length. A very low ratio indicates over-removal.
A minimum length threshold per document type.
Presence of expected markers — the title should appear in the content.
Sampled manual review, which remains the only way to catch subtle errors.
Distribution monitoring. A sudden shift in extracted length across a site means its template changed.

That last one is the practical alarm: extraction quality degrades gradually and invisibly otherwise.

8. Summary

Boilerplate removal isolates a page's main content from navigation, footers, sidebars, and other furniture that typically outweighs the article in word count.

Text and link density generalise across sites, template detection across several pages from one site is the most accurate method available, and the characteristic failure is silent over-removal — so monitor extracted-length distributions and treat a sudden shift as a template change.