What Is a Crawl Frontier? – Definition and How It Works

Informational P3 what is a crawl frontier

Learn what a crawl frontier is, how prioritisation and per-host queues work, and why deduplication belongs inside it.

The crawl frontier is the set of URLs a crawler has discovered but not yet fetched, together with the logic deciding what to fetch next. It is the component that determines what a crawl actually covers.

1. More Than a Queue

A plain queue answers only "what is next in order". A frontier answers harder questions:

Have we seen this URL before, in any equivalent form?
Is it in scope?
How important is it relative to everything else waiting?
When may we next request this host?
How many times has it failed, and should we retry?
Does it survive a restart?

Each of those is a real requirement, and a frontier is the place they live.

2. Deduplication Belongs Inside It

The frontier's most important job. Every page links to pages already discovered, so without deduplication the same URLs re-enter endlessly and the crawl never converges.

class Frontier:
    def __init__(self, store):
        self.seen = store          # persistent set of normalised URLs
    def add(self, url):
        key = normalize(url)       # scheme/host case, fragment, tracking params
        if key in self.seen:
            return False
        self.seen.add(key)
        self.queue_push(key)
        return True

Normalisation must happen before the check, or trivially different strings for one page all pass as new — which is the difference between a crawl that finishes and one that grows without bound.

3. Prioritisation

Order determines what a bounded crawl covers, so the policy matters:

Strategy	Behaviour
Breadth-first	Level by level; good coverage, the sensible default
Depth-first	Follows one path deep; risks traps
Priority by score	Fetch likely-valuable pages first
Sitemap-driven	Follow the site's own declared list
Freshness-driven	Refetch what changes most

Breadth-first is the right default because it reaches broad coverage early, so a crawl stopped at any point has a representative sample rather than one deep branch.

4. Per-Host Scheduling

Politeness cannot be a global delay, or a crawl either hammers one host or crawls a thousand hosts absurdly slowly.

The structure that solves it: one queue per host, plus a global scheduler that picks the next host whose delay has elapsed.

frontier
├── example.com     [url, url, url]   next allowed at T+1.0s
├── other.org       [url, url]        next allowed at T+0.2s
└── third.net       [url]             ready now

Concurrency then applies across hosts while each host sees a polite, spaced sequence. This is the single most important structural decision in a multi-host crawler.

5. Traps

A frontier must defend against structures that generate infinite URLs:

Calendars with endless next-month links.
Faceted search, where filter combinations multiply without limit.
Session identifiers in URLs, making every link unique.
Sort-order parameters producing the same content under many URLs.
Deep pagination past any useful depth.

Defences: a maximum depth, a cap on URLs per host, parameter allowlists, a limit on path segments, and pattern-based exclusions. Without them a crawl is consumed by one calendar.

6. Persistence

An in-memory frontier means a restart re-crawls everything, which is unacceptable for any crawl long enough to be interrupted.

A persistent frontier — a database table, Redis, or a purpose-built store — makes a crawl resumable and lets multiple workers share one view. What must survive: the seen set, the pending queue with priorities, per-host next-allowed times, and failure counts.

7. Failure Handling

Not every failure means the same thing:

Transient (timeout, 503) — requeue with backoff, bounded attempts.
Rate limited (429) — requeue and increase the host's delay.
Permanent (404) — retire after a few confirmations.
Terminal (410) — exclude permanently, and never re-add on rediscovery.

That last clause is the one implementations forget: without a permanent exclusion set, a link elsewhere re-adds a gone URL forever.

8. Summary

The crawl frontier holds discovered-but-unfetched URLs plus the policy for what to fetch next, handling deduplication, scope, priority, per-host timing, failures, and persistence.

Normalise before the seen-set check or the crawl never converges, use per-host queues so politeness and concurrency coexist, defend against infinite URL generators with depth and parameter limits, and persist everything so a restart resumes rather than restarts.