What Is Incremental Crawling? – Definition and How It Works

Learn what incremental crawling is, how change detection and conditional requests work, and how to schedule refetches.

Incremental crawling refetches only what has changed or is likely to have changed, instead of recrawling everything. It is what makes ongoing collection affordable.

1. Why Full Recrawls Do Not Scale

A full recrawl costs the same every time regardless of how little changed. On a large site where a small fraction of pages change between runs, most of the work is wasted — and that waste is load imposed on the target as well as cost to you.

Incremental crawling shifts the question from "fetch everything" to "what is worth fetching now".

2. Conditional Requests Come First

The cheapest change detection is asking the server, using values it gave you:

headers = {}
if stored.etag:
    headers["If-None-Match"] = stored.etag
if stored.last_modified:
    headers["If-Modified-Since"] = stored.last_modified

r = session.get(url, headers=headers, timeout=(10, 60))
if r.status_code == 304:
    mark_unchanged(url)          # no body transferred at all
else:
    process(r)
    store_validators(url, r.headers.get("ETag"), r.headers.get("Last-Modified"))

A 304 costs a request but no body, which is a large saving on any sizeable page. This should be the default behaviour of any refetch, and it is frequently omitted.

3. When the Server Will Not Help

Many pages return no ETag and no useful Last-Modified, so change must be detected after fetching:

Full-body hash — simple, and over-sensitive. A timestamp, a rotating advert, or a "viewed 412 times" counter makes every fetch look changed.

Content hash after extraction — hash the extracted fields rather than the raw page. This is the useful one: it ignores presentational churn and detects only changes in the data you care about.

Field-level comparison — compare each extracted value, so you know what changed, not merely that something did.

Hashing extracted content rather than raw HTML is the single decision that makes change detection meaningful.

4. Deciding What to Refetch

A site's own sitemap or feed is the best available input where it exists, and checking it is far cheaper than probing pages.

5. Adaptive Scheduling

The effective policy learns from each URL's history: pages observed to change often are checked more often, and pages that never change are checked progressively less.

def next_interval(prev_interval, changed, lo=3600, hi=30*86400):
    return max(lo, prev_interval // 2) if changed else min(hi, prev_interval * 2)

Halving on change and doubling on no-change converges quickly on each page's actual rate, and concentrates the crawl budget where change actually happens.

The bounds matter: without a ceiling a stable page eventually stops being checked at all, and without a floor a volatile page is hammered.

6. Detecting Deletions

Incremental crawling naturally sees changes and easily misses removals. A URL no longer linked anywhere is simply never revisited, so the record persists as though it were current.

Two mechanisms address it: periodically revisit known URLs even when nothing suggests a change, and reconcile against the site's sitemap or listing pages, treating a disappearance there as a signal to verify.

Without this, a dataset accumulates items that no longer exist — and confidently reports them as live.

7. What to Store Per URL

• Last fetch time, and last change time.
• ETag and Last-Modified.
• Content hash of the extracted fields.
• Observed change count and current interval.
• Consecutive failure count.
• Status of the last fetch.

Both timestamps are needed: last-fetched drives scheduling, last-changed drives the interval, and conflating them makes adaptive scheduling impossible.

8. Summary

Incremental crawling refetches only what is likely to have changed, using conditional requests so an unchanged page costs no body transfer, and content hashing of extracted fields rather than raw HTML so presentational churn is ignored.

Drive scheduling from sitemaps, feeds, and each URL's observed change rate with bounded adaptive intervals — and add periodic full revisits, because incremental crawling sees changes but silently misses deletions.

Potential Related Glossary

• Web Crawling
• Parsing
• Data Wrangling
• Data Structure