What Is a Spider? – Definition and How It Works
Learn what a spider is, how frontier management and politeness work, why deduplication matters, and how it differs from a scraper.
A spider is a crawler: a program that starts from seed URLs, fetches pages, extracts links, and follows them. The name comes from the web metaphor, and the terms spider and crawler are interchangeable.
1. The Loop
- Take a URL from the frontier — the queue of pending work.
- Fetch it, respecting politeness rules.
- Extract data, if that is the purpose.
- Extract links.
- Filter and normalise them.
- Add the new ones to the frontier.
- Repeat until the frontier is empty or a limit is reached.
Almost all of a spider's difficulty lives in steps five and six.
2. The Frontier
The queue is the spider's state, and three properties matter:
- Deduplication, so a URL is not fetched twice.
- Persistence, so an interrupted crawl resumes rather than restarting.
- Prioritisation, so valuable pages are reached before marginal ones.
A frontier held only in memory turns any interruption into a full restart, which at scale is expensive enough to make persistence non-optional.
3. URL Normalisation Prevents Explosion
The same page is reachable at many addresses. Without normalisation a spider treats each as new:
- Strip tracking and session parameters.
- Resolve relative links against the page's base.
- Lowercase the host, keep the path's case.
- Remove fragments.
- Decide a trailing-slash convention and apply it consistently.
- Prefer the canonical URL where the page declares one.
Skipping this is the most common reason a crawl never finishes.
4. Scope Rules
A spider needs boundaries: which hosts are in scope, which paths to avoid, how deep to go, and how many pages in total.
Without them, one outbound link leads to the rest of the internet. Explicit scope, plus robots.txt compliance, keeps a crawl bounded and polite.
5. Politeness Is Structural
Concurrency limits per host, spacing between requests to the same host, honouring Crawl-delay, and slowing when latency rises all belong in the spider rather than in individual handlers.
A crawler that discovers ten thousand URLs on one host and fetches them as fast as it can is the traffic shape that gets blocked, and rightly.
6. Spider Versus Scraper
| Aspect | Spider | Scraper |
|---|---|---|
| Purpose | Discover pages | Extract fields |
| Input | Seeds | A known target list |
| Output | An inventory, or data plus more URLs | Records |
| Ends when | The frontier empties | The list is done |
Most real systems do both: a spider builds the inventory and a scraper extracts from it. The clean separation is worth keeping, because discovery and extraction fail for different reasons.
7. Prefer Not to Crawl
Where a sitemap or feed exists it supplies the inventory in a few requests, without fetching pages purely to find links.
Crawling is the fallback for sites that publish no inventory — which makes reading robots.txt for sitemap declarations the first step of any spider worth writing.
8. Summary
A spider fetches pages, extracts links, and follows them from a persistent, deduplicated frontier within an explicit scope, respecting per-host politeness.
Normalisation and scope are where crawls succeed or spiral, persistence is what makes them resumable, and a published sitemap is usually a better source of inventory than crawling for it.