Crawling vs. Indexing – Definition and Differences

Learn how crawling differs from indexing, why a crawled page may not be indexed, and which directive controls each stage.

Crawling is fetching a page. Indexing is deciding to store and serve it. They are separate stages with separate controls, and confusing them produces directives that do the opposite of what was intended.

1. The Two Stages

A page can be crawled and not indexed. It can also be indexed without being crawled, if enough external signals describe it.

2. Why a Crawled Page May Not Be Indexed

• Thin or duplicate content, where another page already serves the purpose.
• A canonical pointing elsewhere.
• A noindex directive.
• Quality assessment, which is a judgement rather than a rule.
• Crawl-versus-value economics on very large sites.

Being fetched is a prerequisite, not an entitlement.

3. The Controls Are Different

# robots.txt — controls crawling
User-agent: *
Disallow: /internal/

<!-- page-level — controls indexing -->
<meta name="robots" content="noindex, follow">

robots.txt says do not fetch. noindex says fetch but do not include.

4. The Classic Mistake

Blocking a page in robots.txt to keep it out of search does not reliably work. If the page cannot be fetched, the noindex directive inside it cannot be read — so the page may still be indexed from external signals, with no description available.

To remove a page from an index, allow crawling and serve noindex. Once it has been dropped, blocking crawling is optional.

5. Indexing Without Crawling

A page blocked from crawling can still appear in results, listed from anchor text and other external references, because the engine knows it exists without having read it.

That is why blocking is a poor privacy mechanism: it hides the content from the crawler while leaving the address discoverable.

6. What This Means for Collection

The same distinction applies to your own work. Fetching a page is one decision; storing and republishing its content is another, governed by copyright and terms rather than by whether the fetch succeeded.

Being able to crawl something does not settle whether you may index it — and that is the same asymmetry search engines navigate.

7. Crawl Budget Interacts With Both

Large sites have a finite share of a crawler's attention. Requests spent on parameterised duplicates, infinite filter combinations, and low-value pages are requests not spent on pages that matter.

Canonical tags, sensible internal linking, and blocking genuinely worthless paths all direct that attention — which is a crawling optimisation with an indexing benefit.

8. Summary

Crawling fetches; indexing decides inclusion. robots.txt controls the first, noindex the second, and using the first to achieve the second does not reliably work.

The same separation applies to any collection: obtaining a page and being permitted to store or republish it are different questions with different answers.

Potential Related Glossary

• Web Crawling
• Parsing
• Unstructured Data
• Data Structure