What Is Indexing? – Definition and How It Works
Learn what it means for a page to be indexed, which directives control it, why crawled pages are excluded, and how to diagnose absence.
Indexing is a search engine storing a page and making it eligible to appear in results. It is a separate decision from crawling, and the distinction resolves most confusion about why a page is missing.
1. Crawled Is Not Indexed
Crawling fetches a page. Indexing decides to keep it and serve it.
A page can be crawled repeatedly and never indexed, because inclusion is a judgement about value rather than an automatic consequence of being readable.
2. What Controls It
<meta name="robots" content="noindex, follow">
Or as a header, for non-HTML resources:
X-Robots-Tag: noindex
Note that robots.txt does not control indexing. It controls fetching — and a page blocked there cannot have its noindex read, which is why blocking is the wrong tool for removal.
3. Why Crawled Pages Are Excluded
| Reason | Meaning |
|---|---|
noindex present |
Explicitly requested |
| Canonical points elsewhere | The site nominated a different version |
| Duplicate content | Another page already serves the purpose |
| Thin content | Not enough substance to be worth serving |
| Soft 404 | Looks like an error page |
| Crawl economics | The site's budget did not stretch to it |
Most of these are judgements, and the fix is content or structure rather than a directive.
4. Removing a Page Correctly
- Allow crawling — the directive must be readable.
- Serve
noindexon the page. - Wait for it to be recrawled and dropped.
- Only then, block crawling if desired.
Reversing steps one and four is the classic error and leaves the page indexed with no description.
5. Getting a Page Indexed
- Ensure it is linked internally, and listed in a sitemap.
- Confirm nothing blocks it in
robots.txt. - Confirm no
noindexis present, including from a template. - Confirm the canonical points at itself.
- Make sure the content is substantive and distinct.
- Verify it renders server-side, or that the rendered content is available.
6. Diagnosing Absence
Check in order: is it blocked from crawling, is it excluded by directive, is the canonical pointing away, is it a near-duplicate, is it thin, and has enough time passed.
That sequence separates technical exclusions — which are quick fixes — from judgements, which are not.
7. Indexed Is Not Ranking
An indexed page is a candidate for queries. Whether it appears depends on relevance and competition per query.
"Indexed but no traffic" is therefore not an indexing problem, and treating it as one leads to changes that cannot help.
8. Summary
Indexing is inclusion in the searchable store, controlled by page-level directives rather than by robots.txt, and withheld from crawled pages that are duplicated, thin, canonicalised away, or explicitly excluded.
Removal requires allowing the crawl so noindex can be read. Inclusion requires being linked, permitted, distinct, substantive, and readable — and inclusion is still not the same as ranking.