What Is a Robots Meta Tag? – Definition and How It Works

Learn how the robots meta tag and X-Robots-Tag differ from robots.txt, and why noindex needs the page to be crawlable.

The robots meta tag is a page-level instruction to crawlers, placed in the HTML head, controlling indexing and link following for that specific page.

<meta name="robots" content="noindex, nofollow">

1. The Directives

A crawler-specific form targets one agent:

<meta name="googlebot" content="noindex">

2. The X-Robots-Tag Header

The same directives can be sent as an HTTP header, which is how non-HTML resources are controlled:

X-Robots-Tag: noindex, nofollow
X-Robots-Tag: googlebot: noindex

This matters because a PDF, an image, or a JSON file has no HTML head. A crawler that only reads meta tags misses directives on every non-HTML resource — a real gap in naive implementations.

3. Robots.txt Versus Robots Meta Tag

The distinction that causes the most confusion:

robots.txt controls fetching. The meta tag controls what happens to what was fetched.

4. The Consequence Everyone Gets Wrong

Because the meta tag is inside the page, a crawler must fetch the page to see it. So:

robots.txt:  Disallow: /private/
page:        <meta name="robots" content="noindex">

These conflict. The crawler never fetches the page, never sees the noindex, and — if the URL is linked elsewhere — may index the URL anyway without its content.

The correct pattern for keeping something out of an index is to allow crawling and serve noindex. Blocking in robots.txt prevents the directive from ever being read.

5. What This Means for Your Own Crawler

Honouring these directives is part of behaving well, and it is straightforward:

def indexing_allowed(response, html) -> bool:
    header = response.headers.get("X-Robots-Tag", "")
    directives = {d.strip().lower() for d in header.split(",") if d}
    tag = parse_meta_robots(html)          # meta name=robots, and agent-specific
    directives |= tag
    return not ({"noindex", "none"} & directives)

def follow_links(directives) -> bool:
    return not ({"nofollow", "none"} & directives)

Two points worth getting right: read both the header and the meta tag, and respect nofollow in the frontier rather than only recording it.

6. nofollow on Individual Links

Distinct from the page-level directive, individual links carry their own attribute:

<a href="/x" rel="nofollow">…</a>
<a href="/y" rel="ugc">…</a>
<a href="/z" rel="sponsored">…</a>

ugc marks user-generated content and sponsored marks paid links. For a crawler these are signals about link trustworthiness, and for anyone measuring a site's outbound linking they are the distinction between an editorial link and a paid one.

7. It Is Not an Access Control

Important limit: these directives are requests to well-behaved crawlers. They do not restrict access, do not require authentication, and do not prevent anyone from reading the page.

Anything genuinely sensitive needs authentication, not a noindex. A page marked noindex is still public.

8. Summary

The robots meta tag gives page-level indexing directives in the HTML head, with X-Robots-Tag providing the same control as an HTTP header for non-HTML resources.

It controls indexing rather than crawling, which is why blocking a URL in robots.txt prevents its noindex from ever being read — the correct pattern is to allow crawling and serve noindex — and a compliant crawler must read both the header and the tag while treating neither as an access control.

Potential Related Glossary

• Web Crawling
• Parsing
• Data Structure
• Unstructured Data