What Is a Sitemap? – Definition and How It Works
Learn what an XML sitemap contains, how sitemap indexes work, why it is the best discovery route, and how much to trust its metadata.
A sitemap is an XML file listing a site's URLs, published so that crawlers can discover pages without following every link. For anyone collecting data it is usually the cheapest and most complete route to a site's inventory.
1. The Format
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/product/4192</loc>
<lastmod>2026-08-12</lastmod>
<changefreq>weekly</changefreq>
<priority>0.8</priority>
</url>
</urlset>
Only <loc> is required. The other three are hints, and their reliability varies considerably.
2. Sitemap Indexes
A single sitemap is limited — conventionally fifty thousand URLs or fifty megabytes uncompressed — so large sites publish an index pointing at several files:
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://example.com/sitemap-products-1.xml.gz</loc></sitemap>
<sitemap><loc>https://example.com/sitemap-articles.xml.gz</loc></sitemap>
</sitemapindex>
Files are frequently gzipped, and a fetcher must handle that.
3. Finding Them
robots.txtusually declares them withSitemap:lines./sitemap.xmlis the conventional location.- Search Console-style tools list submitted sitemaps for a site you own.
Reading robots.txt first is the reliable route, since the declared location need not be conventional.
4. Why It Beats Crawling for Discovery
| Approach | Requests | Completeness |
|---|---|---|
| Sitemap | One index plus a few files | Whatever the site chose to list |
| Link crawling | One per page, plus discovery overhead | Only what is reachable by links |
A sitemap can yield tens of thousands of URLs from a handful of requests — orders of magnitude less load on the site than crawling for the same inventory.
5. How Much to Trust the Metadata
lastmod is the most useful field and is only sometimes accurate. Where a site maintains it honestly, it enables incremental collection: fetch only what changed since last time.
changefreq and priority are hints that major crawlers largely ignore, and they are frequently generated with static values across every URL.
Verify lastmod against a few known-changed pages before building an incremental strategy on it.
6. What Sitemaps Omit
They list what the site wants discovered — not necessarily everything that exists. Pages behind authentication, deliberately unlisted pages, and paginated views are commonly absent.
A sitemap is therefore a strong starting inventory rather than a guaranteed complete one.
7. Specialised Variants
Image, video, and news sitemaps carry additional metadata for those media types. News sitemaps in particular are limited to recent articles, which makes them an efficient feed for time-sensitive collection.
8. Summary
A sitemap lists a site's URLs with optional metadata, split across files under an index for large sites and usually declared in robots.txt.
It is the most efficient and most polite discovery route available, lastmod is the field worth using where it is honest, and the listing reflects what the site chose to publish rather than everything that exists.