A single XML sitemap can hold at most 50,000 URLs or 50 MB uncompressed — once you cross that, you split your URLs across multiple sitemap files and tie them together with a sitemap index. A sitemap index is a small master file that lists your other sitemaps instead of listing pages directly. For large sites, it is the difference between an organised, monitorable crawl map and one giant file that is hard to maintain and impossible to diagnose. This guide shows how to structure it.
What a sitemap index actually is
Think of it as a table of contents for your sitemaps. Where a normal sitemap contains <url> entries, a sitemap index contains <sitemap> entries, each pointing to the location of a child sitemap and its last-modified date. You submit the single index file to Google Search Console and Bing, and the crawlers follow it out to every child sitemap automatically.
The structure is minimal — a <sitemapindex> root wrapping one <sitemap> block per child file, each with a <loc> and an optional <lastmod>. You can nest one level deep: an index points to sitemaps, and those sitemaps point to URLs. You cannot point an index at another index. If you have not built a standard sitemap yet, start with our complete XML sitemap guide first, then come back to organise them.
How to split your URLs across child sitemaps
You could split arbitrarily by count, but the smart move is to split by logical section, because it turns your sitemaps into a diagnostic tool. Group them the way your site is organised:
- By content type — one sitemap for products, one for blog posts, one for category pages, one for static pages.
- By section or brand — separate sitemaps per major directory or sub-site.
- By recency — a rolling "new and recently updated" sitemap so fresh URLs are easy for crawlers to find fast.
The payoff shows up in Search Console. Because it reports indexing stats per submitted sitemap, splitting by section lets you see that, say, 98% of your blog URLs are indexed but only 40% of your product URLs are. That immediately tells you where the problem lives. One monolithic sitemap hides that signal completely.
Keeping the index accurate and lean
A stale sitemap is worse than no sitemap, because it wastes crawl attention on URLs that no longer matter. A few rules keep yours trustworthy:
- Only include indexable, canonical URLs. No redirects, no 404s, no noindex pages, no non-canonical duplicates. Every URL in a sitemap is a recommendation to crawl and index — make sure you mean it. Our guide to canonical URLs and duplicate content explains which version belongs there.
- Keep lastmod honest. Set it to the real date the page's content meaningfully changed, not the date you regenerated the file. Crawlers learn to distrust lastmod values that update on every page every day, and once they distrust it they ignore it.
- Compress with gzip. Large sitemaps benefit from .xml.gz compression, which cuts transfer size while staying within the 50 MB uncompressed limit.
- Reference the index in robots.txt. Add a Sitemap: line pointing to your index file so any crawler discovers it, as covered in our complete robots.txt guide.
Why this matters for crawl efficiency and AI
On a large site, crawlers do not visit every page every day — they budget their attention. A clean, sectioned sitemap index tells them exactly which URLs exist and which changed recently, so they spend that budget on the right pages instead of rediscovering your site through internal links alone. This works hand in hand with the pruning strategy in our crawl budget optimization guide.
The same map helps AI systems and instant-indexing pipelines. When you ping lastmod changes or use a protocol like IndexNow, an accurate sitemap backs up those signals so bots can confirm what is new. A well-organised index is not busywork — it is the most direct, machine-readable statement of what your site contains and when it last changed.
Not sure whether your sitemap is clean, current, and actually being found by crawlers? Run the free CheckMy.site scanner to check how bots discover and read your site's structure.