On a small site the sitemap is an afterthought that works fine by accident. On a large site it is infrastructure, and when it is wrong it quietly wastes the crawl budget you depend on. A sitemap is how you tell a search engine which pages exist and which have changed. If that map is full of dead ends, redirects, and pages you have since noindexed, you are sending crawlers to walk into walls, and they learn to trust the map less. Getting sitemaps right is one of the highest leverage, least glamorous jobs in large site SEO.
The sitemap index is not optional
A single sitemap file caps at fifty thousand URLs and fifty megabytes uncompressed. Any serious directory blows past that, so the structure you need is a sitemap index, a small file that lists many child sitemaps, each holding a slice of the site. The index is what you submit, and the engine fans out from there. Get this right and you can map millions of URLs cleanly. Get it wrong, by stuffing everything into one bloated file or by submitting a stale index, and discovery suffers across the whole site.
We split child sitemaps by section rather than at random, so each one maps to a meaningful part of the site. That alignment pays off later, because when one section has a problem you can see it isolated in the data instead of smeared across one giant file. It is the same principle of legible structure that runs through our quality bar for auto built pages, where everything is built so it can be inspected.
List only pages you would be proud to index
The single most common sitemap mistake is treating it as a dump of every URL the system can generate. A sitemap should list only canonical, indexable, live pages. That means no URLs that carry a noindex tag, no URLs that redirect, no URLs that canonical to a different page, and no parameter variants. Every one of those entries teaches the engine that your map is unreliable, and an unreliable map gets crawled less carefully.
Our rule is plain. A page goes in the sitemap only if we would be happy to see it indexed today, exactly as it is. If we are not happy with it, the fix is to improve the page or remove it, not to hide it while still listing it. This is the discipline that keeps the map trustworthy, and it ties straight into how we handle crawl budget and big sites, since a clean sitemap is the cheapest way to spend crawl budget well.
Last modified dates have to be real
The lastmod field is powerful and abused. When it is accurate, it tells the engine which pages genuinely changed so crawlers revisit the right ones first. When it is set to today on every page on every build, it is noise, and engines have learned to ignore lastmod from sites that lie with it. So we set lastmod from the real content change timestamp, not the build timestamp. A page whose underlying data did not change keeps its old lastmod, and only pages that actually changed advertise a fresh one. That honesty is what makes the field useful, and it connects to our approach to refresh cadence for programmatic pages, where real change and reported change have to stay in sync.
Pruning dead URLs
Large sites shed pages constantly. Venues close, categories merge, thin pages get pulled. Every retired page leaves a sitemap entry that now points to a gone or redirected URL, and those entries accumulate into a slow rot. So pruning is not a cleanup project you do once, it is part of every build. When the live page set is regenerated, the sitemaps are regenerated from that same set, so a page that no longer exists drops out of the map automatically. The sitemap and the live site should never disagree about what exists.
Compression and delivery
Child sitemaps are served gzipped to keep transfer light, and the whole structure sits at a stable, declared location referenced from robots.txt. We do not move sitemap URLs around, because a stable location is one less thing for the engine to relearn. Small operational details like these are unglamorous, and they are exactly the kind of thing that separates a site that crawls well from one that does not.
Submitting and monitoring
Submitting the index is the start, not the end. The real value is in monitoring the gap between submitted and indexed. When a section shows a wide gap, with many URLs submitted and few indexed, that is a quality signal pointing back at the pages themselves, not at the sitemap. We read that gap as a diagnostic. A clean sitemap turns indexing coverage into an honest report card, and that report card tells us where to improve content next. It is one of the inputs to how we approach measuring programmatic page quality across a large build.
How this fits the larger picture
A sitemap does not make a bad page rank. What it does is make a good site legible, so that the pages you worked hard on are found, understood, and revisited efficiently. On a large directory that legibility is worth a great deal, because crawl attention is finite and you want it spent on your strongest pages. We treat the sitemap as part of the same craft as everything else we build, set out across our programmatic SEO pillar, and we hold it to the same standard: real, current, and honest, every single build.
Kings Hospitality Group treats the sitemap as a promise, not a dump: our Clean Map rule says a sitemap lists only pages we would be happy to see indexed today. Anything noindexed, redirected, or retired is removed within the next scheduled build, never left to rot.
Common questions
How big can a single sitemap file be?
A single sitemap holds up to fifty thousand URLs and fifty megabytes uncompressed. Beyond that, use a sitemap index that points to many child sitemaps, each within those limits.
Should I include noindexed pages in my sitemap?
No. List only canonical, indexable pages you would be happy to see indexed today. Noindexed, redirected, and retired URLs should be removed so the map stays trustworthy.
How often should I update sitemaps on a large site?
Regenerate them on every build from the current live page set, with accurate last modified dates. The sitemap and the live site should never disagree about which pages exist.