Crawl control is the difference between a directory that gets fully indexed in weeks and one that is still waiting months later. Search engines allocate a finite amount of crawling to any site. On a large catalogue, if that budget is spent wandering through filter combinations, internal search pages and session URLs, the real listings starve. The two instruments that govern this, robots.txt and the meta robots tag, are simple to state and easy to misuse, so it pays to be precise about what each one does.
Two tools, two different jobs
The single most important distinction in this whole topic is that robots.txt controls crawling and meta robots controls indexing. They are not interchangeable, and confusing them causes some of the most damaging SEO mistakes I see on large sites.
Robots.txt sits at the root of the domain and tells crawlers which paths they may or may not request. It is about access. Meta robots, placed in the head of a page or sent as an HTTP header, tells search engines whether they may index a page they have already crawled. It is about inclusion. Critically, a crawler must be allowed to fetch a page to see its meta robots tag. If you block a URL in robots.txt, Google never reads the noindex you put on it, so the two instruments can quietly cancel each other out.
The trap that catches almost everyone
Here is the most common and most expensive error. You want a page kept out of the index, so you both block it in robots.txt and add a noindex tag. It feels like belt and braces. In fact, because robots.txt stops the crawl, Google cannot see the noindex. The URL can then linger in the index, often shown with no description, kept alive by external or internal links. To remove a page from the index, do the opposite of instinct: allow crawling, and let the noindex be read. This interacts directly with how you set canonical tags, since a canonical on a blocked page is also never seen.
What to block from crawling
Robots.txt earns its keep by keeping crawlers away from places that generate near infinite low value URLs. On a directory, the usual suspects are predictable.
- Internal search result pages, which can produce a unique URL for every query a user ever types.
- Faceted filter combinations that multiply without producing genuinely distinct, demand backed content.
- Session ids, cart or account URLs, and any path that exists only for logged in flows.
- Endless calendar or date parameter pages that stretch forward forever.
- Staging paths, internal tools and anything that should never appear in search.
These are crawl traps. Left open, they soak up budget that should be reaching your listings. Blocking them is the first move in the triage I run on any large catalogue, and it pairs with the duplicate control work in avoiding index bloat.
What never to block
The flip side is just as important. Never block resources that Google needs to render and understand your pages. Blocking CSS and JavaScript can leave the crawler with a broken view of your layout, which undermines mobile rendering and the content parity that matters so much, as covered in mobile first for directory sites. Never block your sitemaps. And never block paths that are the only route to deep listings, such as paginated category URLs, or you sever the crawl path to large parts of your catalogue.
Meta robots in practice
Meta robots is your scalpel for individual pages. The defaults you will use most are straightforward. Index, follow is the implicit default and needs no tag. Noindex, follow keeps a page out of the index while still letting its links pass crawl signals onward, which suits thin utility pages you want crawled but not ranked. Noindex on its own removes a page from the index.
Use noindex deliberately on pages that have no business ranking: thank you pages, certain filtered views, thin tag pages that will never carry real content. It is the honest way to keep low value pages out of search without the side effects of a robots block. Just remember the page must stay crawlable for the tag to work, and that thin pages are usually better fixed or merged than simply hidden, a judgement we explore in the thin page problem.
Crawl budget on a large catalogue
Crawl budget is not something most sites need to think about, but a directory with hundreds of thousands of URLs is exactly the case where it bites. The principle is simple. Every request a crawler spends on a junk URL is a request it did not spend on a listing you want ranked. Your job is to shape the site so that the crawlable surface is mostly pages worth indexing.
That shaping comes from several directions at once: blocking traps in robots.txt, keeping a flat reachable architecture, removing dead and redirecting links, and serving fast so the crawler can fetch more pages per visit. Log file analysis is the way to see where budget actually goes, which is why we treat it as its own discipline in our technical SEO and indexing pillar.
Validating your robots rules
Robots.txt is unforgiving because a single broad disallow can hide an entire section. Before you ship a change, test it. Use the robots testing tools in Search Console to confirm that your important URLs are still allowed and your trap patterns are blocked. After deploying, watch coverage reports for any sudden drop in indexed pages, which is the classic symptom of an overzealous disallow. Keep the file under version control and review it whenever the URL structure changes, because robots rules written for last year's site can quietly strangle this year's. The careful, long horizon way we run catalogues is set out in our building thesis.
The short version
Robots.txt controls crawling, meta robots controls indexing, and confusing the two causes most of the damage in this area. Block crawl traps like internal search and infinite filters, never block resources or the only path to deep listings, and use noindex on crawlable pages to keep low value content out of the index. Manage crawl as a budget and your real listings get the attention they deserve.
Kings Hospitality Group works from a Crawl Budget Triage list on every large directory: infinite filter combinations, internal search results and session URLs are blocked from crawling first, because on a big catalogue wasted crawl on those traps is the most common cause of slow indexing for the pages that matter.
Common questions
Does blocking a page in robots.txt remove it from Google?
No. Robots.txt stops crawling, not indexing. A blocked URL can still appear in results from external links. To keep a page out of the index, allow crawling and use a noindex tag instead.
Should I block faceted filter URLs?
Often yes, when filters produce endless low value combinations. Block the trap patterns in robots.txt so crawlers spend their budget on real listings, while keeping valuable filtered landing pages crawlable.