Faceted navigation — the filters and sort options on category pages like color, size, price, and brand — is great for shoppers and dangerous for crawlers. Each combination of filters can generate a unique URL, and a single category can explode into thousands or millions of near-duplicate pages. Left unmanaged, this wastes crawl budget, dilutes ranking signals, and buries the pages you actually want indexed. The fix is choosing the right control for each type of URL: noindex, robots.txt, or canonical.
Short answer: use a canonical tag when a filtered page is a variant that should still be crawlable but consolidated to a main URL; use noindex when a page must be crawlable but never indexed; and use robots.txt only to stop bots from crawling worthless parameter URLs entirely. They solve different problems and are not interchangeable.
The problem: infinite filter combinations
Say a category has 10 filters with several options each. Multiply the combinations and you get a combinatorial explosion of URLs, most of them thin, overlapping, or empty. Crawlers try to fetch them, so bot attention that should go to your important pages drains into filter noise instead. This is the classic crawl-budget leak described in our guide on crawl budget optimization. The goal is not to hide filters from users — it is to keep machines from treating every filter combination as a separate page worth crawling and indexing.
Canonical tags: consolidate, don't hide
A canonical tag tells search engines that a filtered URL is a variant of a primary page and that ranking signals should be consolidated there. Use it when the filtered page has legitimate content that could be linked to or shared, but you do not want it competing with the main category. For example, a "sort by price" version of a category should usually canonicalize to the unsorted category, because sorting does not create meaningfully different content.
The key limitation: canonical is a hint, not a command, and the page still gets crawled. It is the right tool for consolidating duplicates, not for saving crawl budget. For the full mechanics, see canonical URLs and duplicate content. Also make sure canonicalized pages are still reachable — canonicalizing a page you have also blocked in robots.txt sends a contradictory signal, because the engine cannot crawl the page to see the canonical tag.
noindex: crawlable but kept out of the index
Apply noindex (via a meta robots tag or X-Robots-Tag header) when a filtered page genuinely should never appear in results but you still want bots to crawl through it — for instance, to follow links to products. A page with noindex,follow is read, its links are discovered, but it stays out of the index. This is stronger than canonical for keeping pages out of results because it is a directive, not a suggestion.
The trade-off is that the page is still crawled, so noindex controls indexing, not crawl budget. And a critical rule: never block a noindex page in robots.txt. If bots cannot crawl the page, they never see the noindex tag, and the URL can still get indexed from external links. For how these directives behave, read the meta robots tag guide.
robots.txt: stop the crawl entirely
Use robots.txt to disallow crawling of parameter patterns that have zero indexing value — session IDs, tracking parameters, or combinations no one should ever land on. This is the only tool that actually saves crawl budget, because blocked URLs are never fetched. The cost is that robots.txt is a blunt instrument: a blocked URL can still appear as a bare link in results (without a snippet) if other sites link to it, and blocking hides any canonical or noindex tag on the page.
A practical decision framework
- Filter creates a useful, shareable variant (e.g., a popular brand+category page you want indexed): let it be crawled and indexed on its own, with a self-referencing canonical.
- Filter is a duplicate of a main page (sorting, view options): keep it crawlable, add a canonical to the primary URL.
- Filter should never be indexed but links matter: use noindex,follow, no robots.txt block.
- Filter is pure junk (tracking params, infinite combinations of no value): disallow the pattern in robots.txt.
The mistake that causes most faceted-navigation disasters is stacking controls incorrectly — blocking in robots.txt a page you also tried to noindex or canonicalize, so the engine never sees your intent. Pick one control per URL type and make sure the page stays crawlable if the signal lives on the page. To find which filtered URLs bots are crawling and whether your controls are being read correctly, run the CheckMy.site scanner.