Internal search results pages should be noindexed and kept out of your sitemap, because they generate near-infinite thin, duplicate URLs that waste crawl budget and dilute your site's quality signals. Add <meta name="robots" content="noindex,follow"> to your search results template, and in most cases let crawlers still reach them so they can discover the links within — but stop them from being indexed. Google has warned against indexed internal search pages for over a decade, and the problem is worse now that AI crawlers also spend budget on junk URLs.
Why search result URLs are a crawl trap
Every query a user (or a bot following a link) types can create a unique URL like /search?q=blue+widgets&sort=price&page=3. Multiply queries by filters, sort orders, and pagination and you have effectively unlimited URLs, all serving thin listing pages with little unique content. Crawlers discover these through internal links, sitemaps, or external links people share, then spend time fetching them instead of your real pages. This is the same crawl-budget drain covered in our crawl budget optimization guide — search pages and faceted navigation are two faces of the same problem.
Beyond wasted crawling, indexed search pages create duplicate and low-value content: a search for "blue widgets" and a category page for blue widgets can compete for the same query, and neither is as strong as it should be. Search pages also frequently return zero or thin results, producing near-empty pages that drag down how a site is assessed overall.
noindex, not robots.txt disallow — usually
The instinct is often to block /search in robots.txt. That prevents crawling but has a catch: a URL blocked in robots.txt can still get indexed as a bare link if it is referenced elsewhere, because the crawler never fetches the page to see a noindex tag. For control over indexing, the meta robots tag is the right tool; our meta robots tag guide explains why crawl-blocking and index-blocking are different jobs you should not confuse.
The clean pattern for most sites: allow crawling of search pages, serve noindex,follow so they drop from the index while still passing link equity through to the pages they list. Use robots.txt disallow only when the search endpoint is generating such extreme crawl volume that you need to shut off fetching entirely — and accept that you then lose the follow benefit. Choosing between noindex, robots.txt, and canonical for a given URL type is the exact decision framework in our faceted navigation guide.
Implementation checklist
- Add noindex,follow to the search results template's head. Confirm it renders in the raw HTML, not only after JavaScript, so bots that don't execute scripts still see it.
- Keep search URLs out of your XML sitemap. A sitemap is a list of pages you want indexed; search results are not those pages.
- Do not link to search results in navigation or footers. Every internal link is an invitation to crawl — don't hand out invitations to a crawl trap.
- Watch for the soft-404 case. A "no results found" page should ideally return a proper status and never be indexed as thin content.
- Handle parameter clutter. Sort and filter parameters on search URLs multiply the problem; keep them noindexed too rather than trying to canonicalize a moving target.
The exception worth knowing
A small number of very large sites deliberately index curated, high-demand search pages — think a marketplace turning "used road bikes in Berlin" into a landing page with real, stable, valuable content. That only works when the page is genuinely useful, stable, and treated like a real category page, not a raw query dump. For everyone else, the default is unambiguous: keep internal search results out of the index.
Indexed search pages are one of the most common quiet crawl-budget leaks, and they are easy to fix once you spot them. To find out whether your own search or filter URLs are exposed to crawlers and how your pages handle the robots directives, run the CheckMy.site scanner and get a clear picture of what bots can and can't index.