The meta robots tag is a single line of HTML that tells search engines and AI crawlers what they may do with a specific page: index it or not, follow its links or not, show a snippet or not. It lives in the page's <head> and controls indexing at the page level, which makes it fundamentally different from robots.txt — robots.txt controls whether a bot may crawl a URL at all, while meta robots controls what happens after the bot has already read the page. Confusing the two is one of the most common and costly technical SEO mistakes, so this guide makes the distinction concrete and shows exactly when to use each directive.

robots.txt controls crawling; meta robots controls indexing

Here is the distinction that trips up most people. A Disallow in robots.txt tells a bot not to fetch a URL. But if other sites link to that URL, search engines can still list it in results — without a description, because they were never allowed to read it. So robots.txt is the wrong tool for keeping a page out of the index. To reliably keep a page out of results, you must let the crawler fetch it and then find a noindex directive on the page. That is the paradox: to noindex a page, you must not disallow it in robots.txt, or the crawler will never see the noindex. For the crawl-side rules, our Complete robots.txt Guide covers syntax and pitfalls in full.

The directives you actually use

The tag looks like <meta name="robots" content="noindex, nofollow">, and the content value is a comma-separated list of directives. The ones worth knowing:

You can target specific crawlers by changing the name attribute — for example name="googlebot" for Google only, or a specific AI bot's token. The default when there is no tag is index, follow, so you only need the tag when you want to restrict something. Note also the X-Robots-Tag HTTP header, which does the same job but works for non-HTML files like PDFs where you cannot add a meta tag.

When to noindex — and when not to

Reach for noindex on pages that have no business in search results but must stay crawlable and usable: internal search results pages, thank-you and confirmation pages, thin tag or filter archives, staging URLs, and duplicate printer-friendly versions. Keeping these out of the index concentrates your crawl budget and ranking signals on pages that matter.

Be careful, though — noindex is a common accident. A single stray tag left over from a staging environment can silently deindex an important page or a whole section, and traffic quietly disappears. Two rules prevent disaster: never noindex a page you also plan to disallow in robots.txt (the directive would never be seen), and do not use noindex to solve duplicate content — that is what canonical tags are for. Our guide to Canonical URLs and Duplicate Content explains the right tool for that job.

noindex, AI crawlers, and how it interacts with the rest of SEO

AI systems respect these directives too, and there are subtle interactions. A page marked nosnippet tells engines not to show a preview, which can reduce how quotable your content is to AI answer engines — useful for gated content, harmful if applied to pages you want cited. Similarly, noindex keeps a page out of the retrieval pool entirely. Decide deliberately: pages you want AI to quote should be indexable and snippet-friendly, while genuinely private or duplicate pages should be locked down. This page-level control complements the broader meta setup covered in our Complete Guide to Meta Tags.

One more coherence check: make sure your directives do not contradict each other. A page that is noindex should generally not appear in your XML sitemap, since the sitemap invites indexing while the tag forbids it — mixed signals waste crawl budget and confuse engines.

Audit your directives before they cost you traffic

Do a quick sweep of your site: confirm your important pages have no accidental noindex, confirm the pages you meant to hide actually carry it, and confirm nothing you noindexed is also blocked in robots.txt. These errors are invisible until traffic drops, so catching them proactively is worth the ten minutes. To scan your live pages for conflicting robots directives, accidental noindex tags, and crawler-access problems all at once, run the CheckMy.site scanner and fix whatever it surfaces.