The X-Robots-Tag is an HTTP response header that controls how crawlers index a file — and unlike the meta robots tag, it works on any file type, not just HTML. That makes it the only way to tell search engines and AI crawlers to noindex a PDF, an image, a video, or any non-HTML resource, because those files have no <head> to hold a meta tag. If you have PDFs ranking that you would rather keep private, or generated files cluttering search results, this header is the fix.
Why meta robots is not enough
The familiar way to control indexing is the meta robots tag in a page's head. It works well — for HTML. But a huge amount of what crawlers find is not HTML: whitepapers and invoices as PDFs, product shots and diagrams as images, downloadable spreadsheets, generated report files. None of these can carry a meta tag, because there is no HTML head to put it in.
This is exactly the gap the X-Robots-Tag fills. Because it lives in the HTTP header the server sends with the file, it applies to any response regardless of format. Anything you can serve, you can annotate. If you have not worked with the HTML version, our guide on the meta robots tag covers the directives themselves — noindex, nofollow, and the rest — which are identical here.
How the directives work
The X-Robots-Tag accepts the same directives as meta robots. A header of X-Robots-Tag: noindex keeps the file out of search results. noindex, nofollow also tells crawlers not to follow links inside the file. You can target a specific crawler by naming it first — X-Robots-Tag: googlebot: noindex applies only to Google — and you can send multiple lines to give different bots different rules.
One critical rule underlies all of this: the crawler must be able to fetch the file to see the header. If you block a URL in robots.txt, the bot never requests it, never receives your noindex, and may still list the URL based on external links. To reliably deindex something, allow it to be crawled and serve a noindex header — do not block it in robots.txt. Our complete robots.txt guide explains why blocking and noindexing are opposite instructions that cancel each other out.
Setting it in practice
Because it is an HTTP header, you set the X-Robots-Tag at the server or CDN, not in the file. On Apache you add it in the configuration or an .htaccess block, typically scoped to a file type with a FilesMatch rule — for example, applying noindex to everything matching \.pdf$. On Nginx you use an add_header directive inside a location block that matches the extension. On a CDN, an edge rule can attach the header to matching paths without touching origin at all.
Scope matters. A rule that says "noindex every PDF on the site" is easy to write and easy to regret if some PDFs are lead magnets you want found. Target by directory or file pattern so you deindex only what should be private — internal documents, generated exports, staging assets — while leaving marketing PDFs indexable. If you serve files through a CDN, verify the header survives caching; a cached response can pin an old header in place, a trap covered in our guide on how search engines crawl and index.
Verifying it works
Never assume the header is present — confirm it. Request the file and inspect the raw response headers; the X-Robots-Tag line should be there with exactly the directive you intended. A common failure is setting the rule for one file type and finding it silently does not match because of a case-sensitivity or path-pattern mistake, so the header never actually attaches.
After that, use Search Console's URL Inspection to see how Google interprets the file, and watch the Pages report over the following weeks as noindexed files drop out. Deindexing is not instant — the crawler has to re-fetch the file and see the new header before it removes the URL, which can take days to weeks depending on how often that file is crawled. Patience beats re-editing a rule that was already correct.
The X-Robots-Tag is the missing piece for controlling everything on your site that is not an HTML page — and a misconfigured one can either leak private files into search or hide files you wanted found. To check which of your pages and files are indexable and whether their headers say what you think they say, run the CheckMy.site scanner and catch indexing mistakes before a crawler acts on them.