The unavailable_after rule tells search engines the exact date and time after which a page should stop appearing in results. It is the clean, official way to handle content with an expiry date — event pages after the event, limited promotions, job postings that close, seasonal listings — without deleting the URL or leaving stale pages to be cited by AI long after they are irrelevant. You deliver it either as a robots meta tag or, for non-HTML files, as an X-Robots-Tag HTTP header.
What unavailable_after does and does not do
When a crawler sees unavailable_after with a future date, it keeps the page indexed and eligible to rank normally until that moment passes. After the specified time, the page becomes eligible for removal from results, similar to a noindex — but scheduled in advance. Crucially, the page is not deleted, redirected, or blocked. It still returns 200, users with the link can still open it, and you retain the URL for whatever comes next. It simply stops being surfaced in search once its usefulness has a known end.
What it does not do matters just as much. It is not a substitute for proper removal of permanently gone content — for that you want a 410 or 404 response. It does not block crawling, so it belongs in a meta tag or header, never in robots.txt. And it is a Google-supported directive; other engines and AI crawlers may not honor it, so treat it as one layer, not a guarantee across every system.
How to implement it
For an HTML page, add a robots meta tag in the head with the rule and an absolute date. Google accepts common formats such as RFC 822, RFC 850, and ISO 8601. A typical tag reads: meta name="robots" content="unavailable_after: 2026-09-01 17:00:00 +0000". Always include the timezone offset so the cutoff is unambiguous.
- For non-HTML files — a PDF price list, a downloadable offer — you cannot use a meta tag, so send it as an X-Robots-Tag HTTP header from your server or CDN instead.
- Target the bot, not the page. You can scope the rule to specific crawlers using name="googlebot" if you only want it applied to Google.
- Do not also disallow the URL in robots.txt. If crawlers can't fetch the page, they can't read the directive, and the whole mechanism silently fails — the same trap that breaks noindex.
- Set the date in the future. A past date is treated like an immediate noindex.
When to reach for it — and when not to
unavailable_after shines when you know in advance that a page has a shelf life. A conference session page that is meaningless after the date, a flash sale, a fixed-term job listing, a limited registration window — all are natural fits. You schedule the cutoff once and forget it, avoiding both the manual work of pulling pages down and the risk of forgetting. For genuinely recurring or evergreen content, skip it; you don't want your best pages quietly expiring. And for anything you intend to remove permanently, use a real removal status code rather than leaving a soft, indefinitely-live URL behind, which can turn into a soft 404 problem.
Why expiry control matters for AI visibility
AI answer engines pull from whatever is indexed and reachable. An expired event page or a closed promotion that lingers in the index is exactly the kind of stale source that leads an assistant to tell a user about a sale that ended or a deadline that passed — damaging trust in your brand at the moment of the answer. By scheduling expiry, you keep the pool of pages that AI can cite aligned with what is actually current. This is the flip side of freshness: just as you want fresh pages surfaced, you want time-bound pages to gracefully step out of rotation. Combine unavailable_after with disciplined use of the meta robots tag for a complete indexing-control toolkit.
Stale, wrongly-indexed pages are hard to spot by eye across a large site. To find pages that are still indexable when they shouldn't be — and to confirm your directives are actually reachable by crawlers — run your site through the CheckMy.site scanner.