Before a search engine or AI crawler reads a single word of your content, it reads a three digit number: the HTTP status code. That number decides whether the page gets indexed, whether ranking signals transfer, whether the crawler comes back tomorrow or gives up on your site for a while. Most SEO problems that look mysterious in reports turn out to be status code problems underneath. This guide covers the codes that matter, in the situations where they matter.
200: Everything Is Fine, Usually
A 200 response means the page exists and was delivered. The only trap here is the soft 404: a page that returns 200 while its content says something like nothing found here. Content systems that catch missing URLs and render a friendly error page with a success status create these constantly. Search engines detect the pattern, classify the URL as a soft 404, and treat it as an error anyway, except now your monitoring tools show everything green. If a page does not exist, the status code should say so.
301 vs 302: The Most Consequential Choice
A 301 declares a permanent move. Search engines transfer indexing and ranking signals to the new address and eventually replace the old URL in results. Use it whenever content has genuinely moved: domain migrations, restructured URLs, the http to https upgrade, consolidating the www and non www variants.
A 302 declares a temporary detour. The original URL stays indexed, signals stay put, and the crawler expects the redirect to disappear. Use it for genuinely temporary situations: a product briefly redirecting to its parent category, an A/B test, a geo based landing page.
The classic mistake is shipping 302s where 301s were intended, because 302 is the default in many frameworks. Google eventually treats a long lived 302 as permanent, but eventually can take a long time, and other crawlers are less forgiving. Say what you mean the first time. The modern siblings 308 and 307 behave like 301 and 302 while also guaranteeing the request method is preserved, and are equally safe for SEO.
Redirect Chains: The Silent Budget Leak
A visitor requests page A, which redirects to B, which redirects to C, which finally answers 200. Every hop adds latency for users and consumes crawl budget for bots. Google follows up to ten hops but trusts long chains less, and other crawlers give up earlier. Chains grow organically: first the http to https redirect, then a www consolidation, then a site restructure, each layer stacked on the last. The fix is periodic flattening: every redirect should point directly at the final destination in one hop. You can see a chain from any terminal:
curl -sIL https://example.com/old-page | grep -iE "HTTP/|location:"One line of output per hop. More than two hops means cleanup time.
404 vs 410: Deleting Pages Properly
A 404 says not found, with a shrug: maybe temporary, maybe permanent. Search engines recheck 404 URLs for a while before dropping them. A 410 says gone on purpose, and search engines typically deindex the URL faster and stop rechecking sooner. For content you deliberately removed forever, 410 is the more precise answer.
Two rules for deletions. First, if a genuinely equivalent page exists, prefer a 301 to it over any error code, so accumulated signals survive. Second, never mass redirect deleted pages to the homepage. Search engines recognize that pattern, classify those redirects as soft 404s, and you get the worst of both worlds: no error clarity and no signal transfer.
403 and 429: The Accidental Bot Blockers
A 403 Forbidden aimed at crawlers is one of the most common self inflicted wounds of the current era. Firewalls, security plugins and CDN defaults increasingly serve 403 to anything that looks automated, including the AI crawlers you actually want. Your human visitors see a perfect site while GPTBot and ClaudeBot bounce off a wall, exactly the failure we documented in our article on CDN level AI bot blocking.
A 429 Too Many Requests tells a crawler to slow down, and well behaved bots comply by reducing crawl rate for a long period. A rate limiter tuned for abusive traffic but triggering on normal crawler behavior can quietly strangle your indexing for weeks.
5xx: Handle Downtime Without Losing Rankings
Server errors in the 500 range tell crawlers the site is broken. Occasional errors are ignored, but persistent 5xx responses lead to reduced crawling and, if they continue for days, to pages dropping out of the index.
Planned maintenance has a correct protocol: respond with 503 Service Unavailable and include a Retry-After header stating when to come back. Crawlers pause politely without penalizing the site. Serving maintenance pages with status 200 instead gets your under construction page indexed as your content, and serving 404 gets real pages deindexed.
One special case outranks all others in importance: robots.txt itself. If robots.txt returns 5xx errors persistently, Google treats the entire site as uncrawlable and stops fetching pages altogether. A broken robots.txt handler can halt indexing sitewide while every page works perfectly.
A Quick Reference for Common Scenarios
Site migration: map every old URL to its exact new counterpart with one hop 301s, keep the redirects live for at least a year, update the sitemap to list only new URLs.
Discontinued product: 301 to the closest genuinely relevant alternative if one exists, otherwise 410.
Protocol and host cleanup: http, www and trailing slash variants should each reach the canonical form in a single 301, complementing the consolidation work from our canonical URL guide.
Maintenance window: 503 plus Retry-After, never 200, never 404.
Verify the Whole Picture in One Scan
The CheckMy.site scanner follows your redirects hop by hop, verifies that http and www variants consolidate in a single step, tests how your server answers missing pages, and performs live bot fetches that expose 403 walls invisible to normal visitors, all as part of its 158 point report. Status codes are the grammar of the web. Speak it precisely and every crawler, search engine and AI assistant understands your site the way you intended.