HTTP caching headers tell crawlers and browsers whether they can reuse a copy of your page instead of downloading it again. Set them well and bots spend their limited time fetching your new and changed content; set them badly and you either serve stale pages or force wasteful re-downloads of things that never changed. The three headers that do the work are Cache-Control, ETag, and Last-Modified.
The short version: use Cache-Control to state how long a resource stays fresh, and use ETag or Last-Modified so a crawler can ask "has this changed?" and get a cheap 304 Not Modified when it hasn't. Done right, this speeds up crawling, saves bandwidth, and keeps AI bots seeing current content.
Cache-Control: how long is this fresh?
Cache-Control is the primary caching header. Its max-age directive sets, in seconds, how long a client may treat a resource as fresh without rechecking. A long max-age suits assets that rarely change; a short one suits pages you update often.
- Static assets (versioned CSS, JS, images with a hash in the filename) can use a very long max-age — a year — plus immutable, because a new version gets a new URL.
- HTML pages usually want a short max-age or no-cache, which means "you may store it, but revalidate before reusing." That keeps content current while still allowing cheap 304 responses.
- public vs private controls whether shared caches (CDNs, proxies) may store the response. Public HTML can be cached at the edge; anything user-specific should be private.
A powerful pattern for pages is stale-while-revalidate, which lets a cache serve a slightly stale copy instantly while it fetches a fresh one in the background — fast responses without serving badly outdated content. This pairs well with the CDN patterns in our guide on CDN caching and AI bots.
ETag and Last-Modified: has this changed?
These two headers enable conditional requests, the mechanism that saves the most bandwidth. When you serve a page, you attach an ETag (a fingerprint of the content) and/or a Last-Modified timestamp. On the next visit, the client sends that value back in an If-None-Match or If-Modified-Since header. If nothing changed, your server replies 304 Not Modified with an empty body — a tiny response instead of the full page.
For crawlers this is a direct efficiency win. Googlebot and AI bots that support conditional requests can check thousands of URLs for changes without re-downloading them, which frees crawl capacity for pages that did change. Prefer ETag for accuracy (it reflects actual content) and keep Last-Modified honest — a timestamp that updates on every request even when nothing changed defeats the purpose and can also mislead freshness signals, as covered in content freshness and lastmod.
How caching affects crawl efficiency and speed
Every fetch a bot skips because of a valid cache or a 304 is time it can spend elsewhere on your site. On large sites this is the difference between deep, frequent crawling and shallow, occasional crawling — the same crawl-budget logic in our guide to crawl budget optimization. Efficient caching also lowers your server load, which improves response time for real users and lifts your Core Web Vitals.
Common caching mistakes
The two failure modes are opposite extremes. Over-caching HTML with a long max-age and no revalidation means bots and users keep seeing an old version long after you have published updates — a quiet killer of freshness. Under-caching everything with no-store or no ETags forces a full re-download on every request, wasting crawl budget and slowing the site.
Other traps: sending contradictory headers (a long max-age alongside no-cache), forgetting that a CDN in front of your origin has its own cache rules that can override yours, and returning different ETags for identical content, which breaks conditional requests. Audit what your server actually sends — the headers in your config and the headers on the wire are not always the same.
Caching is one of the highest-leverage technical wins because it improves speed, crawl efficiency, and freshness at once. To see the exact response headers your pages return to crawlers, and whether AI bots are getting fresh HTML, run the CheckMy.site scanner.