To confirm a crawler hitting your site is really GPTBot, Googlebot, or another legitimate AI bot — and not a scraper faking the name — verify it two ways: check the request IP against the operator's published IP ranges, or run a reverse-then-forward DNS lookup that must resolve back to the official domain. A user-agent string is just text anyone can send, so never trust it alone. This matters because verification is what lets you welcome the real AI crawlers you want citing you while blocking the impostors abusing their name.
Why the user-agent is not proof
Any client can set its User-Agent header to "GPTBot" or "Googlebot" in one line of code. Scrapers do this constantly to bypass rules meant for humans or to hide behind a trusted name. If you make allow/block decisions purely on the user-agent string, you will simultaneously let fake bots in and — if you block on it — risk blocking real ones spoofed by nobody. The whole point of verification is to separate the identity claim from the proof. For the full roster of real crawler names to check against, see our complete list of AI crawlers in 2026.
Method 1: reverse DNS (the universal check)
The most reliable method most operators support is a two-step DNS verification:
- Reverse lookup the visiting IP address (a PTR query). A genuine Googlebot IP resolves to a hostname on googlebot.com or google.com; other operators publish their own domains.
- Forward-confirm that hostname with an A/AAAA lookup and check that it resolves back to the exact IP that made the request.
Both steps must pass. A scraper can fake a user-agent but cannot make your DNS resolver return an official hostname for its IP, and cannot make that hostname forward-resolve to its address. On the command line this is host <ip> to get the name, then host <name> to confirm it points back. This is the gold standard when a bot operator does not publish static IP ranges.
Method 2: published IP ranges
Major operators publish official IP lists you can match against directly, which is faster than DNS at scale. Google, OpenAI, and others provide JSON files of the ranges their crawlers use; you download the list, keep it fresh, and check each incoming bot IP against it. This is ideal for automated firewall or edge rules because it avoids a DNS round-trip per request. The tradeoff is that the lists change, so you must refresh them on a schedule rather than hardcoding ranges that will silently go stale.
Putting verification to work
Verification only helps if you act on the result. Start by seeing who actually claims to visit — grep your access logs for bot user-agents, as shown in our server log analysis guide — then verify the interesting ones. From there:
- Allow verified good bots the access you intend, and make sure your CDN or firewall isn't silently blocking them; unverified blocking is a common way sites accidentally hide from AI, as covered in our guide to Cloudflare blocking AI bots.
- Rate-limit or block unverified traffic claiming a bot identity that fails both checks — that is a scraper wearing a costume.
- Log the verification outcome, not just the user-agent, so your analytics reflect real crawler behavior instead of spoofed noise.
Do not overreach: blocking too aggressively on a stale IP list or a flaky DNS check can lock out the very assistants you want quoting your content. Verify, then allow the real ones generously.
Knowing which bots are genuinely crawling you — and which are impostors — is the foundation of any AI-visibility strategy, because you cannot optimize for citations from crawlers you are accidentally blocking. To see how your site currently responds to major AI crawlers and whether they can reach your content, run the CheckMy.site scanner and get an instant read on your bot access.