Crawler verification is how a site checks that a request claiming to be a known bot, such as Googlebot, really comes from that operator. Because a user-agent header is trivial to fake, the check relies on DNS lookups or published IP ranges instead.
Any client can send a Googlebot user-agent string, and scrapers sometimes do it hoping for preferential treatment. Sites that allow search crawlers through their defenses therefore need a second signal that an impersonator cannot copy, which is where the network origin of the request comes in.
The reverse is also true: spoofing a crawler identity is easy to detect and often gets an IP blocked faster than an honest, ordinary client would. If your own crawler is a legitimate bot, declare a truthful user-agent with contact information instead of borrowing someone else's.
Google's documented method is two lookups. Run a reverse DNS lookup on the requesting IP, and check that the hostname ends in googlebot.com, google.com, or googleusercontent.com. Then run a forward lookup on that hostname and confirm it resolves back to the same IP. The forward step matters because whoever controls an IP's PTR record can make it claim any name.
Google also publishes machine-readable lists of its crawler IP ranges, which suit high-volume checks better than a DNS query per request. Other operators document their own methods and ranges, and newer schemes such as Web Bot Auth use cryptographic signatures instead of network checks. Cache results, since DNS lookups add latency.
import ipaddress
import socket
GOOGLE_SUFFIXES = ("googlebot.com", "google.com", "googleusercontent.com")
def is_googlebot(ip: str) -> bool:
try:
address = ipaddress.ip_address(ip)
host = socket.gethostbyaddr(ip)[0].rstrip(".").lower()
except (OSError, ValueError):
return False
trusted_host = any(
host == suffix or host.endswith("." + suffix)
for suffix in GOOGLE_SUFFIXES
)
if not trusted_host:
return False
try:
resolved = {
ipaddress.ip_address(result[4][0])
for result in socket.getaddrinfo(host, None)
}
except (OSError, ValueError):
return False
return address in resolved
print(is_googlebot("66.249.66.1"))How Crawlora handles this
Crawlora is a scraping API for fetching public web data, not a way to pose as a search engine, and you should not set a Googlebot user-agent on your requests. If you operate your own crawler, send an honest user-agent and respect robots.txt; the same lookups above are what protect your own site from impersonators.
Related reading
Glossary
FAQ
No. The header is chosen by the client and is trivial to forge. Use reverse and forward DNS, or the operator's published IP ranges, to confirm identity.
A reverse lookup returns whatever the IP owner puts in the PTR record, so it can be set to a trusted-looking name. Resolving that name forward and matching the IP closes the loophole.
Only when the operator publishes a method. Many AI crawlers publish IP ranges instead of DNS conventions, so check each operator's documentation and keep your allowlists current.
Beyond Crawler Verification (Reverse DNS), Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.