How to Block Bad Bots Without Blocking Googlebot

To block bad bots without blocking Googlebot, do not trust the User-Agent header by itself. Verify Google crawler requests using Google's published IP ranges or the documented reverse-and-forward DNS procedure, apply crawler allowance before broad automation rules, and test important URLs with Search Console after every policy change.

Why search crawlers are easy to block accidentally

Search crawlers are automated, request many pages, and may originate from infrastructure networks. Those characteristics can resemble abusive crawling when a rule only checks automation, request volume, or network type. A global policy that blocks every bot therefore risks removing the same crawler access that search indexing depends on.

Google also operates several crawler and fetcher classes. Googlebot affects Google Search and related surfaces, while tools such as Google-InspectionTool are used for testing and do not represent ordinary search crawling. Preserve only the access your site needs and verify the actual requester.

Do not use robots.txt as crawler authentication

The robots exclusion protocol communicates crawl preferences to cooperative crawlers. It does not prove crawler identity, stop an attacker from ignoring the file, or authenticate a request that claims a familiar user agent. Use robots.txt for crawl instructions and a separate verification process for trusted crawler allowance.

Two supported ways to verify Google requests

Google documents two verification methods. At scale, match the source address against Google's published crawler IP ranges. For a manual investigation, perform a reverse DNS lookup, confirm the returned hostname belongs to an expected Google domain, then perform a forward DNS lookup and confirm that it resolves back to the original address.

1. Read the source IP from a trusted server or proxy path, not an untrusted forwarded header.
2. Use Google's current published crawler range data for automatic matching, or begin a manual reverse DNS check.
3. For manual checks, require the documented Google hostname suffix rather than a substring match.
4. Resolve that hostname forward and require the original source IP to appear in the result.
5. Cache successful verification briefly to avoid repeated DNS work, but refresh it instead of creating a permanent assumption.

source IP
  -> reverse DNS
  -> expected googlebot.com / google.com / googleusercontent.com hostname
  -> forward DNS
  -> original source IP must match

Put verified crawlers before broad deny rules

The policy order matters. First reject malformed requests and establish a trustworthy source address. Next verify known crawlers and explicit service integrations. Then evaluate general bot, reputation, country, network, URL, and customer rules. A request that only claims a search user agent should continue through normal detection; only a verified crawler should receive crawler-specific allowance.

• Allow verified search crawlers only on content they need to crawl.
• Keep administrative, account, and private application routes unavailable regardless of crawler identity.
• Treat uptime monitors as a separate allowlist with their own published addresses or authentication mechanism.
• Log the verification method and policy reason so an unexpected allow can be investigated.
• Do not exempt a broad cloud provider range simply because a legitimate service uses the same provider.

Test SEO access after an anti-bot change

1. Check that robots.txt returns HTTP 200 and references the current sitemap.
2. Inspect the home page, product pages, and newly published articles with Search Console URL Inspection.
3. Confirm the fetched page is not redirected to sign-in, a challenge loop, or an error template.
4. Review origin and CDN logs for Googlebot, Google-InspectionTool, 403 responses, and unusual redirects.
5. Monitor crawl and indexing reports after deployment instead of assuming one successful test covers every crawler.

How this maps to Stopbot decisions

A Stopbot integration should apply the final blockAccess decision rather than blocking solely because isBot is 1. Legitimate automation can be classified as automated and still be permitted by the active policy. Keep verified search-engine handling, customer allowlists, and the response reason visible in testing so a crawler exception cannot silently become a universal bypass.