Keep Googlebot access rules current when IP ranges change

A firewall rule based on an old Google IP list can block legitimate requests. Google also uses different range sets and hostname patterns for common crawlers, special-case crawlers, and user-triggered fetchers.

Use Google's request-verification documentation to select the category you intend to allow. Do not assume one range file covers every Google requester.

Check the blocked requests

Compare crawler activity with CDN security events and server logs. Look for legitimate requests receiving 403, timeouts, or rate-limit responses around the time activity changed.

Check both the source list and the last successful update to your firewall or CDN. A current downloaded file does not help if the provider rejected the update or still uses the old rule.

Verify DNS in both directions

Google documents reverse DNS followed by forward DNS confirmation. The hostname must match the expected crawler category, and the forward lookup must resolve to the original client IP.

This Go helper checks a supplied list of permitted hostname suffixes and confirms the address:

import (
    "context"
    "net"
    "strings"
)

func googleHostname(name string, allowedSuffixes []string) bool {
    name = strings.TrimSuffix(strings.ToLower(name), ".")
    for _, suffix := range allowedSuffixes {
        if name == suffix || strings.HasSuffix(name, "."+suffix) {
            return true
        }
    }
    return false
}

func verifyGoogleIP(ctx context.Context, resolver *net.Resolver, source net.IP, allowedSuffixes []string) (bool, error) {
    names, err := resolver.LookupAddr(ctx, source.String())
    if err != nil {
        return false, err
    }
    for _, name := range names {
        if !googleHostname(name, allowedSuffixes) {
            continue
        }
        addresses, err := resolver.LookupIPAddr(ctx, strings.TrimSuffix(name, "."))
        if err != nil {
            continue
        }
        for _, address := range addresses {
            if address.IP.Equal(source) {
                return true, nil
            }
        }
    }
    return false, nil
}

Supply only the suffixes appropriate to the category being checked, and enforce any additional hostname pattern required by Google's documentation. A broad Google hostname match alone is not proof that the requester is Googlebot.

Run DNS checks with timeouts and caching rather than blocking on a fresh lookup for every public request. Resolve the real source IP through your configured trusted proxies; an arbitrary inbound X-Forwarded-For value is not a trusted client address.

Validate a range file before updating rules

If the provider requires IP ranges, fetch the appropriate published JSON and reject empty or malformed results. For the common-crawler list:

import requests
import ipaddress

GOOGLE_BOT_URL = "https://developers.google.com/static/crawling/ipranges/common-crawlers.json"
def fetch_googlebot_ranges():
    response = requests.get(GOOGLE_BOT_URL, timeout=15)
    response.raise_for_status()
    ranges = []
    for prefix in response.json().get('prefixes', []):
        value = prefix.get('ipv4Prefix') or prefix.get('ipv6Prefix')
        if value:
            ranges.append(ipaddress.ip_network(value, strict=True))
    if not ranges:
        raise ValueError('Google range file contained no prefixes')
    return sorted(ranges, key=lambda network: (network.version, int(network.network_address)))

Stage the complete new list, check provider limits, and confirm that its update task finishes. Keep the last working list if fetching or validation fails. Avoid deleting the active list before its replacement is ready.

Alert on failures in downloading, validation, provider updates, and final activation. Treat these as separate steps so a successful HTTP response does not conceal a failed rule update.

Test user-agent behavior separately

The following request examples can reveal application code that treats user agents differently:

CRAWLER_TESTS = [
    ('Googlebot Desktop', 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'),
    ('Googlebot Mobile', 'Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'),
    ('Googlebot-Image', 'Googlebot-Image/1.0'),
    ('Storebot-Google', 'Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36'),
    ('GoogleOther', 'Mozilla/5.0 (compatible; GoogleOther)'),
    ('AdsBot-Google', 'AdsBot-Google (+http://www.google.com/adsbot.html)'),
]

def test_all_crawlers(test_url):
    results = []
    for name, ua in CRAWLER_TESTS:
        try:
            response = requests.get(test_url, headers={'User-Agent': ua}, timeout=10)
            results.append((name, response.status_code, False))
        except requests.RequestException:
            results.append((name, 0, True))
    return results

These are test strings, not a current authoritative inventory of Google user agents. They also do not test whether Google's source IP is permitted: changing a header does not move your request onto Google's network.

Use the comparison after middleware changes, then check real verified traffic and CDN rule diagnostics for access decisions. Monitor sustained 401, 403, 429, and 5xx responses after the update.