Separate Google crawlers in your server logs

A single “Googlebot” total can hide crawler-specific problems. HTML pages may load while image requests are blocked, or Search Console's inspection tool may succeed while regular Googlebot requests fail. Separate the crawler families in your logs before comparing errors.

Use Google's crawler list to keep names current. The user-agent string helps classify a request, but anyone can copy it. Verify the source IP before using that string for access decisions.

Which crawler made the request?

Check the linked documentation for the complete, current list. Googlebot Smartphone and Desktop both contain Googlebot/2.1; distinguish them by the surrounding device string if that split is useful. Google-Extended is different: it is a robots.txt control token and does not have its own HTTP user-agent string.

Classify specific agents before generic ones

Order matters because some specialized agents also contain familiar Google terms. A Rust classifier can assign the narrow buckets first:

fn classify_google_crawler(user_agent: &str) -> &'static str {
    if user_agent.contains("Googlebot-Image") {
        "Googlebot-Image"
    } else if user_agent.contains("Googlebot-Video") {
        "Googlebot-Video"
    } else if user_agent.contains("Storebot-Google") {
        "Storebot-Google"
    } else if user_agent.contains("Google-InspectionTool") {
        "Google-InspectionTool"
    } else if user_agent.contains("AdsBot-Google") {
        "AdsBot-Google"
    } else if user_agent.contains("GoogleOther") {
        "GoogleOther"
    } else if user_agent.contains("Googlebot/2.1")
        && user_agent.contains("Mobile")
    {
        "Googlebot Smartphone"
    } else if user_agent.contains("Googlebot/2.1") {
        "Googlebot Desktop"
    } else {
        "Not classified"
    }
}

Save the crawler group with the path, status, response size, duration, cache result, client IP reported by a trusted proxy, and verification result. Then compare questions such as:

• Did 403 responses affect one crawler or every Google crawler?
• Did image requests shift to a new CDN hostname after a deployment?
• Is Storebot-Google repeatedly hitting out-of-stock product URLs?
• Are slow responses isolated to rendered HTML or shared assets?

Keep the original user-agent string

Do not replace the original user-agent value with the bucket. Keep both. When Google adds or changes a crawler, historical raw values let you reclassify old logs without losing detail.

For Nginx, a dedicated crawler log can include the fields needed for analysis:

log_format crawler '$time_iso8601 $remote_addr $status '
                   '$request_time $body_bytes_sent "$request_uri" '
                   '"$http_user_agent"';

At the CDN, retain the connecting client IP supplied by the platform rather than trusting an arbitrary inbound X-Forwarded-For value. Applications should only trust forwarding headers from configured proxies.

Verify before allowlisting

To check one request:

1. Extract the actual client IP from a trusted network boundary.
2. Reverse-resolve it and check the documented Google hostname suffix for that crawler class.
3. Resolve that hostname forward.
4. Confirm the original IP is in the returned address set.

Alternatively, compare against the appropriate current JSON range file in Google's request-verification documentation. Common crawlers, special-case crawlers, and user-triggered fetchers use different hostname patterns and published range files, so one stale “Googlebot list” is not a universal allowlist.

Alert on patterns, not single requests

A single 404 or timeout may be expected. Alert on sustained changes by crawler and route group: a rising 403 ratio, a crawl-volume drop, a latency increase, or the disappearance of a normally active crawler. Compare the same weekday and hour where traffic is cyclical.

Keep routine Googlebot crawling separate from testing tools and other Google fetchers. That makes it possible to see which requests changed after a release and investigate the affected URLs.