How to spot a crawl drop in server logs

A sudden fall in Googlebot requests can reveal an access or server problem before Search Console's aggregated reports make it clear. It can also reflect a change in crawl demand. Compare the drop with normal activity, response errors, and recent deployments before deciding what to fix.

CDN or trusted edge logs are useful here because they include requests blocked before reaching the origin.

Verify which requests belong to Google

Do not classify a request by its user-agent string alone. Verify the source against Google's published IP ranges or reverse-and-forward DNS checks. Keep the original user agent and trusted source IP so you can investigate classification errors later.

Use explicit fields rather than relying on positions in a free-form log line. For example:

type CrawlEvent struct {
    At             time.Time
    Crawler        string
    Verified       bool
    Host           string
    RouteFamily    string
    Status         int
    Duration       time.Duration
    CacheStatus    string
    ResponseBytes  int64
    DeploymentID   string
}

Group requests by crawler, hostname, page type, status, cache result, and time. Keep absolute counts next to percentages: one error from one request needs a different response from 500 errors in 500 requests.

Compare equivalent periods

A simple seven-day average mixes weekdays, weekends, and release days. Compare several matching weekday-and-hour periods, and leave incomplete current periods out of the baseline.

For spiky traffic, medians and median absolute deviation can be more useful than a mean. Whatever calculation you choose, set a minimum request volume and review alerts against actual incidents. Start by observing alerts before using them to trigger operational changes.

Useful patterns to investigate include:

• requests stop for a host or page type that is usually active;
• 401, 403, 429, or 5xx responses increase;
• response times and origin cache misses rise together;
• requests move from final pages to parameters, redirect chains, or error URLs;
• one crawler disappears while others continue normally.

Check recent changes first

1. Robots rules: compare the current file with the last working deployment.
2. CDN and WAF events: check whether a rule or rate limit caught legitimate requests.
3. Server health: look for timeouts, server errors, and slower responses.
4. Application releases: compare routing, authentication, rendering, and cache settings.
5. Crawler verification: check whether an IP-range snapshot or classification rule became stale.

Changing a test request's user agent to Googlebot can reveal application branching. It cannot show whether an IP-based firewall rule accepts real Google traffic; use verified logs and security events for that.

Record what changed, when it changed, and which requests were affected. After a repair, compare the same page types and time periods to see whether successful crawling resumes.