A request that says Googlebot in its user agent is not automatically a Google request. User-agent strings are supplied by the client and are trivial to copy.
That distinction matters whenever server, CDN, WAF, or analytics data is used to diagnose crawling. A dashboard can accurately count requests whose claimed identity is Googlebot while still being unable to establish the provider behind them.
The useful evidence chain is:
- preserve the request before aggregation
- verify the source independently
- retain the exact route, time, status, and response path
- compare the request with current search-platform evidence
- stop the claim at the layer the evidence supports
Preserve the raw request fields
Keep enough data to reproduce the verification without retaining unnecessary personal or payload data:
| Field | Why it is needed |
|---|---|
| Source IP observed at the trusted edge or origin | Input to provider verification |
| Forwarding chain and trusted proxy boundary | Prevents a client-supplied forwarding header from becoming the source |
| Timestamp with timezone | Joins the request to releases, WAF events, and Search Console observations |
| Method, host, and normalized path | Identifies what was actually requested |
| User agent | Classifies the claimed crawler family; does not verify it |
| Status, response bytes, and duration | Shows how the server handled that request |
| Cache, edge, and request ID headers | Locates the delivery layer and a corresponding event |
Do not treat the leftmost X-Forwarded-For value as trusted by default. The application must know which proxy wrote the header and which hops can be supplied by the client.
Use one of Google's documented verification paths
Google documents two verification methods in its crawler verification guide.
Published IP ranges for automation
For repeated or high-volume verification:
- retrieve the applicable Google-published crawler IP range file
- validate its format and retrieval time
- parse addresses as networks, not string prefixes
- test the observed source IP against those networks
- cache the range data with an explicit refresh and failure policy
- retain the range source and version used for the decision
Google publishes different ranges for common crawlers, special-case crawlers, and user-triggered fetchers. Match the request to the correct class instead of treating every Google-owned address as Googlebot.
An unavailable or stale range source should produce unverified or verification unavailable, not a silent provider confirmation.
Reverse and forward DNS for manual checks
For an individual request:
- perform a reverse DNS lookup on the observed source IP
- verify that the returned hostname ends in an applicable Google domain such as
googlebot.com,google.com, orgoogleusercontent.com - perform a forward lookup on that hostname
- verify that the original source IP appears in the forward result
The forward confirmation closes the obvious reverse-DNS spoofing gap. Record both results and the verification time.
Use explicit identity states
A crawler dashboard should not compress every state into “Google.”
| Identity state | Meaning | Safe use |
|---|---|---|
| UA-claimed | The user agent matches a known family | Initial classification and triage |
| Provider-verified | Source passed the documented IP or DNS check | Provider-level crawl analysis |
| Verification unavailable | Required source field or verification service was unavailable | Queue for later review; do not promote |
| Verification failed | Claimed identity did not pass the selected check | Investigate spoofing, proxy handling, or stale rules |
| Other Google fetcher | Verified Google source, but not the crawler family being analyzed | Keep separate from Googlebot search crawling |
Store the claimed family and verified provider separately. That preserves useful user-agent analysis without turning it into an identity claim.
Verification does not prove the page was indexed
A provider-verified request can establish that a Google crawler requested a named route at a named time and received an observed response through the logging layer.
It does not establish:
- that the response body was complete when processed downstream
- that rendered output matched the raw response
- that Google selected the URL or its content for indexing
- that the URL appeared in search results
- that a later impression or click came from that crawl
Join the request to raw response capture, rendered comparison, URL Inspection, Page indexing data, and later search performance only when those questions matter.
Verification checklist
- source IP came from a trusted edge or origin field
- proxy and forwarding-header ownership is documented
- verification method and data-source time are retained
- crawler class is separated from provider identity
- request path, timestamp, status, and request ID remain available
- failed and unavailable checks do not become verified counts
- dashboards label UA-claimed and provider-verified totals distinctly
- conclusions stop at crawl evidence unless indexation or visibility evidence is also present
The purpose of verification is not to make a crawler chart look more authoritative. It is to make the chart usable for a specific technical decision.