Googlebot keeps crawling the wrong URLs: what to measure

If Googlebot repeatedly requests filters, old URLs, and errors while updated product pages go untouched, start with a breakdown of the actual requests. A crawl total cannot show which pages received attention.

Google's crawl-budget guide describes crawl capacity and demand at hostname level. Removing duplicate URLs does not automatically transfer every saved request to a page you want crawled.

Group verified requests by URL pattern

Verify Googlebot traffic before counting it. Group requests by hostname, path or page type, response status, and cache result. Keep both total requests and the number of unique final URLs.

This Rust example assumes a log format with a trusted verification marker and a request path in the seventh field:

use std::collections::HashMap;
use std::fs::File;
use std::io::{BufRead, BufReader};

#[derive(Debug)]
struct CrawlStats {
    total_requests: u64,
    by_prefix: HashMap<String, u64>,
}

fn aggregate_verified_logs(path: &str) -> CrawlStats {
    let file = File::open(path).unwrap();
    let reader = BufReader::new(file);
    let mut stats = CrawlStats {
        total_requests: 0,
        by_prefix: HashMap::new(),
    };

    for line in reader.lines().map_while(Result::ok) {
        if !line.contains("verified-googlebot=true") {
            continue;
        }

        let request_path = line.split_whitespace().nth(6).unwrap_or("/");
        let prefix = request_path
            .trim_start_matches('/')
            .split('/')
            .next()
            .unwrap_or("root");

        stats.total_requests += 1;
        *stats.by_prefix.entry(prefix.to_string()).or_insert(0) += 1;
    }

    stats
}

Adapt the parser to your actual log format. It is a small aggregation example, not a complete source-IP verifier or production log parser. Explicit fields, error handling, and URL normalization are needed before relying on the totals.

A large product-page count may be healthy. Repeated requests to redirect chains or endless filter combinations deserve a closer look at the templates that expose them.

Remove unintended URL combinations

• Stop linking to duplicate filters and alternate sorts that add no useful destination.
• Use one URL form in navigation and contextual links.
• Redirect obsolete aliases to their final replacement without long chains.
• Return 404 or 410 for permanently removed content without a replacement.
• List canonical URLs you want indexed in the sitemap.
• Update lastmod when the page meaningfully changes.
• Check that important pages remain fast and reliably available.

Fix the source of the unwanted URLs rather than continually removing individual examples.

Keep crawl blocking separate from removing indexed pages

For URL spaces you do not want crawled, carefully scoped robots rules may be appropriate. These examples match specific search and sorting patterns:

User-agent: *
Disallow: /search?
Disallow: /*?orderby=

Test them against the URLs your application actually produces. If a page must leave the index, allow Google to fetch its noindex response. Blocking it at the same time prevents Google from reading that instruction.

Do not treat section-wide nofollow as a reliable way to make Google forget URLs already known through sitemaps or external links.

Preserve category pagination

Products beyond the first category page still need a crawlable route. Keep stable pagination URLs, self-referencing canonicals, and ordinary previous/next links. Handle duplicate filters and sorts separately rather than removing the route to deeper products.

Use accurate cache validators

ETag and Last-Modified help avoid transferring an unchanged response. Derive them from the content rather than setting a new timestamp on every request.

In a Laravel controller, a simple page whose content changes with the product record could use:

$response = response()->view('products.show', ['product' => $product]);
$response->setLastModified($product->updated_at);
$response->isNotModified($request);

return $response;

The underlying HttpFoundation method updates the response to 304 when the condition matches. Return the response object, not the method's boolean result. If the page also changes with pricing, stock, related content, or templates, its validator must account for those dependencies.

Compare the same groups after cleanup

Review request counts, unique URLs, errors, cache results, and response times for equivalent periods. Mark catalog imports, releases, and sitemap changes on the timeline so an expected shift is not mistaken for a new incident.

Use the result to check whether unnecessary requests declined and important changes are being fetched. Measure indexing and search performance separately.