A large catalog can expose far more URLs than products. Filters, sorting, internal search, and old aliases can create thousands of combinations that crawlers repeatedly request.
Before changing crawl controls, use verified crawler logs to check whether that is happening on your site. Google's crawl-budget guidance mainly applies to large or rapidly changing sites and sites with many URLs stuck at discovery.
Find the unnecessary URL patterns
Group requests by path and parameter pattern. Look for:
- filter combinations that repeat the same product set;
- multiple sort orders for a category;
- unbounded search, calendar, or archive URLs;
- old aliases and redirect chains;
- variants that show no distinct product information.
Check which templates and links expose those URLs. A large number of requests alone does not prove waste: active product pages may reasonably be crawled often.
Choose the right treatment
| Situation | Appropriate action |
|---|---|
| Obsolete URL with a relevant replacement | Redirect to that replacement |
| Duplicate that remains useful to visitors | Use a consistent canonical where appropriate |
| Unbounded URL space you do not want crawled | Consider tested robots.txt rules and stop generating unnecessary links |
| Page that must be removed from search | Allow crawling so Google can read noindex |
| Permanently removed page without a replacement | Return 404 or 410 |
noindex is not a crawl-saving instruction: Google must fetch the page to read it. Blocking the same page in robots.txt prevents that check.
Keep products reachable
Keep sitemap entries on canonical URLs you want indexed. Link important products from relevant categories, and preserve linked pagination so products beyond page one remain discoverable.
After cleanup, compare requests to unnecessary variants with requests to final product and category pages. Check response errors and new-page discovery too. Fewer duplicate requests are useful; higher rankings still need to be measured separately.