If pages stop appearing in search after a launch or update, check the live robots.txt, page HTML, and response headers. A staging rule may have reached production, or a template may be adding noindex to pages that should be searchable.
These controls do different jobs. `robots.txt` limits crawling; `noindex` tells Google to exclude a page from search once it can read that instruction. A robots-blocked URL can still appear in results. Google's robots.txt guide explains the distinction.
Check the live rules
For an affected page:
- Open the production host's
/robots.txtand look for rules matching the path. Check the served file even if the repository version looks correct. - Inspect the initial HTML for a robots meta tag containing
noindex. - Check the response headers for
X-Robots-Tag: noindex. - Open URL Inspection in Search Console and compare crawl permission, fetch status, and indexing permission.
If both robots blocking and noindex are present, Google cannot read the page's directive while crawling is blocked. Choose the control that fits the page's intended use.
Look at the last relevant change
Common mistakes include:
- A launch still serving a staging
Disallow: /rule. - A shared template adding
noindexto product, service, or article pages. - A wildcard blocking more filter or pagination URLs than intended.
- A web-server or CDN rule adding an unexpected robots header.
Find the setting or template producing the live result. Changing a second copy of the rule can leave the original problem in place.
Confirm the correction
Fetch the live file and affected pages again after deployment and any required cache refresh. Verify that the intended URLs are crawlable and no longer contain an unwanted noindex directive. Test a few neighbouring paths to catch wildcard mistakes.
Use Search Console to follow later crawls and indexing changes. A successful live test confirms current access; it does not mean Google has already updated its index.