Blocked by robots.txt
A robots.txt rule is preventing Googlebot from crawling this URL. That controls access to the content; it does not reliably keep the URL itself out of search.
Troubleshooting guide · Public signals and practical next steps
What this means
Search Console reports that robots.txt prevents Googlebot from fetching the URL. A blocked page’s content and its noindex directive may therefore be unavailable to Google.
When it’s normal—and when to investigate
Often expected
You may deliberately limit crawling of unnecessary filter combinations or sections that do not need crawling. Confirm the rule matches the intended paths only.
Worth investigating
A broad rule can accidentally block product pages, articles, or an entire production site. A rule copied from staging is a common place to look after launch.
Work through these checks
Open the correct robots.txt
Rules are scoped to the origin, including protocol, hostname, and port. Check the origin of the submitted URL and any redirect destination rather than assuming one file controls every version.
Find the applicable Googlebot group
Look for a specific Googlebot group as well as the wildcard group. A specific group takes precedence; do not assume every group is combined.
Match the actual path and query
Review Allow and Disallow patterns, case, wildcard characters, and end anchors. More specific matching paths take precedence, with Allow winning equivalent matches.
Check the goal before editing
If the goal is indexing, remove the unintended block. If the goal is exclusion from search, use a crawlable noindex response or appropriate access control instead of relying on robots.txt.
Common causes
- Disallow: / was left in the production file.
- A wildcard rule covers more paths than intended.
- The wrong host’s robots.txt was edited.
- A CDN serves an older file after a change.
Possible fixes
- Narrow or remove the offending rule and test representative affected URLs.
- Purge relevant caches, then confirm the publicly served file.
- Allow crawling if you need Google to process a noindex directive.
- Protect confidential content with authentication; robots.txt is publicly readable and is not access security.
Make changes that match the page’s intended purpose. A technically valid page is not guaranteed to be indexed.
What this tool can and cannot verify
It can
Fetch current robots files and evaluate relevant paths for the Googlebot product token, including redirects.
It cannot
Read Google’s cached robots file, simulate every specialized crawler, or prove that a blocked URL is absent from search.
Google’s own indexing status is available through Search Console / URL Inspection for a verified property. The live test and the recorded indexing report answer different questions.
A useful follow-up
Why could a blocked URL still appear in search?
Google can discover a URL from links even when it cannot crawl the content. Crawling restrictions and indexing exclusions are different controls.
Official reference: Google: robots.txt specification. See also the Page indexing report documentation.