Blocked by Robots.txt in Search Console
search console guideHey sister, hey brother, whichever you are. “Blocked by robots.txt” means your robots.txt file told Googlebot not to crawl a page, and Googlebot obeyed. Sometimes that is exactly right. Sometimes one stray line is hiding half your shop. Here is how to tell, and how to fix it.
- Google’s definition, quoted
- a block-or-allow checklist
- the noindex trap explained
| rule | blocks | verdict |
|---|---|---|
| Disallow: /wp-admin/ | admin screens | good |
| Disallow: /cart/ | cart pages | good |
| Disallow: /products/ | every product page | too wide |
| Disallow: / | the whole site | stop |
// example data: a small online shop in Toronto. The last two lines would hide real revenue pages.
What the Status Means in One Line
Your robots.txt file forbids Googlebot from crawling the URL, so Google did not read the page. If the page should be public, edit the rule. If it should stay private, robots.txt is not the right tool.
“This page was blocked by your site’s robots.txt file.”
Google adds a note on that same page: this does not guarantee that the page won’t be indexed through some other means. To keep a page out of the index, Google’s advice is to remove the robots.txt block and use a noindex directive instead.
Robots.txt Controls Crawling, Not Indexing
A robots.txt file tells crawlers which URLs they may access. Google’s documentation says it mainly helps keep bots from overwhelming a server, and that it is not a mechanism for keeping a web page out of Google.
That sentence surprises many site owners. Picture a bakery in Leeds that puts a “staff only” sign on a door. Customers cannot walk in, but they can still hear about what is inside from someone else. In the same way, if other sites link to a blocked page, Google may list the bare URL without ever reading the page.
So there are two different jobs. To stop crawling, use robots.txt. To keep a page out of results, use noindex, or password protection, as Google’s guide recommends. Learn more in my robots.txt and sitemap guide and the noindex tag explainer.
Why Blocking and Noindex Fight Each Other
If you block a page in robots.txt, Google cannot see a noindex tag on it. Google says the page must not be blocked for noindex to work.
- Disallow in robots.txt, plus noindex in the page
- Googlebot never fetches the page
- The noindex is never read
- The bare URL can still appear in results
- Allow crawling in robots.txt
- Add noindex to the page
- Googlebot fetches it and reads the rule
- The page drops out of the index
// the rule is from Google Search Central (block indexing). The two-column layout is my own illustration.
Block It or Open It? Follow the Tree
First decide whether the page belongs in Google. The answer decides the fix.
URL blocked by robots.txt
from the report
Should it be in Google?
yes: open the rule
Which rule blocks it?
test the URL
Edit or remove it
narrowest change
Must it stay out?
allow crawl, add noindex
Retest, validate
live test, recheck
// decision aid from my own method, not an official Google flowchart.
How to Fix It Without Breaking Anything
Read your robots.txt as Google sees it, find the line that matches the URL, and change only that line. Small edits keep you safe.
1. Open the Live File
Visit yourdomain.com/robots.txt in a browser. Look for Disallow lines that match the blocked URL paths.
2. Match the Rule
Paste a blocked URL into the URL Inspection tool to confirm the status. Then compare its path with each Disallow line, from the top down.
3. Edit the Narrowest Line
Change a broad rule such as Disallow: /products/ into specific ones, or delete it. Keep the file in the site root and save a backup copy first.
4. Retest and Request
Run “Test live URL”, confirm Google can crawl it, then use “Request indexing”. Google notes a request does not guarantee indexing.
On WordPress, robots.txt may be virtual, written by a plugin or by the “discourage search engines” setting rather than a file on disk. Check the SEO plugin’s file editor before hunting through folders. A cheap SEO service reading your site should check this in minute one.
Common Mistakes and Their Fixes
Most mistakes come from using robots.txt for things it was never meant to do.
- Trying to hide private pages with Disallow
- Leaving “Disallow: /” from the staging site
- Blocking CSS and JavaScript folders
- Adding noindex to a page you also disallowed
- Use noindex or a password, as Google recommends
- Review robots.txt in every launch checklist
- Allow the files a page needs to render
- Pick one: block crawling, or noindex, not both
The classic horror story: a developer ships the staging robots.txt to the live site on Friday afternoon. It happens, and a monthly check from my technical SEO work catches it.
How to Verify the Fix Worked
The live test should say the page can be crawled. After that, wait for Google to recrawl.
First, fetch /robots.txt again and confirm the changed line is live. Then run a live test on two or three affected URLs. Finally, click Validate Fix. Google says validation typically takes up to about two weeks, so give it that long. My Validate Fix guide explains each state. If a page still stays out of Google, look at discovered, currently not indexed.
Questions People Ask
Does robots.txt keep a page out of Google?
No. Google’s documentation says robots.txt is not a mechanism for keeping a web page out of Google. Use noindex or password protection.
Can a blocked page still show up in search?
Yes. Google’s note on this status says blocking does not guarantee the page won’t be indexed through some other means, for example through links from other pages.
Should I fix every URL in this list?
No. Admin pages, carts and filter URLs are often blocked on purpose. Fix only the pages that should be public.
Why doesn’t my noindex tag work on a blocked page?
Because Googlebot cannot fetch the page, it never reads the tag. Google says the page must not be blocked by robots.txt for noindex to work.
How long until Google crawls the page after I unblock it?
It varies by site and page. Validation can take up to about two weeks per Google, and I can’t promise a specific date.
Unsure What Your File Blocks? Send Me the URL.
I’ll read it during a SEO audit. Related guides: URL marked noindex, crawl budget and the page indexing report.
// one slash too many can hide a whole site.
Sources and Last Check
- Search Console Help: Page indexing report (status definition and note)
- Google Search Central: Introduction to robots.txt (crawl control, not index control)
- Google Search Central: Block search indexing with noindex (page must not be blocked)
- Search Console Help: URL Inspection tool (live test, request indexing)
// last checked 11 Nov 2026. Example data is labeled. Opinions are marked as opinions.