Indexed Though Blocked by robots.txt in 2027: What It Means and How to Fix It
search console referenceGood to see you, sir. “Indexed, though blocked by robots.txt” means Google stored a URL that your robots.txt file told it not to crawl. The fix surprises most people, because blocking a page is not the same as hiding it. Here is how to resolve it for good.
- why blocking is not hiding
- two clear fixes
- how to verify
robots.txt says Disallow
another site links to the URL
Google indexes the bare URL
result has no description
| URL | robots.txt | status |
|---|---|---|
| /thank-you/ | Disallow | Indexed, though blocked |
| /cart/ | Disallow | not indexed |
// example data for a bakery shop in Leeds. Whether a blocked URL gets indexed depends on links to it.
What “Indexed, Though Blocked by robots.txt” Means
It means Google added a page to its index even though your robots.txt file forbids crawling it. Google knows the address, but it has not read the page.
Robots.txt is a small text file at the root of your site that tells crawlers which URLs they may visit. Indexing means Google stores a page so it can appear in results. Those are two different jobs, and robots.txt only controls the first. Google’s definition in the Page indexing report is brief: “The page was indexed despite being blocked by your website’s robots.txt file.”
Google classes this as a warning, not an error. You’ll find it under the indexed pages in the report, and my page indexing report guide shows where. For the basics of the file itself, read robots.txt and sitemaps.
Why Google Indexes a Blocked URL
Robots.txt manages crawling, not indexing. If other pages link to a blocked URL, Google can still index the address without reading it.
Google states it directly: robots.txt “is not a mechanism for keeping a web page out of Google.” It also says Google “might still find and index a disallowed URL if it is linked from other places on the web.” Because Google cannot read the page, the result usually shows up with no description.
Controls Crawling
- Tells bots which URLs to visit
- Mainly avoids overloading your site with requests
Does Not Hide Pages
- Does not remove a URL from Google
- Does not stop linked URLs from being indexed
// source: Google Search Central, Introduction to robots.txt.
This is the classic trap. Someone adds Disallow to a thank-you page to “keep it out of Google.” A newsletter links to it. Google indexes the bare URL, and the owner sees the warning weeks later.
Which Fix Do You Need?
The right fix depends on one question: do you want this page in Google, or not?
Do you want it in Google?
decide per page
Remove the Disallow
let Googlebot crawl it
Add noindex
and keep the page crawlable
Add a password
or remove the page
Recrawl and check
inspect the URL
// the three branches follow Google’s guidance on keeping pages out of search. The order of the questions is mine.
How to Fix Indexed, Though Blocked
Make the page crawlable, then tell Google what you want with the right tool. Do not just delete the warning and hope.
- List the affected URLs. Open the reason in the Page indexing report and write down the pattern, such as a folder or a parameter.
- Find the Disallow rule. Open yoursite.com/robots.txt and match the rule to the URLs. On WordPress, an SEO plugin or a virtual robots.txt may generate it.
- Decide per group. Wanted pages stay indexed. Unwanted pages need removal from the index, not just a block.
- For pages you want: delete the Disallow line so Googlebot can read them. Then they can be indexed normally with a real description.
- For pages you do not want: remove the Disallow and add a noindex directive, either a meta robots tag in the page head or an X-Robots-Tag HTTP header. The header also works on PDFs and images.
- For private pages: Google’s other options are to password-protect the page or remove it entirely.
- Request a recrawl. Inspect the URL and ask Google to crawl again. See the guide on requesting indexing. Then validate the fix in the report.
Why remove the Disallow for noindex? Google says noindex only works if the page is crawlable. If robots.txt blocks it, “the crawler will never see the noindex rule,” so the page can still show up when other pages link to it. This ordering trap catches almost everyone, and it is the heart of this status.
Robots.txt vs Noindex vs Password
Each tool solves a different problem. Choose by what you want Google to do.
| robots.txt Disallow | noindex | password | |
|---|---|---|---|
| What it controls | Whether Google crawls the URL | Whether Google keeps the page in its index | Whether anyone can read the page |
| Keeps a page out of Google? | No, per Google | Yes, if Google can crawl the page | Recommended by Google |
| Needs the page to be crawlable? | It blocks crawling | Yes, or the rule is never seen | Not applicable |
// source: Google Search Central on robots.txt and on blocking indexing. The wording of the table is mine.
For the difference between noindex and canonical alternates, see noindex vs alternate page with canonical. If you need help, the technical SEO service handles robots.txt cleanups all the time.
Common Mistakes and Fixes
Almost every case comes from using robots.txt for a job it was not made for.
Using Disallow to Hide Pages
- Thank-you pages, test pages, staging copies
Fix: use noindex with crawling allowed, or a password.
Noindex and Disallow Together
- Both rules on one URL
Fix: remove the Disallow. Google can’t see a noindex rule it is not allowed to fetch.
Blocking Pages to Fix Duplicates
- Disallowing duplicate URLs
Fix: Google says disallowed URLs may still be indexed without their content. Use canonical tags or redirects. See canonical tags and duplicate content.
Expecting an Instant Change
- Checking the next morning
Fix: Google must recrawl first. If the page still appears, it may not have recrawled since you added the tag.
How to Check the Fix Worked
Check three things: the rule is gone, Google can fetch the page, and the new instruction is visible in what Google crawled.
- Open your robots.txt and confirm the Disallow line is gone or narrowed.
- Run the live test in the URL inspection tool and confirm the page is available to Google.
- View the crawled page and confirm that the noindex tag appears in the HTML Googlebot received.
- Open the Page indexing report and confirm that the URL moves to a noindex status, or to indexed if you wanted it.
If the list is long and the rules are tangled, you don’t have to untangle it alone. Many of my cheap SEO services jobs start with exactly this kind of robots.txt cleanup, and I’ll explain every change in plain English.
Blocked-but-Indexed Questions Answered
Is Indexed, though blocked by robots.txt an error?
No. Google lists it as a warning. The page is in the index, but Google could not read it, so the result usually has no description. It is worth fixing if you did not intend it.
Will adding noindex remove the page if robots.txt still blocks it?
No. Google says the crawler will never see the noindex rule on a blocked page. Remove the Disallow so Google can crawl the page and read the tag.
Does robots.txt keep a page out of Google?
No. Google says robots.txt is not a mechanism for keeping a web page out of Google, and a disallowed URL can still be indexed if other places link to it.
How do I keep a private page out of Google?
Google recommends a noindex directive for pages that can be crawled, or password protection for private ones. You can also remove the page entirely.
How long until the warning disappears?
It depends on when Google recrawls the URL. Google says validation in the Page indexing report typically takes up to about two weeks, but it can take longer.
Send Me Your robots.txt URL. I’ll Read It With You.
One WhatsApp message is enough for a first look.
Sources and Last Check
- Google Search Console Help: Page indexing report (status definition, validation)
- Google Search Central: Introduction to robots.txt
- Google Search Central: Block search indexing with noindex
- Google Search Central: Consolidate duplicate URLs
// last checked 11 Nov 2026. Example data is invented. Decision rules and the example scenario are my opinion.