Blog

Robots.txt does not stop indexing, and blocking often makes it worse.

Almost everyone who blocks a page in robots.txt believes they have removed it from Google. They have not. They have removed Google's ability to read it, which is a different thing, and occasionally the opposite of what they wanted.

One idea, stated plainlyFrom a real Search Console statusIncludes the correct order

Two different instructions

robots.txt controls crawling. It says: do not fetch this. A noindex directive controls indexing. It says: you may fetch this, but do not list it in results. People use the first when they mean the second, constantly.

Google can index a URL it has never fetched. If enough other pages link to it with consistent anchor text, Google can conclude the URL exists and is about something, and list it on that basis alone. That is why the Search Console status "indexed, though blocked by robots.txt" exists, and it catches people out every time.

The trap

Here is the part that gets missed. Someone sees the page in Google, decides to be thorough, and adds a noindex tag to it as well while leaving the robots.txt block in place.

That cannot work, and the reason is almost funny once you see it. The noindex lives in the page. To obey it, Google has to read the page. You have just told Google not to read the page. The block prevents the crawler from ever discovering the instruction that would have removed it, so the URL stays listed indefinitely.

The order that actually works

Remove the robots.txt block first. Let Google crawl the page and see the noindex. Wait for the URL to drop out of the index. Then, if you still want to save the crawl budget, put the block back. Doing it in the other order leaves you with the worst of both.

So when is robots.txt the right tool?

When crawling itself is the cost. Infinite filter combinations, calendar pages stretching to the year 3000, internal search results, and faceted URLs that multiply without limit. You block those because you do not want the crawler spending its time there, not because you are worried about them ranking.

Use noindex when the page must not appear in results: thank-you pages, thin archives, staging content, anything with a genuine reason to stay out of search. Use robots.txt when the page must not be fetched. Occasionally you want both, in that order, with a gap in between.

The wider point

Most Search Console statuses are misread in the same direction: as errors to be driven to zero rather than as the system reporting what it did. "Indexed, though blocked" is Google telling you your two instructions disagree with each other. Some statuses genuinely need action, like crawled, currently not indexed, and some are the system working exactly as designed, like alternate page with proper canonical tag. Knowing which is which is most of the job.

Book me

Pages showing up that should not be?

Send us the URL and your robots.txt. We will tell you which of the two directives is fighting the other, and the order to unpick them in.

Reply within one working day No obligation Your details stay with us

Takes about 60 seconds. No newsletter and no CRM sequence. Your details are used to reply to this enquiry and nothing else. See the privacy notice.