Manage Indexed Pages the Right Way
- Why Use Meta Noindex Instead of Blocking with robots.txt? The Right Way to Remove Already Indexed Pages
- What Happens When You Block an Already Indexed Page with robots.txt?
- Why Meta Noindex Works in the Opposite Way
- That is why SOHO SEO does not touch robots.txt at all.
- The Correct Process for Removing an Already Indexed Page
- There Is Still a Place for robots.txt
- Conclusion
Why Use Meta Noindex Instead of Blocking with robots.txt? The Right Way to Remove Already Indexed Pages
In the previous article, I briefly mentioned that the Index Selection feature uses meta noindex rather than robots.txt.
This article takes a closer look at that decision. It was not simply a technical preference; it was a deliberate choice designed to prevent a common and consequential mistake.
When people say, “I want to keep this page out of search results,” they usually think of two options.
They can add a rule to robots.txt to block crawling, or add noindex to the page’s meta tags. Many assume the two produce roughly the same result, but I consider that one of the most common—and most dangerous—misunderstandings in SEO. The two directives operate at entirely different stages.
robots.txt says, “Do not come here.” It prevents Googlebot from accessing the page at all. meta noindex says the opposite: “You may come here, but once you do, don’t include this page in the index.” Access is allowed; only the outcome is controlled. To show why that distinction matters, let’s look at what happens in practice.
What Happens When You Block an Already Indexed Page with robots.txt?
Suppose a page has already been indexed by Google. You want it removed, so you add a blocking rule to robots.txt. Many people expect the page to disappear from search results as soon as they do.

What you may actually see in Google Search Console is a status that reads, “Indexed, though blocked by robots.txt.”
It sounds contradictory, but it really can happen. Google cannot read the page’s content, but it can see links pointing to the URL, so it may keep the URL itself in the index.
In search results, the page may appear as nothing more than a bare URL, without a title or description. Because Google cannot read the page, it has no way to verify even an instruction telling it to remove the page.
By blocking the crawl, you have paradoxically eliminated the very channel through which the removal instruction could be delivered.
I call this a “zombie index” situation: the content is dead, but its shell continues to drift through the search results.
The site administrator adds a blocking rule to robots.txt and assumes the problem is safely dealt with. In reality, nothing has been fixed; the issue has simply been left in place.
Why Meta Noindex Works in the Opposite Way

<meta name="robots" content="noindex"> tag inside the page.”} /–>
Once Google sees this instruction, it understands that the page should be removed and begins the process of taking it out of the index. Even if the page has already been indexed, Google must be able to actually read the instruction before removal can begin. That is why keeping the page crawlable is the key condition that makes removal possible.
This is how I draw the distinction: robots.txt is for things you do not want to make known as content in the first place.
For example, it is appropriate for blocking areas that have no value as content—such as an administrator login page or an internal system path—at the source, without wasting crawl budget.
<!– wp:decoblocks/highlight {"color":"green","content":"By contrast, meta noindex is for pages whose existence is already known and which may already have been indexed, but which you now want to remove. The date archives, tag archives, and search-results pages discussed in the previous article are typical examples.”} /–>Pages like these may already have been generated, linked, and indexed naturally through the course of running a site. If you block them with robots.txt only after the fact, the zombie-index problem I described above can occur all over again.
That is why SOHO SEO does not touch robots.txt at all.
This principle is reflected consistently throughout the design. SOHO SEO does not include a feature for editing the robots.txt file or adding rules to it.
When the Index Selection feature controls whether a page is indexed, its only mechanism is a meta tag delivered through the wp_robots filter. As for robots.txt, the plugin’s only role is to check the file’s current contents automatically each day through its site error-checking feature and notify you if something looks wrong.
It checks the file, but does not provide a way to edit it directly.
<!– wp:decoblocks/highlight {"content":"This is an intentional limitation. Adding a robots.txt editor would not be technically difficult, but I decided that the feature posed a greater risk of steering users in the wrong direction.”} /–>The intuitive connection—”I want to keep it out of search, so I’ll block it in robots.txt”—is actually the wrong solution far more often than people realize. I decided that removing this feature altogether and routing all index control through a single path, meta noindex, was the best way to prevent user errors at the source.
The Correct Process for Removing an Already Indexed Page
What should you do if the problem has already occurred—if a page blocked by robots.txt remains in the index as a zombie URL? I recommend the following sequence.
First, remove the blocking rule from robots.txt so that crawling is allowed again. Then add meta noindex to the page.
Wait for Googlebot to return and discover the instruction. This does not happen instantly. Depending on Googlebot’s revisit schedule, it can take anywhere from several days to several weeks.
If the page needs to disappear immediately, you can also use Search Console’s Removals tool, but that is only a temporary measure. In the long term, the issue is not truly resolved until Google has verified the meta noindex directive.
There Is Still a Place for robots.txt
That does not mean robots.txt is a useless file.
On a large site with genuinely limited crawl budget, robots.txt can still be an effective strategy for blocking areas where vast numbers of URLs unrelated to search—such as product-listing URLs with filtering parameters—are consuming Googlebot’s time.
Conclusion
robots.txt and meta noindex are not two ways of achieving the same result.
One blocks access altogether; the other permits access while controlling the outcome. If you want to actually remove a page that has already been indexed, you need the latter.
When you use the former, Google may lose the ability to revisit the page, allowing the problematic URL to remain in search results indefinitely as an empty shell.
<!– wp:decoblocks/highlight {"color":"orange","content":"Once I understood this principle, I decided not to build a robots.txt editor into SOHO SEO. I concluded that the most reliable way to protect users was to ensure that index control always follows one correct path: meta noindex.”} /–>Starting with the next article, we will move on to sitemap groups. I’ll begin by explaining how XML sitemaps that include image tags differ from WordPress’s default sitemaps.
