Manage Indexed Pages the Right Way

Why Use Meta Noindex Instead of Blocking with robots.txt? The Right Way to Remove Already Indexed Pages

In the previous article, I briefly mentioned that the Index Selection feature uses meta noindex rather than robots.txt.

This article takes a closer look at that decision. It was not simply a technical preference; it was a deliberate choice designed to prevent a common and consequential mistake.

When people say, “I want to keep this page out of search results,” they usually think of two options.

They can add a rule to robots.txt to block crawling, or add noindex to the page’s meta tags. Many assume the two produce roughly the same result, but I consider that one of the most common—and most dangerous—misunderstandings in SEO. The two directives operate at entirely different stages.

robots.txt says, “Do not come here.” It prevents Googlebot from accessing the page at all. meta noindex says the opposite: “You may come here, but once you do, don’t include this page in the index.” Access is allowed; only the outcome is controlled. To show why that distinction matters, let’s look at what happens in practice.

What Happens When You Block an Already Indexed Page with robots.txt?

Suppose a page has already been indexed by Google. You want it removed, so you add a blocking rule to robots.txt. Many people expect the page to disappear from search results as soon as they do.

In practice, that often does not happen. As long as links to a URL remain elsewhere, Google may continue to recognize a URL it already knows about. Blocking the crawl does not make the fact that the URL exists disappear.

What you may actually see in Google Search Console is a status that reads, “Indexed, though blocked by robots.txt.”

It sounds contradictory, but it really can happen. Google cannot read the page’s content, but it can see links pointing to the URL, so it may keep the URL itself in the index.

In search results, the page may appear as nothing more than a bare URL, without a title or description. Because Google cannot read the page, it has no way to verify even an instruction telling it to remove the page.

By blocking the crawl, you have paradoxically eliminated the very channel through which the removal instruction could be delivered.

I call this a “zombie index” situation: the content is dead, but its shell continues to drift through the search results.

It is difficult to predict how long this will last. Google may eventually remove the URL if it determines that all links pointing to it have disappeared, but no one can say when that will happen.

The site administrator adds a blocking rule to robots.txt and assumes the problem is safely dealt with. In reality, nothing has been fixed; the issue has simply been left in place.

Why Meta Noindex Works in the Opposite Way

<!– wp:decoblocks/highlight {"color":"orange","content":"meta noindex solves the problem in precisely the opposite way. Because crawling is allowed, Googlebot continues to visit the page. Each time it does, it checks for the <meta name="robots" content="noindex"> tag inside the page.”} /–>

Once Google sees this instruction, it understands that the page should be removed and begins the process of taking it out of the index. Even if the page has already been indexed, Google must be able to actually read the instruction before removal can begin. That is why keeping the page crawlable is the key condition that makes removal possible.

This is how I draw the distinction: robots.txt is for things you do not want to make known as content in the first place.

For example, it is appropriate for blocking areas that have no value as content—such as an administrator login page or an internal system path—at the source, without wasting crawl budget.

<!– wp:decoblocks/highlight {"color":"green","content":"By contrast, meta noindex is for pages whose existence is already known and which may already have been indexed, but which you now want to remove. The date archives, tag archives, and search-results pages discussed in the previous article are typical examples.”} /–>

Pages like these may already have been generated, linked, and indexed naturally through the course of running a site. If you block them with robots.txt only after the fact, the zombie-index problem I described above can occur all over again.

That is why SOHO SEO does not touch robots.txt at all.

This principle is reflected consistently throughout the design. SOHO SEO does not include a feature for editing the robots.txt file or adding rules to it.

When the Index Selection feature controls whether a page is indexed, its only mechanism is a meta tag delivered through the wp_robots filter. As for robots.txt, the plugin’s only role is to check the file’s current contents automatically each day through its site error-checking feature and notify you if something looks wrong.

It checks the file, but does not provide a way to edit it directly.

<!– wp:decoblocks/highlight {"content":"This is an intentional limitation. Adding a robots.txt editor would not be technically difficult, but I decided that the feature posed a greater risk of steering users in the wrong direction.”} /–>

The Correct Process for Removing an Already Indexed Page

What should you do if the problem has already occurred—if a page blocked by robots.txt remains in the index as a zombie URL? I recommend the following sequence.

First, remove the blocking rule from robots.txt so that crawling is allowed again. Then add meta noindex to the page.

Wait for Googlebot to return and discover the instruction. This does not happen instantly. Depending on Googlebot’s revisit schedule, it can take anywhere from several days to several weeks.

If the page needs to disappear immediately, you can also use Search Console’s Removals tool, but that is only a temporary measure. In the long term, the issue is not truly resolved until Google has verified the meta noindex directive.

I built this sequence directly into the way the Index Selection feature works. From the moment the feature is enabled, the relevant pages remain crawlable and receive only the noindex instruction. Users do not have to manage a separate robots.txt block and meta tag while worrying about how the two interact. The system is designed to follow one correct path from the outset.

There Is Still a Place for robots.txt

That does not mean robots.txt is a useless file.

On a large site with genuinely limited crawl budget, robots.txt can still be an effective strategy for blocking areas where vast numbers of URLs unrelated to search—such as product-listing URLs with filtering parameters—are consuming Googlebot’s time.

Conclusion

robots.txt and meta noindex are not two ways of achieving the same result.

One blocks access altogether; the other permits access while controlling the outcome. If you want to actually remove a page that has already been indexed, you need the latter.

When you use the former, Google may lose the ability to revisit the page, allowing the problematic URL to remain in search results indefinitely as an empty shell.

<!– wp:decoblocks/highlight {"color":"orange","content":"Once I understood this principle, I decided not to build a robots.txt editor into SOHO SEO. I concluded that the most reliable way to protect users was to ensure that index control always follows one correct path: meta noindex.”} /–>

Starting with the next article, we will move on to sitemap groups. I’ll begin by explaining how XML sitemaps that include image tags differ from WordPress’s default sitemaps.