When the “Blocked by robots.txt” warning suddenly increases

When the number of URLs blocked by “robots.txt” suddenly increases in the Search Console, the following questions arise: Is my website broken?

Most of the time, numbers alone cannot be used to determine the problems with a site as a whole. First, you need to determine whether the blocked URL is an important page that must appear in search results, or a supplementary URL such as an administrator screen or a search results page.

A sudden increase in numbers does not necessarily indicate a problem.

This warning indicates that Google attempted to access a specific address, but was unable to do so due to the website’s robots.txt rules. The robots.txt file is a configuration file that controls which parts of a website search robots are allowed to access.

The site operator usually notices this number right after publishing a new post. Or they open the report after search traffic has declined and are startled, thinking, “Has Google abandoned our site?”

However, the Search Console’s page indexing report aggregates all addresses that Google is aware of. It may include addresses that should not appear in search results, such as those in the administrator interface, internal search results, filtered addresses, and duplicate addresses.

On January 11, 2021, Google stated that the status and classification of reports may change, which could result in different numbers and types of issues appearing for the same website. The number of URLs also increases when Google discovers new addresses.

The role of addresses is more important than numbers.

First, identify whether any important posts have been blocked.

The conclusion is simple: if important posts have been blocked, the blocking should be fixed. However, if only the number of secondary addresses has increased, there is no need to set the number to zero.

In particular, the robots.txt blocking feature does not completely hide content from search results. If Google discovers an address through links from other web pages, there is a possibility that the blocked address itself may appear in the search results.

If a page should not appear in search results, relying solely on robots.txt is not sufficient. It may be more appropriate to allow Google to read the content and then set the “noindex” meta tag on the page.

Conversely, if they are administrator pages, internal search results, shopping carts, or endlessly proliferating filter URLs, you may continue blocking them. Whether the main article that should appear in search is accessible is more important than the number.

The problem is not the warning itself.

There are also points that cannot be verified through direct observation.

Do not try to tackle the entire list from the beginning. Even selecting just three representative addresses can significantly narrow down the possible causes.

  1. To access the actual file, append “/robots.txt” to the end of the browser’s address.
  2. Check if there are any “Disallow” rules that block the entire site, post paths, or category paths.
  3. In the Search Console’s URL inspection tool, you can check whether the specified address is allowed to be crawled.
  4. We also check the last crawling date and the status of page fetches.
  5. Open the sample list of blocked addresses in the page index generation report to differentiate roles.

If it is difficult to determine whether an address is important, consider the title or the path. Pages that visitors need to find, such as newly written posts, service introductions, and inquiry guides, should be prioritized for review.

On the other hand, if the address appears on a search results page or is a combination of tags and filters, it is likely a secondary address. The robots.txt report allows you to check the last time Google processed a file and any errors that occurred.

The problem lies in the fact that it doesn’t end with just one rule. Multiple “Disallow” rules can overlap on the same path, and there may be differences between the actual file and the settings in WordPress or the hosting environment.

If there are many addresses, you must compare each one to determine which rule is blocking which page. Even after making direct corrections, there is no guarantee that Google will immediately revisit the page or change the search results.

If managing a large number of addresses and organizing the robots.txt rules directly within WordPress is cumbersome, you can edit the files in SOHO SEO’s robots.txt & llms.txt. However, this feature does not actually change the server settings or hosting operations; it only manages the files being edited. After making changes, you must re-check the crawling status of the canonical URL in the Search Console.

Direct inspections certainly have limitations.

If you find it tedious to constantly compare various addresses and settings, you can use general SEO tools to help organize the checking process. However, the actual modification of the robots.txt file and the server settings, which are essential for these processes, are handled by WordPress, the hosting provider, and the server configuration itself.

Managing robots.txt and subdomains with SOHO SEO.

The key to this issue is not simply to eliminate all blocked addresses, but to differentiate between rules that prevent important content and addresses that are inherently less relevant for search. In SOHO SEO’s robots.txt & llms.txt, the robots.txt file can be edited within WordPress, allowing users to view and manage multiple Disallow rules in one place.

When you need to specify auxiliary addresses to be excluded from search results, such as dates, authors, tags, or site-wide search results, the Noindex Management feature is also linked. This feature applies the search exclusion setting to the selected taxonomy archive pages, but it does not exclude individual posts and pages from the search. Therefore, you should use it separately for cases where you want to unblock important posts and for cases where you want to manage the search exclusion of auxiliary addresses. After making changes, it is necessary to separately check the crawling and indexing status.

Every method of repair comes with a cost.

Removing the rules that prevent the publication of important posts.

Delete or narrow the scope of the “Disallow” rule that is blocking access to important content. This will allow the Googlebot to re-crawl those addresses.

For you, the number of server requests may increase, and there is no guarantee that even if a modification is made, it will be immediately re-crawled. This does not mean that the changes will immediately appear in the search results.

Even if the block is removed, it will not be immediately exposed.

To exclude a page only from search results, use the “noindex” directive.

To ensure that Google verifies the content but the content does not appear in search results, you need to unblock the page in robots.txt and set the “noindex” meta tag on the page. Google states that it cannot read the “noindex” tag for pages that are blocked by robots.txt.

In other words, search exclusion and access blocking are separate issues. If these two are combined, a page may not appear in search results, and Google may not even be able to read the exclusion instructions, resulting in an ambiguous situation.

It is important to clearly state the objective for which something is being hidden.

Maintain the intended blocking as is.

If only the number of administrator screens or internal search addresses has increased, you can maintain the block. However, in that case, the number of blocked items in the Search Console may remain unchanged.

The operational goal is not to reduce all warnings to zero. It is much more important that the posts visitors need to find are accessible.

After the revision, please review the representative article again, rather than just looking at the numbers.

If an important post is included, check the paths blocked in the `/robots.txt` file, disable the corresponding rule, and then re-inspect the canonical URL. During URL inspection, you can separately check whether crawling is allowed, whether indexing is allowed, and the status of page fetches.

After modifying the robots.txt file, you can request reprocessing in the Search Console’s robots.txt report. However, it is not guaranteed that the addresses that have been unblocked will be crawled again immediately.

If the posts you want to appear in the search results are being blocked, choose to modify the rules. If only admin, search, filter, or duplicate URLs have increased, maintain the block. Otherwise, it is better to focus on managing the status of the main pages rather than the number of items.

Open important posts and manage secondary addresses according to your objectives.

Ultimately, what needs to be verified is not the total amount of warnings, but whether the representative article that should appear in the search results has been blocked. If you are a WordPress site operator who needs to repeatedly compare important articles with secondary addresses, SOHO SEO allows you to manage the robots.txt editing and the exclusion of some secondary addresses from search results in a unified manner.

Nevertheless, the actual impact of the rules must be reconfirmed through Search Console URL inspections and robots.txt reports. The key to this issue is not simply reducing the number of URLs, but rather allowing or blocking them according to the role of each address.

  • To check if there are any important posts among the blocked addresses.
  • Finding overlapping Disallow rules in robots.txt.
  • In URL inspection, view crawling and indexing status separately.
  • If the objective is to exclude content from search results, consider using the “noindex” meta tag instead of robots.txt.
  • After making corrections, re-verify the canonical URL and the robots.txt processing status.

Frequently Asked Questions.

Q. Does the search ranking decrease if the number of URLs suddenly increases?

A. An increase in numbers alone cannot determine a drop in search ranking or a problem with the entire site. There may be cases where Google has discovered new addresses or the report classification has changed, so it is important to first check which addresses have increased.

Q. If an important post has been blocked, what should be done first?

A. Append “/robots.txt” to the site address to find any conflicting “Disallow” rules, and then disable those rules for the relevant paths. After that, re-check the crawling permission and page fetch status in the Search Console URL inspection tool.

Q. Can this problem be easily solved using a tool?

A. This tool can be used as an auxiliary function to compare various addresses and settings. However, actual rule modifications must be made directly within the WordPress, hosting, and server settings, and the tool does not guarantee search engine visibility.

Q. Is it possible to manage both the robots.txt file and the noindex directive in one place?

In SOHO SEO, you can edit the robots.txt file and apply the “noindex” setting to dates, authors, tags, search results within the site, and selected taxonomy archive pages.

However, this feature does not allow you to exclude individual posts or pages from indexing. Therefore, if you want to exclude important posts from search results, you need to check the separate settings. Also, it is not a tool that guarantees crawling and indexing after you modify the robots.txt file.

WordPress-related links

https://sohoseo.luxdigest.com
https://store.luxdigest.com
https://wpcm.luxdigest.com