Block Google from Indexing Unnecessary Search/404 Links in WordPress

Is Google indexing unnecessary pages on your WordPress site? This can harm your SEO efforts. Learn how to reclaim control of your Google index and boost your rankings by blocking unwanted links using PHP and robots.txt.

Block Google Indexing for Unwanted Pages

There are various blocking methods, such as using .htaccess, configuring nginx, or modifying the robots.txt file. However, based on my years of experience addressing this issue, I’ll share the most effective approach: utilizing PHP code and directly editing files.

Using Custom Code to Prevent Crawling and Indexing

This method will modify the meta robots tag to the value “follow, noindex” for websites using the Rankmath SEO plugin. Edit the functions.php file and add the following code:

add_filter( 'rank_math/frontend/robots', function( $robots ) {
    $url = home_url( $_SERVER['REQUEST_URI'] );

    if(strpos($url,"/search/") !== false || strpos($url,"?s=") !== false ||
        strpos($url,"404.php") !== false || is_404())  {
        $robots["index"]  = 'noindex';
    }

    if (strpos($url, "/comment-page-") !== false ||
        strpos($url, "#comments") !== false
    ) {
        $robots = [
            "index"  => "noindex",
        ];
    }

    return $robots;
});

This code snippet will prevent crawling for search pages, 404 pages, and comment pages. If you want to add other pages, simply include them within the ‘if’ condition. For example: || strpos($url,"/yourpage/") !== false.

Using the ‘follow’ and ‘noindex’ attributes allows Google’s bots to still follow the links on your page, enabling Google to better understand your overall content while only indexing the main content. If you set it to “nofollow, noindex,” Google might not be able to fully evaluate all of your content.

If you’re using other SEO plugins, you can replace the hook with the corresponding one below:

Hook:

• Rankmath : rank_math/frontend/robots
• Yoast: wpseo_init
• Aio SEO: aioseo
• Default Wordpress: wp_robots

Block Google Indexing with Robots.txt

You can also use robots.txt in conjunction with the above methods to further block crawling of unnecessary links by utilizing the ‘Disallow’ directive:

User-agent: *
Disallow: /wp-admin/
Disallow: /?s
Disallow: /?p

Take control of your WordPress site’s SEO by blocking Google from indexing unnecessary links. Utilizing PHP code and robots.txt allows you to refine what Google sees and improves your site’s overall search performance.