Block indexing of unnecessary pages in Wordpress

Blocking unnecessary pages from being indexed in WordPress is a crucial SEO technique that enhances Google crawl performance and boosts your website rankings. This practice is particularly beneficial for pages such as:

• Search pages:  Indexing internal search results pages can lead to duplicate content and negatively affect your site’s rankings.
• Download pages: Google may index download files or download pages, leading to a waste of crawl budget and providing no SEO benefits.
• 404 error pages:  Google doesn’t need to index 404 error pages because they provide no value to users.
• Admin page:  The WordPress Admin page contains sensitive information and is not intended for general users. Indexing these pages can cause security issues and affect SEO.

Effective SEO: Prevent unnecessary page indexing in WordPress

Prevent indexing with Rankmath SEO

First, open the functions.php file of the WordPress theme you are using. Then, copy and paste the code below at the end of the file:

add_filter( 'rank_math/frontend/robots', function( $robots ) {
    $url = home_url( $_SERVER['REQUEST_URI'] );
    if(strpos($url,"/search/") !== false ||
        strpos($url,"?s=") !== false ||
        substr($url, -9) === "/download" ||
        strpos($url,"404.php") !== false || is_404()
    )  {
        $robots["follow"] = 'nofollow';
    }
    return $robots;
});

The provided PHP code utilizes the Rank Math SEO plugin to prevent unnecessary pages from being indexed in WordPress. It examines the current URL to determine which pages should be blocked from indexing, targeting pages such as:

• Search results page (contains “/search/” or “?s=”)
• Download page (ends with “/download”)
• 404 error page (“404.php” or use  is_404())

Prevent indexing with YoastSEO

Code to block google indexing for Yoast SEO plugin .

add_filter( 'wpseo_robots', function( $robots) {
    $url = home_url( $_SERVER['REQUEST_URI'] );

    if(strpos($url,"/search/") !== false ||
        strpos($url,"?s=") !== false ||
        substr($url, -9) === "/download" ||
        strpos($url,"404.php") !== false || is_404()
    )  {
        $robots = array(
            'index' => 'noindex', 'follow' => 'nofollow',
            'archive' => 'noarchive', 'snippet' => 'nosnippet',
        );
    }
    return $robots;
});

Prevent indexing with AIO SEO Tool

Code to block “ google index ” for All In One Seo Pack.

add_filter( 'aioseo_robots_meta', function($robots) {
    $url = home_url( $_SERVER['REQUEST_URI'] );

    if(strpos($url,"/search/") !== false ||
        strpos($url,"?s=") !== false ||
        substr($url, -9) === "/download" ||
        strpos($url,"404.php") !== false || is_404()
    )  {
        $robots = array(
            'index' => 'noindex', 'follow' => 'nofollow',
            'archive' => 'noarchive', 'snippet' => 'nosnippet',
        );
    }
    return $robots;
});

How Do These Index Blocking Codes Work?

• When a page is determined to be blocked, the code will apply changes to that URL’s meta tag.
• It changes  "follow" to  "nofollow" for these pages.
• The command  nofollow will tell the Google BOT not to follow the links on these pages.

To verify whether you have successfully applied the code, open any of the pages you have blocked, press F12 to view the HTML, and search for the term “robots”. You will see the meta robot tag as follows: content=”noindex”, similar to the screenshot below.

Notes and benefits when blocking indexing:

Benefit:

Blocking unnecessary pages from being indexed by Google helps:

• Reduce duplicate content and improve site rankings.
• Optimize Google’s crawl budget.
• Avoid wasting time crawling on pages that aren’t useful.

Note:

• Blocking certain pages from indexing may affect how Google discovers your site.
• Make sure you only block pages that are absolutely unnecessary and don’t affect the user experience.
• You can also add “ strpos ($url,”/tag”) !== false ” or “ strpos ($url,”/comment-page-“) !== false ” to block google from indexing tags and comments discussion on the website.

Blocking unnecessary pages from being indexed in WordPress is a powerful SEO technique that enhances Google crawl performance and boosts your website rankings. Utilize the PHP code provided above to achieve this effortlessly and efficiently.