Knowledge Base
How to Prevent Search Engines from Indexing a WordPress Site
What is SEO Optimization?
It is a set of rules for optimizing your website so that it can achieve higher rankings in the search engines’ organic results. If you have a website, blog, or online store, SEO can help you get targeted free traffic from search engines.
What are Web Crawlers or Bots?
Web crawlers, also known as web spiders or internet bots, are programs that browse the web in an automated manner for the purpose of indexing content.
Indexing of the webpages on the domains helps in improving the rankings of your website and thereby helps in achieving a great listing on the Search engine results.
We often come across issues where the Web crawler/Bots end up using most of the Website bandwidth on the server and the client would like to stop the bots from crawling the website pages.
We can surely discourage Search Engines From Indexing WordPress Sites. However, it will affect the ranking of the website on the Search engine.
There are 2 ways in which the Web crawler/Bot can be stopped from indexing the WordPress Sites. They are as mentioned below:
Discouraging Search Engines From Indexing WordPress Sites from Wordpress Admin Area
-
Login to WordPress admin area and go to Settings -> Reading.
-
Scroll down and locate the Search Engine Visibility option.
-
Check the option that says Discourage search engines from indexing this site. This image shows you how to enable the Discourage search engine from indexing this site option in WordPress
-
Save Changes, and that’s it! WordPress will automatically edit its robots.txt file for you.
Editing robots.txt File Manually
If you prefer the manual option, you can use File Manager or an FTP client to edit the robots.txt file.
-
Login to Hosting panel and locate File Manager under the Files area.
-
Go to your WordPress root directory folder (in most cases, it’s public_html for cPanel and httpdocs for Windows) and find the robots.txt file. If you can’t find it, create a new blank file.
-
Right-click on the file and select Edit. Enter the following syntax:
-----
User-agent: *
Disallow: /
-----
The code above will prevent search engines from indexing your whole site. If you want to apply the disallow rule to a specific page, write the page’s subdirectory and slug.
For example:
-----
Disallow /blog/food-review-2019.
-----
User-agent: *
Disallow: /
-----
The code above will prevent search engines from indexing your whole site. If you want to apply the disallow rule to a specific page, write the page’s subdirectory and slug.
For example:
-----
Disallow /blog/food-review-2019.
-----