What do I put in the robots.txt file?

13 Aug 15 12:22 am

Hi Alain,


There are two ways you can create a robots.txt for your site. First is by using an SEO plugin called Yoast WordPress SEO and second is by using a plugin called WP-FileManager.

If your site already has Yoast WordPress SEO plugin installed, then you should use this tool instead for creating a robots.txt file (Part A below).

If you do not have Yoast WordPress SEO installed yet, then you can just install this plugin and follow the relevant steps below (Part B).


A. Creating robots.txt via Yoast WordPress SEO plugin:


1. Go to SEO > Tools File Editor > Create robots.txt file. The plugin will automatically create a robots.txt file for you.

2. Edit your robots.txt file based on the suggested robots.txt file below. (See Part C).

3. Once done, access your robots.txt file using the URL format http://www.yourdomain.com/robots.txt and see if it now exists. Please note that you might need to clear your browser's cache for you to see the new robots.txt file on your end. If your browser still does not show the robots.txt, use another browser or another computer instead.



B. Creating robots.txt via WP-Filemanager plugin:

1. Create your own robots.txt file on your computer using Notepad. Follow the suggested format below (See Part C).

2. Upload the generated robots.txt to your website using a plugin called WP-FileManager.

2.1 To install WP-FileManager, go to plugins > Add New and Search for the WP-Filemanager plugin.

2.2 Install the plugin developed by Anant Shrivastava and Johannes Ries.

2.3 Once installed, go to FileManager (below Settings) > Configuration and enter the following in the "Filemanager Default Home location"

../

(that's two dots followed by a forward slash, no spaces before, between or after)

2.4 Make sure to also check all checkboxes provided on that page.

2.5 Leave the rest of the fields empty and then click "Save changes".


3. Upload the robots.txt file that you created using the FileManager plugin to the home directory of your website (same level as your wp-config.php file).

4. Once done, access your robots.txt file using the URL format http://www.yourdomain.com/robots.txt and see if it now exists. Please note that you might need to clear your browser's cache for you to see the new robots.txt file on your end. If your browser still does not show the robots.txt, use another browser or another computer instead.


C. Suggested robots.txt format (replace 'www.yourdomain.com' with your actual domain name):


User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /author
Disallow: /wget/
Disallow: /httpd/
Disallow: /i/
Disallow: /f/
Disallow: /t/
Disallow: /c/
Disallow: /j/
Disallow: /page/
Disallow: /comments/

User-agent: Mediapartners-Google
Allow: /

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: ia_archiver-web.archive.org
Disallow: /

28 Sep 15 3:22 pm

mikeantiga wrote:Hi Alain,


There are two ways you can create a robots.txt for your site. First is by using an SEO plugin called Yoast WordPress SEO and second is by using a plugin called WP-FileManager.

If your site already has Yoast WordPress SEO plugin installed, then you should use this tool instead for creating a robots.txt file (Part A below).

If you do not have Yoast WordPress SEO installed yet, then you can just install this plugin and follow the relevant steps below (Part B).


A. Creating robots.txt via Yoast WordPress SEO plugin:


1. Go to SEO > Tools File Editor > Create robots.txt file. The plugin will automatically create a robots.txt file for you.

2. Edit your robots.txt file based on the suggested robots.txt file below. (See Part C).

3. Once done, access your robots.txt file using the URL format http://www.yourdomain.com/robots.txt and see if it now exists. Please note that you might need to clear your browser's cache for you to see the new robots.txt file on your end. If your browser still does not show the robots.txt, use another browser or another computer instead.



B. Creating robots.txt via WP-Filemanager plugin:

1. Create your own robots.txt file on your computer using Notepad. Follow the suggested format below (See Part C).

2. Upload the generated robots.txt to your website using a plugin called WP-FileManager.

2.1 To install WP-FileManager, go to plugins > Add New and Search for the WP-Filemanager plugin.

2.2 Install the plugin developed by Anant Shrivastava and Johannes Ries.

2.3 Once installed, go to FileManager (below Settings) > Configuration and enter the following in the "Filemanager Default Home location"

../

(that's two dots followed by a forward slash, no spaces before, between or after)

2.4 Make sure to also check all checkboxes provided on that page.

2.5 Leave the rest of the fields empty and then click "Save changes".


3. Upload the robots.txt file that you created using the FileManager plugin to the home directory of your website (same level as your wp-config.php file).

4. Once done, access your robots.txt file using the URL format http://www.yourdomain.com/robots.txt and see if it now exists. Please note that you might need to clear your browser's cache for you to see the new robots.txt file on your end. If your browser still does not show the robots.txt, use another browser or another computer instead.


C. Suggested robots.txt format (replace 'www.yourdomain.com' with your actual domain name):


User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /author
Disallow: /wget/
Disallow: /httpd/
Disallow: /i/
Disallow: /f/
Disallow: /t/
Disallow: /c/
Disallow: /j/
Disallow: /page/
Disallow: /comments/

User-agent: Mediapartners-Google
Allow: /

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: ia_archiver-web.archive.org
Disallow: /

I'm looking for this. How lucky I am. Thank you for this information