Blog of Arifa Sultana Rumpa

Share your ideas which promote your virtual identity

Read & Comment

Getting confused with robots.txt!!!

The robots.txt file

The robots.txt file is a simple text file used to inform Googlebot about the areas of a domain that may be crawled by the search engine’s crawler and those that may not. In addition, a reference to the XML sitemap can also be included in the robots.txt file.

But most of the time we become confused about it. What should we do?? what not??

To overcome these confusion I am giving you some robots.txt samples which will lead you towards your destination.

The “User-agent: *” means this section applies to all robots. The “Disallow: /” tells the robot that it should not visit any pages on the site.

To exclude all robots from the entire server
User-agent: *
Disallow: /

To allow all robots complete access
User-agent: *

To exclude all robots from part of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

To exclude a single robot
User-agent: BadBot
Disallow: /

To allow a single robot
User-agent: Google
User-agent: *
Disallow: /

To exclude all files except one
User-agent: *
Disallow: /~joe/stuff/

After submitting your robots.txt file you wanna be sure if you are doing right or wrong!! Is your site will be crawled by Google or not??

To get relief from this there is an awesome tool to test robots.txt if your web contents can be accessed by crawlers.

Robots.txt Test Tool

Share This:

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Mr WordPress said:

"Hi, this is a comment. To delete a comment, just log in and view the post's comments. There you will have the option to edit or delete them."

richards said:

"hellonice jimrmz :-) "

Bella said:

"Get get targeted paid traffic sent to your Keep your WordPress site and plugins up-to-date - rumpa 07 cse - Full Time Web Developer | rumpa 07 cse - Full Time Web Developer website for 7 days free here:"

arifa said:

"Please share your comments."