Monday, March 7, 2011

How to create a /robots.txt file

You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt its basically for search engine crawler's

First of all you will have to create a txt file(robots.txt) and keep the below mentioned code which one is you needed and and upload the robots.txt in your root folder on your website server.


To exclude all robots from the entire server
Specifically, the Google robot may not index anything, because the root path (/) is disallowed.
User-agent: *
Disallow: /


To allow all robots complete access
everything is allowed for every robot nothing is disallowed.
User-agent: *
Disallow:



No comments:

Post a Comment