By Volume4
via robotstxt.org
Published: Mar 11 2007 / 22:12
The Robots Exclusion Protocol is very straightforward. In a nutshell it works like this:
When a compliant Web Robot vists a site, it first checks for a "/robots.txt" URL on the site. If this URL exists, the Robot parses its contents for directives that instruct the robot not to visit certain parts of the site.
As a Web Server Administrator you can create directives that make sense for your site. This page tells you how.
Comments
entrodus replied ago:
I dont use a robots.txt file in my site.
I dont know how bad would this be in SEO ways.
But really... i cant find i way to set a valid robots.txt
In my example, i only want robots.txt to have access to 4 specific files.
Nothing else in my whole server.
How can i achieve that, since i dont have an allow method in robots.txt specification?
Voters For This Link (10)
Voters Against This Link (0)