By Volume4
via robotstxt.org
Published: Mar 11 2007 / 22:12
The Robots Exclusion Protocol is very straightforward. In a nutshell it works like this:
When a compliant Web Robot vists a site, it first checks for a "/robots.txt" URL on the site. If this URL exists, the Robot parses its contents for directives that instruct the robot not to visit certain parts of the site.
As a Web Server Administrator you can create directives that make sense for your site. This page tells you how.
-
Yesod Web Framework
-
The Fortunes Of Solitude: Susan Cain On Introverts, The "New Groupthink," And The Problems With Brainstorming
-
Comparing Heap Dumps with NetBeans 7.1
-
Top 10 Project Management Influencers to Follow on the Web
-
Cross page posting Submit Form Server.Transfer methods ASP.NET
-
Cloud: Why Organisations Should Consider a Hybrid Infrastructure
Comments
entrodus replied ago:
I dont use a robots.txt file in my site.
I dont know how bad would this be in SEO ways.
But really... i cant find i way to set a valid robots.txt
In my example, i only want robots.txt to have access to 4 specific files.
Nothing else in my whole server.
How can i achieve that, since i dont have an allow method in robots.txt specification?
Voters For This Link (11)
Voters Against This Link (0)