By ETHiLiK
via blog.sb2.fr
Published: Dec 04 2008 / 03:56
Today an article talking about blocking Crawlers which are not Robots.txt Compliant. Number of web sites grows exponentially day by day and crawlers too ! But there is a lot of Web Crawlers that don't follow Robots.txt Disallow Rules or totally ignore it ..
Add your comment
