HTML5 Canvas
Written by: Simon Sarris
Featured Refcardz: Top Refcardz:
  1. Apache Hadoop
  2. Web Driver
  3. MVVM
  4. REST
  5. ADO.NET
  1. HTML5
  2. Ajax
  3. jQuery Selectors
  4. CSS Part 1
  5. Git
150+ Refcardz Available · Get them all

Link Details

Previous | Next

Surprise Me!
Link 138594 thumbnail
User 355648 avatar

By ETHiLiK
via blog.sb2.fr
Published: Dec 04 2008 / 03:56

Blocking not Robots.txt Compliant Crawlers
Today an article talking about blocking Crawlers which are not Robots.txt Compliant. Number of web sites grows exponentially day by day and crawlers too ! But there is a lot of Web Crawlers that don't follow Robots.txt Disallow Rules or totally ignore it ..
Vote up
Vote down
  • 6
  • 0
  • 1460
  • 0
SaveShareSend Tags: , , ,

Add your comment


Html tags not supported. Reply is editable for 5 minutes. Use [code lang="java|ruby|sql|css|xml"][/code] to post code snippets.

Voters For This Link (6)



Voters Against This Link (0)