Link Details

Link 1189055 thumbnail
User 746883 avatar

By k17gaurav
Published: Aug 03 2014 / 17:55

Robots.txt file found in root directory of any website. As its name indicated Robots, It is being made for robots, crawler or web robots. It gives instructions to web robots, is called The Robots Exclusion Protocol.
  • 10
  • 0
  • 350
  • 1242
User 1266487 avatar

nickdez1999 replied ago:

0 votes Vote down Vote up Reply

if i don't have robots.text file in y will harm??

User 746883 avatar

k17gaurav replied ago:

0 votes Vote down Vote up Reply

no, if you don't have robot.txt it will not harm but if you wish not to index some section of your website by search engines then you should create a robots.txt file and place it in your website root folder.

Add your comment

Html tags not supported. Reply is editable for 5 minutes. Use [code lang="java|ruby|sql|css|xml"][/code] to post code snippets.

Apache Hadoop
Written by: Piotr Krewski
Featured Refcardz: Top Refcardz:
  1. Play
  2. Akka
  3. Design Patterns
  4. OO JS
  5. Cont. Delivery
  1. Play
  2. Java Performance
  3. Akka
  4. REST
  5. Java