DZone Snippets is a public source code repository. Easily build up your personal collection of code snippets, categorize them with tags / keywords, and share them with the world

Snippets has posted 5883 posts at DZone. View Full User Profile

Robots.txt

10.02.2006
| 240 views |
  • submit to reddit
        // commented example of robots.txt 

# example robots.txt 
# comments start with "#"

# blocks dissallow directories for useragent
# * means all user agents!
# user agents may contain wildcards
# directory names may not!

User-agent: xyz/1.0
User-agent: ya_robot
Disallow: /special/secret/

User-agent: *
Disallow: /temp/
Disallow: /rework/