DZone Snippets is a public source code repository. Easily build up your personal collection of code snippets, categorize them with tags / keywords, and share them with the world

Snippets Manager

Snippets has posted 5883 posts at DZone. View Full User Profile

Robots.txt

10.02.2006

| 130 views |

        // commented example of robots.txt

# example robots.txt 
# comments start with "#"

# blocks dissallow directories for useragent
# * means all user agents!
# user agents may contain wildcards
# directory names may not!

User-agent: xyz/1.0
User-agent: ya_robot
Disallow: /special/secret/

User-agent: *
Disallow: /temp/
Disallow: /rework/

Tags: