# robotstxt.org/

User-agent: *
Crawl-delay: 10
Disallow: /pub/
Disallow: /search/
Disallow: /cgi-bin/