#Keep friendly bots out of sensitive parts of the site User-agent: * Disallow: /cgi-bin/ Disallow: .htaccess Disallow: /auth/ Disallow: /include/ Disallow: /qualitymanual/include/ Disallow: /qualitymanual/usermanage/ Disallow: /qualitymanual/ #Block unfriendly bots... User-agent: downloadexpress Disallow: / User-agent: NPBot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: Zeus Disallow: / User-agent: Terradex Disallow: / User-agent: potbot Disallow: / User-agent: LinkWalker Disallow: / #This one, despite its name, appears to be a Spam harvesting bot User-agent: Microsoft URL Control Disallow: / User-agent: Scrubby Disallow: /