some servlets are very expensive, and search engines which crawl the site can trigger them (this has caused at least some of the recent 500 errors). use the standard robots.txt to prevent well behaved robots from causing trouble.
some servlets are very expensive, and search engines which crawl the
site can trigger them (this has caused at least some of the recent 500
errors). use the standard robots.txt to prevent well behaved robots
from causing trouble.

* www/robots.txt: new file.

Patch by: Ed Korthof <edk@collab.net>


git-svn-id: f8a4e5e023278da1e04e203c7fe051e3c4285d88@845630 13f79535-47bb-0310-9956-ffa450edef68

YueLinHo/Subversion,YueLinHo/Subversion,YueLinHo/Subversion,wbond/subversion,YueLinHo/Subversion,YueLinHo/Subversion,wbond/subversion,YueLinHo/Subversion,wbond/subversion,wbond/subversion,wbond/subversion,wbond/subversion,YueLinHo/Subversion,wbond/subversion,YueLinHo/Subversion