Make the robots.txt fetcher act like Googlebot
Make the robots.txt fetcher act like Googlebot

- It treats too many redirections as full allow.
- It treats any 4xx error as full allow.

cf. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en

sparklemotion/mechanize,eligoenergy/mechanize,flavorjones/mechanize,sparklemotion/mechanize,eligoenergy/mechanize