Do not expose WebRobots::ParseError, but just log it and ignore the invalid robots.txt.
Do not expose WebRobots::ParseError, but just log it and ignore the invalid robots.txt.

eligoenergy/mechanize,bugmelody/mechanize,uksa/mechanize,jeremiahmarks/mechanize,jedisct1/mechanize,sparklemotion/mechanize,sjchakrav/mechanize,gchan/mechanize,jeremiahmarks/mechanize,gchan/mechanize,pwnall/mechanize,sparklemotion/mechanize,flavorjones/mechanize,uksa/mechanize,eligoenergy/mechanize,sjchakrav/mechanize,pwnall/mechanize,bugmelody/mechanize