Cover edge case in robotsTxtFetchCondition
Cover edge case in robotsTxtFetchCondition

Sometimes, the punycode module has been known to throw an error as
the robots-parser module is determining whether to allow a URL or
not. We want to catch those errors and avoid those URL's.

simplecrawler/simplecrawler,simplecrawler/simplecrawler,cgiffard/node-simplecrawler,civalin/node-simplecrawler