Added try/catch block around URL parsing in Crawler#getRobotsTxt redirection logic
Added try/catch block around URL parsing in Crawler#getRobotsTxt redirection logic

More in #363

simplecrawler/simplecrawler,simplecrawler/simplecrawler,cgiffard/node-simplecrawler