handle exceptions in robots.txt fetching/parsing
handle exceptions in robots.txt fetching/parsing

nlevitt/brozzler,nlevitt/brozzler,nlevitt/brozzler,internetarchive/brozzler,nlevitt/brozzler,internetarchive/brozzler,internetarchive/brozzler,internetarchive/brozzler,nlevitt/brozzler