Updated README about <meta name="robots"> tags + respectRobotsTxt setting
Updated README about <meta name="robots"> tags + respectRobotsTxt setting

civalin/node-simplecrawler,simplecrawler/simplecrawler,cgiffard/node-simplecrawler,simplecrawler/simplecrawler