remove robots.txt restrictions
remove robots.txt restrictions

let's see how well the HRD/Python 2.7 version handles spidering

davepeck/openradar,timburks/openradar,timburks/openradar,timburks/openradar,davepeck/openradar,timburks/openradar,davepeck/openradar,timburks/openradar,davepeck/openradar