crawler now implements time outs (up to 3 tries before giving up) and only parses pages that are being crawled
crawler now implements time outs (up to 3 tries before giving up) and
only parses pages that are being crawled

rouge8/hitsearch,rouge8/hitsearch,rouge8/hitsearch