Fix bug where checkForBatchRun() did not work if the current crawl was not currently running (eg, batch_process finished and was waiting for the next cronjob to start again). Seems like we keep getting hung trying to delete the rows from production so break the job up into even smaller jobs. For Chrome crawl start doing the "other" URLs.
Fix bug where checkForBatchRun() did not work if the current crawl was not currently running (eg, batch_process finished and was waiting for the next cronjob to start again). Seems like we keep getting hung trying to delete the rows from production so break the job up into even smaller jobs. For Chrome crawl start doing the "other" URLs.

HTTPArchive/httparchive,paulcalvano/httparchive,HTTPArchive/httparchive,paulcalvano/httparchive