store crawl-delays of -1 so we at least know we tried to get it from robots.txt and can spider with the default 3 concurrent spiders.
store crawl-delays of -1 so we at least know
we tried to get it from robots.txt and can
spider with the default 3 concurrent spiders.

gigablast/open-source-search-engine,yaolinz/open-source-search-engine,1yvT0s/open-source-search-engine,kalatestimine/open-source-search-engine,yaolinz/open-source-search-engine,bowlofstew/open-source-search-engine,miketung/open-source-search-engine,bowlofstew/open-source-search-engine,wakermahmud/open-source-search-engine,yaolinz/open-source-search-engine,wakermahmud/open-source-search-engine,gigablast/open-source-search-engine,bowlofstew/open-source-search-engine,MikeLx/open-source-search-engine,yaolinz/open-source-search-engine,yaolinz/open-source-search-engine,kalatestimine/open-source-search-engine,bowlofstew/open-source-search-engine,1yvT0s/open-source-search-engine,1yvT0s/open-source-search-engine,kalatestimine/open-source-search-engine,gigablast/open-source-search-engine,MikeLx/open-source-search-engine,miketung/open-source-search-engine,yaolinz/open-source-search-engine,gigablast/open-source-search-engine,miketung/open-source-search-engine,gigablast/open-source-search-engine,MikeLx/open-source-search-engine,kalatestimine/open-source-search-engine,1yvT0s/open-source-search-engine,MikeLx/open-source-search-engine,wakermahmud/open-source-search-engine,miketung/open-source-search-engine,gigablast/open-source-search-engine,MikeLx/open-source-search-engine,wakermahmud/open-source-search-engine,wakermahmud/open-source-search-engine,kalatestimine/open-source-search-engine,1yvT0s/open-source-search-engine,1yvT0s/open-source-search-engine,wakermahmud/open-source-search-engine,kalatestimine/open-source-search-engine,gigablast/open-source-search-engine,bowlofstew/open-source-search-engine,bowlofstew/open-source-search-engine,MikeLx/open-source-search-engine,miketung/open-source-search-engine