Do not end a crawl while urls are still being spidered because they might add more links to spiderdb when they finally complete.
Do not end a crawl while urls are still being spidered
because they might add more links to spiderdb when
they finally complete.

1yvT0s/open-source-search-engine,bowlofstew/open-source-search-engine,miketung/open-source-search-engine,1yvT0s/open-source-search-engine,yaolinz/open-source-search-engine,gigablast/open-source-search-engine,gigablast/open-source-search-engine,kalatestimine/open-source-search-engine,bowlofstew/open-source-search-engine,kalatestimine/open-source-search-engine,gigablast/open-source-search-engine,gigablast/open-source-search-engine,gigablast/open-source-search-engine,yaolinz/open-source-search-engine,miketung/open-source-search-engine,wakermahmud/open-source-search-engine,MikeLx/open-source-search-engine,MikeLx/open-source-search-engine,kalatestimine/open-source-search-engine,gigablast/open-source-search-engine,bowlofstew/open-source-search-engine,wakermahmud/open-source-search-engine,gigablast/open-source-search-engine,miketung/open-source-search-engine,miketung/open-source-search-engine,yaolinz/open-source-search-engine,wakermahmud/open-source-search-engine,yaolinz/open-source-search-engine,yaolinz/open-source-search-engine,yaolinz/open-source-search-engine,1yvT0s/open-source-search-engine,wakermahmud/open-source-search-engine,MikeLx/open-source-search-engine,1yvT0s/open-source-search-engine,wakermahmud/open-source-search-engine,kalatestimine/open-source-search-engine,1yvT0s/open-source-search-engine,MikeLx/open-source-search-engine,kalatestimine/open-source-search-engine,1yvT0s/open-source-search-engine,bowlofstew/open-source-search-engine,MikeLx/open-source-search-engine,bowlofstew/open-source-search-engine,miketung/open-source-search-engine,wakermahmud/open-source-search-engine,kalatestimine/open-source-search-engine,MikeLx/open-source-search-engine,bowlofstew/open-source-search-engine