Updated robots.txt to block crawlers that were mis-behaving. Spinn3r seemed to be hitting every link whilst other crawlers seemed to be fetching all possible feeds thousands of times an hour. I've also raised the crawl-delay so that there should be less load from crawlers.
Updated robots.txt to block crawlers that were mis-behaving. Spinn3r seemed to be hitting every link whilst other crawlers seemed to be fetching all possible feeds thousands of times an hour. I've also raised the crawl-delay so that there should be less load from crawlers.
ekkleesia3/jaikuengine,niceQWER007/jaikuengine,SaturninoMateus/jaikuengine,codegooglecom/jaikuengine,patrickrolanddg/jaikuengine,justingrayston/jaikuengine,KinKir/jaikuengine,codegooglecom/jaikuengine,ridhoillyasa/jaikuengine,ekkleesia3/jaikuengine,patrickrolanddg/jaikuengine,ridhoillyasa/jaikuengine,codegooglecom/jaikuengine,rhijul/jaikuengine,seckyn/jaikuengine,ekkleesia3/jaikuengine,kraymond70/jaikuengine,rhijul/jaikuengine,kraymond70/jaikuengine,Git-Host/jaikuengine,niceQWER007/jaikuengine,rhijul/jaikuengine,ridhoillyasa/jaikuengine,niceQWER007/jaikuengine,SaturninoMateus/jaikuengine,KinKir/jaikuengine,Git-Host/jaikuengine,KinKir/jaikuengine,dharamgollapudi/jaikuengine,dharamgollapudi/jaikuengine,SaturninoMateus/jaikuengine,hj91/jaikuengine,hj91/jaikuengine,seckyn/jaikuengine,ekkleesia3/jaikuengine,seckyn/jaikuengine,dharamgollapudi/jaikuengine,ekkleesia3/jaikuengine,Git-Host/jaikuengine,kraymond70/jaikuengine,patrickrolanddg/jaikuengine,rhijul/jaikuengine,ridhoillyasa/jaikuengine,adam-pawluczuk/jaikuengine,CarolinaSGomes/jaikuengine,Git-Host/jaikuengine,rhijul/jaikuengine,CarolinaSGomes/jaikuengine,dharamgollapudi/jaikuengine,SaturninoMateus/jaikuengine,seckyn/jaikuengine,kraymond70/jaikuengine,ridhoillyasa/jaikuengine,niceQWER007/jaikuengine,niceQWER007/jaikuengine,justingrayston/jaikuengine,KinKir/jaikuengine,adam-pawluczuk/jaikuengine,adam-pawluczuk/jaikuengine,CarolinaSGomes/jaikuengine,codegooglecom/jaikuengine,patrickrolanddg/jaikuengine,justingrayston/jaikuengine,codegooglecom/jaikuengine,seckyn/jaikuengine,hj91/jaikuengine,patrickrolanddg/jaikuengine,dharamgollapudi/jaikuengine,justingrayston/jaikuengine,CarolinaSGomes/jaikuengine,justingrayston/jaikuengine,SaturninoMateus/jaikuengine,hj91/jaikuengine,KinKir/jaikuengine,CarolinaSGomes/jaikuengine,adam-pawluczuk/jaikuengine,Git-Host/jaikuengine,adam-pawluczuk/jaikuengine,kraymond70/jaikuengine,hj91/jaikuengine