Don't completely die out when robots.txt not found
Don't completely die out when robots.txt not found

AlfiyaZi/nbviewer,iamjakob/nbviewer,roxyboy/nbviewer,imsparsh/nbviewer,roxyboy/nbviewer,christophelec/nbviewer,imsparsh/nbviewer,thomasyu888/nbviewer,Jay-Oh-eN/nbviewer,Sage-Bionetworks/nbviewer,bollwyvl/nbviewer,AlfiyaZi/nbviewer,Sage-Bionetworks/nbviewer,bollwyvl/nbviewer,thomasyu888/nbviewer,iamjakob/nbviewer,Jay-Oh-eN/nbviewer,christophelec/nbviewer