fixed bug in robots middleware reported by fencer in #101
fixed bug in robots middleware reported by fencer in #101

jamesblunt/scrapy,sardok/scrapy,dangra/scrapy,haiiiiiyun/scrapy,fontenele/scrapy,rahul-c1/scrapy,umrashrf/scrapy,arush0311/scrapy,Partoo/scrapy,legendtkl/scrapy,chekunkov/scrapy,hectoruelo/scrapy,jiezhu2007/scrapy,ArturGaspar/scrapy,Ryezhang/scrapy,Slater-Victoroff/scrapy,TarasRudnyk/scrapy,Preetwinder/scrapy,famorted/scrapy,taito/scrapy,mouadino/scrapy,YeelerG/scrapy,Allianzcortex/scrapy,coderabhishek/scrapy,songfj/scrapy,mlyundin/scrapy,csalazar/scrapy,pablohoffman/scrapy,Bourneer/scrapy,yidongliu/scrapy,johnardavies/scrapy,Ryezhang/scrapy,amboxer21/scrapy,mgedmin/scrapy,pranjalpatil/scrapy,ashishnerkar1/scrapy,wenyu1001/scrapy,starrify/scrapy,olorz/scrapy,Djlavoy/scrapy,kashyap32/scrapy,starrify/scrapy,crasker/scrapy,dhenyjarasandy/scrapy,agreen/scrapy,Geeglee/scrapy,hectoruelo/scrapy,lacrazyboy/scrapy,pfctdayelise/scrapy,OpenWhere/scrapy,finfish/scrapy,kalessin/scrapy,Parlin-Galanodel/scrapy,raphaelfruneaux/scrapy,dacjames/scrapy,hansenDise/scrapy,liyy7/scrapy,ENjOyAbLE1991/scrapy,kimimj/scrapy,Allianzcortex/scrapy,ylcolala/scrapy,farhan0581/scrapy,heamon7/scrapy,AaronTao1990/scrapy,huoxudong125/scrapy,w495/scrapy,cursesun/scrapy,AaronTao1990/scrapy,rootAvish/scrapy,zorojean/scrapy,bmess/scrapy,kimimj/scrapy,shaform/scrapy,wangjun/scrapy,mlyundin/scrapy,zorojean/scrapy,ssteo/scrapy,w495/scrapy,Bourneer/scrapy,webmakin/scrapy,tntC4stl3/scrapy,jiezhu2007/scrapy,famorted/scrapy,Djlavoy/scrapy,URXtech/scrapy,KublaikhanGeek/scrapy,olafdietsche/scrapy,elijah513/scrapy,farhan0581/scrapy,kimimj/scrapy,GregoryVigoTorres/scrapy,hwsyy/scrapy,CodeJuan/scrapy,dangra/scrapy,aivarsk/scrapy,rootAvish/scrapy,gbirke/scrapy,yusofm/scrapy,olafdietsche/scrapy,mgedmin/scrapy,ramiro/scrapy,bmess/scrapy,nguyenhongson03/scrapy,jeffreyjinfeng/scrapy,xiao26/scrapy,Zephor5/scrapy,stenskjaer/scrapy,pombredanne/scrapy,Allianzcortex/scrapy,scorphus/scrapy,webmakin/scrapy,dhenyjarasandy/scrapy,zjuwangg/scrapy,nikgr95/scrapy,rahulsharma1991/scrapy,devGregA/scrapy,godfreyy/scrapy,ssh-odoo/scrapy,ArturGaspar/scrapy,arush0311/scrapy,tntC4stl3/scrapy,kmike/scrapy,Geeglee/scrapy,Cnfc19932/scrapy,nowopen/scrapy,johnardavies/scrapy,fpy171/scrapy,zhangtao11/scrapy,hansenDise/scrapy,rootAvish/scrapy,elijah513/scrapy,rahul-c1/scrapy,nguyenhongson03/scrapy,arush0311/scrapy,famorted/scrapy,rklabs/scrapy,rdowinton/scrapy,wenyu1001/scrapy,curita/scrapy,ArturGaspar/scrapy,Adai0808/scrapy-1,aivarsk/scrapy,jorik041/scrapy,agusc/scrapy,cursesun/scrapy,1yvT0s/scrapy,fqul/scrapy,nowopen/scrapy,cyrixhero/scrapy,Cnfc19932/scrapy,mlyundin/scrapy,haiiiiiyun/scrapy,aivarsk/scrapy,hansenDise/scrapy,pfctdayelise/scrapy,ssh-odoo/scrapy,tagatac/scrapy,coderabhishek/scrapy,dracony/scrapy,GregoryVigoTorres/scrapy,CENDARI/scrapy,JacobStevenR/scrapy,emschorsch/scrapy,Djlavoy/scrapy,curita/scrapy,darkrho/scrapy-scrapy,wenyu1001/scrapy,yidongliu/scrapy,hwsyy/scrapy,chekunkov/scrapy,huoxudong125/scrapy,scrapy/scrapy,eLRuLL/scrapy,Timeship/scrapy,barraponto/scrapy,shaform/scrapy,cyrixhero/scrapy,kalessin/scrapy,rolando-contrib/scrapy,profjrr/scrapy,CodeJuan/scrapy,pombredanne/scrapy,tliber/scrapy,dracony/scrapy,ENjOyAbLE1991/scrapy,devGregA/scrapy,eLRuLL/scrapy,cyberplant/scrapy,moraesnicol/scrapy,Adai0808/scrapy-1,tliber/scrapy,fafaman/scrapy,webmakin/scrapy,kmike/scrapy,beni55/scrapy,Ryezhang/scrapy,eLRuLL/scrapy,zhangtao11/scrapy,Lucifer-Kim/scrapy,kmike/scrapy,heamon7/scrapy,bmess/scrapy,mgedmin/scrapy,irwinlove/scrapy,gnemoug/scrapy,cleydson/scrapy,moraesnicol/scrapy,joshlk/scrapy,nikgr95/scrapy,jorik041/scrapy,github-account-because-they-want-it/scrapy,CENDARI/scrapy,kalessin/scrapy,Digenis/scrapy,AaronTao1990/scrapy,Chenmxs/scrapy,dgillis/scrapy,YeelerG/scrapy,jc0n/scrapy,stenskjaer/scrapy,nowopen/scrapy,dgillis/scrapy,dacjames/scrapy,WilliamKinaan/scrapy,Digenis/scrapy,farhan0581/scrapy,yarikoptic/scrapy,hyrole/scrapy,yarikoptic/scrapy,olorz/scrapy,sigma-random/scrapy,wangjun/scrapy,TarasRudnyk/scrapy,Partoo/scrapy,hbwzhsh/scrapy,ssteo/scrapy,finfish/scrapy,github-account-because-they-want-it/scrapy,eliasdorneles/scrapy,agreen/scrapy,cyrixhero/scrapy,Slater-Victoroff/scrapy,legendtkl/scrapy,smaty1/scrapy,sardok/scrapy,pombredanne/scrapy,nett55/scrapy,umrashrf/scrapy,olafdietsche/scrapy,pranjalpatil/scrapy,fafaman/scrapy,joshlk/scrapy,songfj/scrapy,pablohoffman/scrapy,w495/scrapy,jamesblunt/scrapy,profjrr/scrapy,ssh-odoo/scrapy,foromer4/scrapy,avtoritet/scrapy,liyy7/scrapy,foromer4/scrapy,wujuguang/scrapy,fqul/scrapy,hwsyy/scrapy,yusofm/scrapy,ramiro/scrapy,johnardavies/scrapy,rklabs/scrapy,hyrole/scrapy,shaform/scrapy,scrapy/scrapy,github-account-because-they-want-it/scrapy,xiao26/scrapy,beni55/scrapy,godfreyy/scrapy,zjuwangg/scrapy,Adai0808/scrapy-1,agusc/scrapy,jdemaeyer/scrapy,kashyap32/scrapy,godfreyy/scrapy,Timeship/scrapy,eliasdorneles/scrapy,wzyuliyang/scrapy,tntC4stl3/scrapy,xiao26/scrapy,cleydson/scrapy,raphaelfruneaux/scrapy,fontenele/scrapy,pawelmhm/scrapy,mouadino/scrapy,emschorsch/scrapy,nett55/scrapy,scrapy/scrapy,IvanGavran/scrapy,devGregA/scrapy,nfunato/scrapy,Slater-Victoroff/scrapy,JacobStevenR/scrapy,z-fork/scrapy,JacobStevenR/scrapy,Timeship/scrapy,raphaelfruneaux/scrapy,irwinlove/scrapy,eliasdorneles/scrapy,lacrazyboy/scrapy,avtoritet/scrapy,rklabs/scrapy,nett55/scrapy,irwinlove/scrapy,beni55/scrapy,amboxer21/scrapy,curita/scrapy,rolando/scrapy,chekunkov/scrapy,ashishnerkar1/scrapy,Parlin-Galanodel/scrapy,Lucifer-Kim/scrapy,darkrho/scrapy-scrapy,WilliamKinaan/scrapy,hectoruelo/scrapy,jeffreyjinfeng/scrapy,pawelmhm/scrapy,sigma-random/scrapy,Preetwinder/scrapy,stenskjaer/scrapy,lacrazyboy/scrapy,ylcolala/scrapy,TarasRudnyk/scrapy,carlosp420/scrapy,dgillis/scrapy,fqul/scrapy,ndemir/scrapy,yidongliu/scrapy,Bourneer/scrapy,gbirke/scrapy,umrashrf/scrapy,OpenWhere/scrapy,carlosp420/scrapy,ndemir/scrapy,agusc/scrapy,z-fork/scrapy,rahul-c1/scrapy,Parlin-Galanodel/scrapy,yusofm/scrapy,finfish/scrapy,wujuguang/scrapy,zjuwangg/scrapy,pablohoffman/scrapy,scorphus/scrapy,smaty1/scrapy,rolando/scrapy,dracony/scrapy,nguyenhongson03/scrapy,ramiro/scrapy,livepy/scrapy,rolando-contrib/scrapy,taito/scrapy,jc0n/scrapy,KublaikhanGeek/scrapy,barraponto/scrapy,elacuesta/scrapy,elacuesta/scrapy,wzyuliyang/scrapy,rahulsharma1991/scrapy,elijah513/scrapy,rdowinton/scrapy,Partoo/scrapy,kazitanvirahsan/scrapy,WilliamKinaan/scrapy,Zephor5/scrapy,IvanGavran/scrapy,GregoryVigoTorres/scrapy,Cnfc19932/scrapy,codebhendi/scrapy,legendtkl/scrapy,songfj/scrapy,zorojean/scrapy,wzyuliyang/scrapy,gnemoug/scrapy,crasker/scrapy,csalazar/scrapy,emschorsch/scrapy,URXtech/scrapy,Preetwinder/scrapy,snowdream1314/scrapy,Zephor5/scrapy,dhenyjarasandy/scrapy,moraesnicol/scrapy,barraponto/scrapy,profjrr/scrapy,pawelmhm/scrapy,URXtech/scrapy,ENjOyAbLE1991/scrapy,tliber/scrapy,Geeglee/scrapy,pranjalpatil/scrapy,jc0n/scrapy,CodeJuan/scrapy,snowdream1314/scrapy,darkrho/scrapy-scrapy,KublaikhanGeek/scrapy,wangjun/scrapy,dacjames/scrapy,codebhendi/scrapy,livepy/scrapy,kazitanvirahsan/scrapy,Chenmxs/scrapy,joshlk/scrapy,olorz/scrapy,codebhendi/scrapy,1yvT0s/scrapy,liyy7/scrapy,taito/scrapy,huoxudong125/scrapy,elacuesta/scrapy,OpenWhere/scrapy,kazitanvirahsan/scrapy,carlosp420/scrapy,cursesun/scrapy,hbwzhsh/scrapy,haiiiiiyun/scrapy,CENDARI/scrapy,jdemaeyer/scrapy,zhangtao11/scrapy,coderabhishek/scrapy,redapple/scrapy,hbwzhsh/scrapy,avtoritet/scrapy,livepy/scrapy,snowdream1314/scrapy,jiezhu2007/scrapy,dangra/scrapy,jorik041/scrapy,Lucifer-Kim/scrapy,scorphus/scrapy,tagatac/scrapy,cyberplant/scrapy,z-fork/scrapy,wujuguang/scrapy,redapple/scrapy,rdowinton/scrapy,fontenele/scrapy,foromer4/scrapy,fafaman/scrapy,fpy171/scrapy,Digenis/scrapy,IvanGavran/scrapy,rolando/scrapy,zackslash/scrapy,starrify/scrapy,hyrole/scrapy,amboxer21/scrapy,smaty1/scrapy,zackslash/scrapy,nfunato/scrapy,cleydson/scrapy,Chenmxs/scrapy,jeffreyjinfeng/scrapy,redapple/scrapy,fpy171/scrapy,nfunato/scrapy,nikgr95/scrapy,kashyap32/scrapy,ylcolala/scrapy,pfctdayelise/scrapy,rahulsharma1991/scrapy,yarikoptic/scrapy,1yvT0s/scrapy,agreen/scrapy,YeelerG/scrapy,heamon7/scrapy,jdemaeyer/scrapy,ssteo/scrapy,csalazar/scrapy,cyberplant/scrapy,zackslash/scrapy,crasker/scrapy,rolando-contrib/scrapy,tagatac/scrapy