Fix handling of already failed deferreds when downloading page in robots.txt middleware.
Fix handling of already failed deferreds when downloading page in robots.txt middleware.

elacuesta/scrapy,Parlin-Galanodel/scrapy,cyberplant/scrapy,rolando/scrapy,scrapy/scrapy,redapple/scrapy,w495/scrapy,dracony/scrapy,rootAvish/scrapy,YeelerG/scrapy,kmike/scrapy,pawelmhm/scrapy,ArturGaspar/scrapy,shaform/scrapy,scrapy/scrapy,darkrho/scrapy-scrapy,finfish/scrapy,foromer4/scrapy,carlosp420/scrapy,arush0311/scrapy,starrify/scrapy,darkrho/scrapy-scrapy,arush0311/scrapy,rootAvish/scrapy,taito/scrapy,elacuesta/scrapy,Ryezhang/scrapy,foromer4/scrapy,starrify/scrapy,crasker/scrapy,eLRuLL/scrapy,dangra/scrapy,elacuesta/scrapy,barraponto/scrapy,ssteo/scrapy,wenyu1001/scrapy,carlosp420/scrapy,pablohoffman/scrapy,ArturGaspar/scrapy,GregoryVigoTorres/scrapy,redapple/scrapy,w495/scrapy,dangra/scrapy,umrashrf/scrapy,ArturGaspar/scrapy,starrify/scrapy,Ryezhang/scrapy,kmike/scrapy,w495/scrapy,crasker/scrapy,scrapy/scrapy,eLRuLL/scrapy,finfish/scrapy,jc0n/scrapy,dracony/scrapy,wenyu1001/scrapy,wujuguang/scrapy,dangra/scrapy,pawelmhm/scrapy,pawelmhm/scrapy,rootAvish/scrapy,GregoryVigoTorres/scrapy,wujuguang/scrapy,shaform/scrapy,rolando-contrib/scrapy,pablohoffman/scrapy,barraponto/scrapy,crasker/scrapy,shaform/scrapy,Parlin-Galanodel/scrapy,umrashrf/scrapy,redapple/scrapy,ssteo/scrapy,cyberplant/scrapy,GregoryVigoTorres/scrapy,foromer4/scrapy,YeelerG/scrapy,dracony/scrapy,pablohoffman/scrapy,taito/scrapy,rolando/scrapy,jc0n/scrapy,eLRuLL/scrapy,kmike/scrapy,rolando-contrib/scrapy,taito/scrapy,rolando-contrib/scrapy,arush0311/scrapy,carlosp420/scrapy,cyberplant/scrapy,jc0n/scrapy,barraponto/scrapy,rolando/scrapy,darkrho/scrapy-scrapy,YeelerG/scrapy,umrashrf/scrapy,Parlin-Galanodel/scrapy,Ryezhang/scrapy,wenyu1001/scrapy,ssteo/scrapy,wujuguang/scrapy,finfish/scrapy