fix user-agent lookup on follow-robots
fix user-agent lookup on follow-robots

dvogel/scrapelib,arowla/scrapelib,dvogel/scrapelib,JoeGermuska/scrapelib,h4ck3rm1k3/scrapelib,fgregg/scrapelib,fgregg/scrapelib,h4ck3rm1k3/scrapelib,arowla/scrapelib,olberger/scrapelib,sunlightlabs/scrapelib,jamesturk/scrapelib,olberger/scrapelib,pombredanne/scrapelib,JoeGermuska/scrapelib