updates on robot.txt
updates on robot.txt

- add baiduspider and many other bots to blacklist
- add a minimum of crawling delay as 10 seconds

muslih/alfanous,saifmahamood/alfanous,keelhaule/alfanous,saifmahamood/alfanous,keelhaule/alfanous,keelhaule/alfanous,keelhaule/alfanous,keelhaule/alfanous,abougouffa/alfanous,abougouffa/alfanous,muslih/alfanous,saifmahamood/alfanous,muslih/alfanous,muslih/alfanous,muslih/alfanous,keelhaule/alfanous,abougouffa/alfanous,saifmahamood/alfanous,abougouffa/alfanous,abougouffa/alfanous,muslih/alfanous,abougouffa/alfanous,saifmahamood/alfanous,abougouffa/alfanous,keelhaule/alfanous,saifmahamood/alfanous,muslih/alfanous,saifmahamood/alfanous,saifmahamood/alfanous