Control request speed to avoid block Modify: 1. use Multithread only in one page instead of through multipages 2. use list comprehension to store images urls instead of for loop -> list to optimize the speed Todo: 1. generate MultiThread class for higher level abstract 2. test with multi pages for block checking
Control request speed to avoid block
Modify:
1. use Multithread only in one page instead of through multipages
2. use list comprehension to store images urls instead of for loop -> list to optimize the speed
Todo:
1. generate MultiThread class for higher level abstract
2. test with multi pages for block checking

cosystem/python_crawler_test