Added limit parameter to PyCrawler constructor. Added limit check to search. Added function to figure out robots.txt url (not in use yet). Made resolving relative links more robust (still not great but it works).
Added limit parameter to PyCrawler constructor. Added limit check to search. Added function to figure out robots.txt url (not in use yet). Made resolving relative links more robust (still not great but it works).

nh0815/PyCrawler