Added domain constructor parameter. Fixed robots.txt parsing. Added support for domain regex. Fixed bug in resolving relative paths.
Added domain constructor parameter. Fixed robots.txt parsing. Added support for domain regex. Fixed bug in resolving relative paths.

nh0815/PyCrawler