this fix will prevent Crawler::parse_domains from adding the same domain twice
this fix will prevent Crawler::parse_domains from adding the same domain twice

before it was possible (if a domain occured multiple times on a scanned
index page), that Crawler would add this domain multiple times to the work
queue.
This would lead to the next N processes processing this domain.

chtisgit/robot-farmer,chtisgit/robot-farmer