No longer downloading each project twice! The problem was that scrapy's dedupe thought that "...com/thing/foo" was different to "...com/foo/" I fixed it by removing a trailing '/' if it exists, but it may be worth doing this more inteligently, and/or sending scrapy a pull request, cause it's kinda annoying default behaviour.
No longer downloading each project twice!
The problem was that scrapy's dedupe thought that "...com/thing/foo" was
different to "...com/foo/"
I fixed it by removing a trailing '/' if it exists, but it may be worth
doing this more inteligently, and/or sending scrapy a pull request,
cause it's kinda annoying default behaviour.

Rhombik/rhombik-object-repository,Rhombik/rhombik-object-repository,Rhombik/rhombik-object-repository,Rhombik/rhombik-object-repository,Rhombik/rhombik-object-repository