do better detection of deep URLs that should not be added because the base URL is already in the crawl
do better detection of deep URLs that should not be added because the base URL is already in the crawl

git-svn-id: d5a8a4092b1c3eddbe100e0c5fc6446cfb12afd1@1172 fc7d47d3-c008-acd5-f51f-d19787b8a02f

paulcalvano/httparchive,HTTPArchive/httparchive,HTTPArchive/httparchive,paulcalvano/httparchive