crawler: prevent potential race condition in test
crawler: prevent potential race condition in test

theoretically during testing if multiple requests for a robots.txt
fails to be prevented, a race condition could occur giving false
positives.

sitemapsjson/sitemapbot,sitemapsjson/sitemapbot