query: Why google index the pages that has been blocked on robots.txt?
positive_passages.docid: 24173
positive_passages.text: Why do Google search results include pages disallowed in robots.txt?
negative_passages.docid: 142655
negative_passages.text: Google Search Console: 404 errors on existing pages
negative_passages.docid: 100717
negative_passages.text: How can I force Google to re-index my site?
negative_passages.docid: 124047
negative_passages.text: Google reports soft 404 errors and doesn't remove several of my website URLs from the index after I 301 redirect them
negative_passages.docid: 13608
negative_passages.text: How to Remove URLs from Google Search Engine
negative_passages.docid: 94959
negative_passages.text: Screen scraping: getting around "HTTP Error 403: request disallowed by robots.txt"
negative_passages.docid: 57589
negative_passages.text: Can search engine spiders see content I add using jQuery?
negative_passages.docid: 146906
negative_passages.text: How can I use robots.txt to disallow subdomain only?
negative_passages.docid: 47679
negative_passages.text: Is it possible with Google searches to ban any and all results from a domain?
negative_passages.docid: 88032
negative_passages.text: Can I invoke Google to check my robots.txt?
negative_passages.docid: 134490
negative_passages.text: What does it mean when a file is blocked in Windows?
negative_passages.docid: 4348
negative_passages.text: Google is still crawling and indexing my old, dummy, test pages which now are 404 not found
negative_passages.docid: 138856
negative_passages.text: Any way to support users who block Google APIs?
negative_passages.docid: 108804
negative_passages.text: Warning after submitting sitemap to Google: "We found that some URLs redirect to other locations"
negative_passages.docid: 86431
negative_passages.text: Add-on for blocking sites in Google Search
negative_passages.docid: 77463
negative_passages.text: Changing URLs in Google's Index
negative_passages.docid: 102103
negative_passages.text: Removing full site from Google index