Tell Google not to index Via 3 pages
Tell Google not to index Via 3 pages

Add `X-Robots-Tag: noindex, nofollow` headers to all Via 3's responses.
This tells Google not to index `/proxy/static/` pages and not to follow
links on those pages. See:

* https://github.com/hypothesis/checkmate/issues/167
* https://developers.google.com/search/docs/advanced/crawling/block-indexing#http-response-header
* https://developers.google.com/search/reference/robots_meta_tag#xrobotstag

Testing
-------

You should see an `X-Robots-Tag: noindex, nofollow` header in any Via 3
response. For example:

Responses from the NGINX `/proxy/static/*` endpoint:

    curl -sSL -D - 'http://localhost:9083/proxy/static/https://h.readthedocs.io/_/downloads/en/latest/pdf/' -o /dev/null

Responses from the Pyramid app:

    curl -sSL -D - 'http://localhost:9083/route?url=https%3A%2F%2Fh.readthedocs.io%2F_%2Fdownloads%2Fen%2Flatest%2Fpdf' -o /dev/null
    curl -sSL -D - 'http://localhost:9083/pdf?url=https%3A%2F%2Fh.readthedocs.io%2F_%2Fdownloads%2Fen%2Flatest%2Fpdf' -o /dev/null

Responses from static files served by Whitenoise:

    curl -sSL -D - 'http://localhost:9083/static/ced8d00c07a038ba7e7b7077407ca60d/vendor/pdfjs-2/build/pdf.js' -o /dev/null

hypothesis/via,hypothesis/via,hypothesis/via