Redirect to strip a trailing period from a URL
Redirect to strip a trailing period from a URL

Story:
https://trello.com/c/JLda8Xgk/258-404s-caused-by-users-adding-a-full-stop-at-the-end-of-a-url

Strip trailing periods from URLs in our Varnish configuration, similar
to what we already do for trailing slashes, by redirecting the user to
the same URL but without the trailing period.

Ignores URLs that have query strings, which may legitimately have a
trailing period as part of a GET query parameter.

The following will redirect:

    https://www.gov.uk/foo. -> https://www.gov.uk/foo

But this will not:

    https://www.gov.uk/foo?q=mr.

This is to address a significant number of 404 errors which seem to
occur because a user enters a URL from printed media that include a full
stop immediately after the URL.

In the story above from the GOV.UK Core team, 107,000 404s were seen
over 6 months. Some common examples include:

    /dart-charge.
    /register-to-vote.
    /view-driving-licence.
    /lloydsshares.
    /visa-processing-times.

Without this redirect, these users would see a 'Page Not Found' page.

I checked several data sources to be sure that no existing content has a
trailing period (which the change in this commit) would make
inaccessible. Due to the changes we're currently making to our
publishing pipeline, there's no single source of truth that exposes all
of the URLs we used. I believe, however, that querying the databases of
the router, content-api and content-store applications, we should catch
most of not all occurrences:

URLs containing a trailing period, without a query string, receiving a
HTTP 200 response, from our CDN logs over the month of November 2015
(potentially sensitive data redacted with 'R' or 'REDACTED'):

    mattbostock@production-logs-cdn-1:/mnt/logs_cdn$ zcat -f cdn-{assets,govuk}.log-201511* | awk '$11 ~ "\.$" && $12 == '200' { print $11 }' | grep -v '\?.*\.$' | sort | uniq'
    /calculate-state-pension/y/age/female/RRRR-RR-RR.
    /calculate-statutory-sick-pay/y/none/RRR/RR/RRRR-RR-...
    /calculate-your-holiday-entitlement/y/casual-or-irregular-hoursREDACTEDhours%20worked/holiday%20hours%20taken.
    /check-uk-visa/y/jorda...
    /marriage-abroad/y/france/marriage.
    /overseas-passports/y/egypt/applying/adult/united-kingdom.
    /overseas-passports/y/japan/ap...
    /overseas-passports/y/thailand/renewing_new/adult.
    /overseas-passports/y/usa/applying/child/usa%20When%20applying%20fromREDACTED.
    /overseas-passports/y/vietnam.
    /sold-bought-vehicle/y/yes/bought-a-vehicle-into-the-trade.
    mattbostock@production-logs-cdn-1:~$

None of the above are legitimate uses of a trailing period.

The same search, this time in the router database:

    mattbostock@staging-router-backend-3:~$ mongo router --eval 'db.routes.find({}, {incoming_path:1, _id:0}).forEach(function(r) { print(r.incoming_path); });' | wc -l
    46153
    mattbostock@staging-router-backend-3:~$ mongo router --eval 'db.routes.find({}, {incoming_path:1, _id:0}).forEach(function(r) { print(r.incoming_path); });' | grep '\.$'
    /yourvotematters.
    /dart-charge.
    mattbostock@staging-router-backend-3:~$

The two routes are both used to redirect to a URL not having a trailing
period; the same intention as this pull-request.

The same search on the content-api:

    mattbostock@staging-mongo-1:~$ mongo govuk_content_production --eval 'db.artefacts.find({}, {paths:1, _id:0}).forEach(function(a) { if (a.paths) { print(a.paths.join("\n")); } });' | wc -l
    9501
    mattbostock@staging-mongo-1:~$ mongo govuk_content_production --eval 'db.artefacts.find({}, {paths:1, _id:0}).forEach(function(a) { if (a.paths) { print(a.paths.join("\n")); } });' | grep '\.$' | sort | uniq -c
    mattbostock@staging-mongo-1:~$

The same search on the content-store:

    mattbostock@staging-api-mongo-2:~$ mongo content_store_production --eval 'db.content_items.find({}, { _id:1 }).forEach(function(i){ print(i._id); });' | wc -l
    200572
    mattbostock@staging-api-mongo-2:~$ mongo content_store_production --eval 'db.content_items.find({}, { _id:1 }).forEach(function(i){ print(i._id); });' | grep '\.$'
    /yourvotematters.
    mattbostock@staging-api-mongo-2:~$

The above route is also used as a redirect.

Conclusion: we don't appear to have any legitimate content using a
trailing period in a URL, excluding URLs with query strings.

We'll make a corresponding change in the publishing-api application to
prevent new content from being created with a trailing period.

alphagov/govuk-puppet,alphagov/govuk-puppet,alphagov/govuk-puppet,alphagov/govuk-puppet,alphagov/govuk-puppet,alphagov/govuk-puppet