Increase Graphite disk time warning threshold
Increase Graphite disk time warning threshold

Increase the warning threshold for `graphite-1.management`'s high disk
time to 280 milliseconds, up from 200ms.

We frequently see alert noise from this check and often, no remedial
action is necessary. Looking over the last 6 months, disk time seems to
be steadily increasing:

https://graphite.production.alphagov.co.uk/render?from=09%3A00_20150120&until=17%3A00_20140720&width=900&height=600&target=movingMedian(sum(graphite-1_management_production.disk-sd%3F.disk_time.*)%2C30)&target=alias(dashed(constantLine(200))%2C%22warning%22)&target=alias(dashed(constantLine(400))%2C%22critical%22)&_uniq=0.36485243029892445

That said, our Graphite metrics lose resolution over time due to
averaging, so the increase may not be as severe as it appears. In
other words, a low baseline could cause historical spikes to be averaged
out to a lower value compared to our most recent spikes.

If we need to increase this threshold again, so we should review the
reasons for increasing disk more closely. But for now, this change will
suppress unnecessary alert noise from Icinga.

alphagov/govuk-puppet,alphagov/govuk-puppet,alphagov/govuk-puppet,alphagov/govuk-puppet,alphagov/govuk-puppet,alphagov/govuk-puppet