[tumble/memlock] Retry memcache calls up to 5 times
[tumble/memlock] Retry memcache calls up to 5 times

A quick view at memcache reliability shows that under our usage (Dedicated 2GB instance),
it returns successfully about 99.96% of the time.
While this is okay for most applications, tumble can run for up to 10min, hitting memcache
at least once per second.  That's 600 memcache operations.

With 99.96% success rate, our expected failure rate is about 21%.

If we retry memcache failure up to 5 times, our expected failure rate approaches 0.

Bug: 795156
Change-Id: Ib9cf74ac3372a628ec597426853b98fbd7ad866f
Reviewed-on: https://chromium-review.googlesource.com/1121338
Reviewed-by: Andrii Shyshkalov <a30c74fa30536fe7ea81ed6dec202e35e149e1fd@chromium.org>
Reviewed-by: Robbie Iannucci <40f3d43a28ebae3cb819288542e1c84d73d962d5@chromium.org>
Commit-Queue: Ryan Tseng <2190fe9a39b4920ec7bd59c848ec5cccd163056f@chromium.org>

luci/luci-go,luci/luci-go,luci/luci-go,luci/luci-go,luci/luci-go,luci/luci-go