[swarming] increase update_bot_info retries to 5 and hide transaction failures
[swarming] increase update_bot_info retries to 5 and hide transaction failures

The BotInfo entity is somehow very slow to update. This is weird because this is
happening for dead bots, which in practice means that the entity group has been
idle for 10 minutes. ¯\_(ツ)_/¯

Trap the CommitError exception, so that the cron job returns HTTP 200 even if
some entities were not updated. This is fine because it is run every minute, so
that it will be caught on the next run, 1 minute later. This needs to be
silenced because the failure rate becomes high enough it makes it fail the
canary, which blocks the deployment.

Change-Id: If4c9f7c80ff7fd58fcc6fa07ba887bf2964b7176
Reviewed-on: https://chromium-review.googlesource.com/1050105
Commit-Queue: Marc-Antoine Ruel <d2bf0fc09b08f7b0888b3ddab32b3e89c7122c8b@chromium.org>
Reviewed-by: Quinten Yearsley <747cf52e0cef2e3e9371802a2d344b6c87126a4b@chromium.org>

luci/luci-py,luci/luci-py,luci/luci-py,luci/luci-py