[cv] reduce Datastore contention on the busiest LUCI projects.
[cv] reduce Datastore contention on the busiest LUCI projects.

Previously, if Datastore contention was detected, the TQ task will
be retried, which for busiest projects is likely to be concurrent
with another newly scheduled task, which leads to continued contention.

Now, if Datastore contention is detected, a new TQ task is created,
which provides for de-duplication, while the current task is failed
permanently. This should reduce chances of future contention.

R=yiwzhang

Change-Id: Ic3389627479e9939bb496dbfad8c09d2c058cd1a
Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/2964778
Auto-Submit: Andrii Shyshkalov <a30c74fa30536fe7ea81ed6dec202e35e149e1fd@google.com>
Commit-Queue: Yiwei Zhang <50b2a565e2e78c292794832469a30ce5abc9959c@google.com>
Reviewed-by: Yiwei Zhang <50b2a565e2e78c292794832469a30ce5abc9959c@google.com>

luci/luci-go,luci/luci-go,luci/luci-go,luci/luci-go,luci/luci-go,luci/luci-go