[Swift] add simple heuristic to detect incomplete DLOs during read ops
[Swift] add simple heuristic to detect incomplete DLOs during read ops

This is similar to waitForSegmentsToShowUp which is called during
Close/Commit. Intuitively, you wouldn't expect missing segments to be a
problem during read operations, since the previous Close/Commit
confirmed that all segments are there.

But due to the distributed nature of Swift, the read request could be
hitting a different storage node of the Swift cluster, where the
segments are still missing.

Load tests on my team's staging Swift cluster have shown this to occur
about once every 100-200 layer uploads when the Swift proxies are under
high load. The retry logic, borrowed from waitForSegmentsToShowUp, fixes
this temporary inconsistency.

Signed-off-by: Stefan Majewsky <23888fbbf34aeec099df058e607cbf00c99a1b6b@sap.com>

miminar/distribution,mikebrow/distribution,spacexnice/distribution,huamai-io/distribution,yuwaMSFT2/distribution,rbarlow/distribution,harrisonfeng/distribution,jzwlqx/distribution,sadlil/distribution,shyr/distribution,rbarlow/distribution,RichardScothern/distribution,duglin/distribution,harrisonfeng/distribution,miminar/distribution,RichardScothern/distribution,duglin/distribution,docker/distribution,vdemeester/distribution,dobril-stanga/distribution,estesp/distribution,yuwaMSFT2/distribution,dmcgowan/distribution,xiekeyang/distribution,jzwlqx/distribution,schu/distribution,mattrgreen/distribution,manoj535/distribution,xiekeyang/distribution,konnase/distribution,nwt/distribution,konnase/distribution,thaJeztah/distribution,spacexnice/distribution,Seb-Solon/distribution,manoj535/distribution,konstruktoid/distribution,stevvooe/distribution,estesp/distribution,aprajshekhar/distribution,sergeyfd/distribution,mikebrow/distribution,cezarsa/distribution,mattrgreen/distribution,mbentley/distribution,dmcgowan/distribution,cezarsa/distribution,Seb-Solon/distribution,dobril-stanga/distribution,stevvooe/distribution,aprajshekhar/distribution,sergeyfd/distribution,mbentley/distribution,sadlil/distribution,shyr/distribution,schu/distribution,jwhonce/distribution,thaJeztah/distribution,nwt/distribution,konstruktoid/distribution,huamai-io/distribution,vdemeester/distribution,docker/distribution,jwhonce/distribution