GEODE-7554: Add retry mechanism for failed tests (#4461)
GEODE-7554: Add retry mechanism for failed tests (#4461)

* GEODE-7554: Add retry mechanism for failed tests

There are still some flaky benchmarks. Retry tests that have failed, up
to 5 times, to determine if the failure is legitimate or just a flaky
test. If the test fails 5 times in a row, we know that it is a
legitimate failure.

With each run of run_against_baseline (in the apache/geode-benchmarks
repo), the class names of all failed tests will be written to a file.
The run_benchmarks script called by CI will read the failed tests from
the file and run only those tests. Once the return code is 0, or once
we've tried 5 times, exit.

* build failedFile path dynamically

Signed-off-by: Helena A. Bales <3dde782caccce990561d545a30e7ae2a545ed511@pivotal.io>

* add print of iteration number for CI debugging of failures

Signed-off-by: Robert Houghton <8ee5723e9b22f5849055c1010807733cae5c117f@pivotal.io>

* triggering CI again. Previous status never updated

Signed-off-by: Helena A. Bales <3dde782caccce990561d545a30e7ae2a545ed511@pivotal.io>

* refactor if statement for readability

* restarting CI

* remove hardcoded cluster tag

smgoller/geode,davebarnes97/geode,masaki-yamakawa/geode,jdeppe-pivotal/geode,davinash/geode,davinash/geode,jdeppe-pivotal/geode,jdeppe-pivotal/geode,masaki-yamakawa/geode,davebarnes97/geode,davinash/geode,masaki-yamakawa/geode,davinash/geode,jdeppe-pivotal/geode,smgoller/geode,davebarnes97/geode,davebarnes97/geode,davebarnes97/geode,smgoller/geode,smgoller/geode,jdeppe-pivotal/geode,davinash/geode,masaki-yamakawa/geode,jdeppe-pivotal/geode,masaki-yamakawa/geode,davinash/geode,masaki-yamakawa/geode,smgoller/geode,davebarnes97/geode,davebarnes97/geode,smgoller/geode,davinash/geode,smgoller/geode,masaki-yamakawa/geode,jdeppe-pivotal/geode