Make gpactivatestandby do retry loop after standby promote
Make gpactivatestandby do retry loop after standby promote

The pg_ctl promote wait logic checks for IN_PRODUCTION state in the
promoted coordinator's control file to determine if the standby
coordinator has successfully been promoted or not. Unfortunately, that
IN_PRODUCTION state does not guarantee that the promoted coordinator
is ready to accept database connections yet. Make gpactivatestandby do
a retry loop on its forced CHECKPOINT logic to guarantee that the
CHECKPOINT goes through and immediately reflects the new timeline
id. With this patch, intermittent gpactivatestandby Behave test
flakes seen frequently in the Concourse pipeline are resolved.

The logic in this patch is also actually somewhat of a revert to
previous logic as we used to do the very same retry loop. The retry
loop was removed because we wrongly assumed pg_ctl promote wait logic
would wait until database connections were ready to be accepted. The
try/catch would actually catch the error but it was being ignored
(which was intentional from the previous retry logic but wrong in the
non-retry logic).

GPDB commit references:
https://github.com/greenplum-db/gpdb/commit/1fe901d4bb9b141fd6931977eb93515982ac6edd
https://github.com/greenplum-db/gpdb/commit/5868160a73ba9cfcebdb61820635301bfb8cc0c1

greenplum-db/gpdb,greenplum-db/gpdb,greenplum-db/gpdb,greenplum-db/gpdb,greenplum-db/gpdb,greenplum-db/gpdb,greenplum-db/gpdb,greenplum-db/gpdb