Add more robust retry logic to gp_replica_check, so that it can be run online.
Add more robust retry logic to gp_replica_check, so that it can be run online.

Instead of waiting for the primary and mirror to have the exact same LSN,
add logic to retry the file comparisons a few times if there are any
differences. This is a natural continuation of the earlier retry-loops I
added there, but now the LSN checks are made so that we don't even expect
the primary and mirror to sync on a particular value, and we retry not
while trying to sync the LSNs, but during the comparison itself.

This makes it possible to run gp_replica_check on a running cluster, while
modifying tables. (The extra checkpoints it emits will have a performance
impact on the other queries, though)I tested this by running pgbench at the
same time. You'll get a few NOTICEs about mismatches, but those are
harmless. After a few automatic retries, it eventually passes.

lisakowen/gpdb,greenplum-db/gpdb,50wu/gpdb,Chibin/gpdb,greenplum-db/gpdb,jmcatamney/gpdb,Chibin/gpdb,xinzweb/gpdb,adam8157/gpdb,lisakowen/gpdb,jmcatamney/gpdb,xinzweb/gpdb,Chibin/gpdb,lisakowen/gpdb,Chibin/gpdb,50wu/gpdb,greenplum-db/gpdb,jmcatamney/gpdb,adam8157/gpdb,ashwinstar/gpdb,xinzweb/gpdb,jmcatamney/gpdb,50wu/gpdb,ashwinstar/gpdb,50wu/gpdb,adam8157/gpdb,greenplum-db/gpdb,50wu/gpdb,Chibin/gpdb,Chibin/gpdb,ashwinstar/gpdb,adam8157/gpdb,adam8157/gpdb,xinzweb/gpdb,greenplum-db/gpdb,ashwinstar/gpdb,xinzweb/gpdb,Chibin/gpdb,Chibin/gpdb,jmcatamney/gpdb,lisakowen/gpdb,50wu/gpdb,greenplum-db/gpdb,lisakowen/gpdb,Chibin/gpdb,ashwinstar/gpdb,jmcatamney/gpdb,greenplum-db/gpdb,xinzweb/gpdb,Chibin/gpdb,ashwinstar/gpdb,xinzweb/gpdb,jmcatamney/gpdb,ashwinstar/gpdb,xinzweb/gpdb,greenplum-db/gpdb,50wu/gpdb,50wu/gpdb,lisakowen/gpdb,adam8157/gpdb,lisakowen/gpdb,lisakowen/gpdb,jmcatamney/gpdb,adam8157/gpdb,ashwinstar/gpdb,adam8157/gpdb