Don't lose walreceiver start requests due to race condition in postmaster.
Don't lose walreceiver start requests due to race condition in postmaster.

Backport of the following commit from upstream:

commit 47fec424cbc9ea281d5bc4649ccedcba3e1116cf
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Mon Jun 26 17:31:56 2017 -0400

    Don't lose walreceiver start requests due to race condition in postmaster.

    When a walreceiver dies, the startup process will notice that and send
    a PMSIGNAL_START_WALRECEIVER signal to the postmaster, asking for a new
    walreceiver to be launched.  There's a race condition, which at least
    in HEAD is very easy to hit, whereby the postmaster might see that
    signal before it processes the SIGCHLD from the walreceiver process.
    In that situation, sigusr1_handler() just dropped the start request
    on the floor, reasoning that it must be redundant.  Eventually, after
    10 seconds (WALRCV_STARTUP_TIMEOUT), the startup process would make a
    fresh request --- but that's a long time if the connection could have
    been re-established almost immediately.

    Fix it by setting a state flag inside the postmaster that we won't
    clear until we do launch a walreceiver.  In cases where that results
    in an extra walreceiver launch, it's up to the walreceiver to realize
    it's unwanted and go away --- but we have, and need, that logic anyway
    for the opposite race case.

    I came across this through investigating unexpected delays in the
    src/test/recovery TAP tests: it manifests there in test cases where
    a master server is stopped and restarted while leaving streaming
    slaves active.

    This logic has been broken all along, so back-patch to all supported
    branches.

    Discussion: https://postgr.es/m/21344.1498494720@sss.pgh.pa.us

We'll get there anyway as part of the REL9_4_STABLE merge, but this race
has been injecting random 10-second delays into cluster startup for a
while now. Let's fix it.

For posterity, here's the order of operations that gets us into this
situation, as hinted by the above commit message:

- gpstart launches the cluster and waits for a COMMIT (which requires
  sync'd replication)
- the primary and mirror begin to come up
- mirror's startup process asks the WAL receiver to start, and
  eventually ends up waiting in WaitForWALToBecomeAvailable() on the
  XLogCtl->recoveryWakeupLatch
- postmaster sees the request and forks off a new WAL receiver, setting
  WalReceiverPID
- the new WAL receiver tries to connect to the primary, but it's told
      FATAL: the database system is starting up
  and so it starts to exit, setting the XLogCtl->recoveryWakeupLatch
- the startup process wakes up again, sees that the WAL receiver is
  stopped, and once again requests WAL receiver startup from the
  postmaster
- postmaster sees the startup request but it still has a valid
  WalReceiverPID; the request is ignored
- The dying WAL receiver process finally exits, triggering a SIGCHLD on
  the postmaster
- postmaster notes that the receiver is gone and clears WalReceiverPID
- everything comes to a halt for 10 seconds, as gpstart waits for the
  mirror, the mirror's startup process waits for a WAL receiver that is
  not actually starting, and the mirror's postmaster waits for a
  START_WALRECEIVER request that was already ignored

50wu/gpdb,lisakowen/gpdb,adam8157/gpdb,adam8157/gpdb,jmcatamney/gpdb,xinzweb/gpdb,ashwinstar/gpdb,lisakowen/gpdb,50wu/gpdb,xinzweb/gpdb,greenplum-db/gpdb,jmcatamney/gpdb,50wu/gpdb,greenplum-db/gpdb,lisakowen/gpdb,lisakowen/gpdb,adam8157/gpdb,xinzweb/gpdb,greenplum-db/gpdb,jmcatamney/gpdb,ashwinstar/gpdb,greenplum-db/gpdb,50wu/gpdb,xinzweb/gpdb,xinzweb/gpdb,xinzweb/gpdb,50wu/gpdb,ashwinstar/gpdb,ashwinstar/gpdb,50wu/gpdb,lisakowen/gpdb,jmcatamney/gpdb,lisakowen/gpdb,adam8157/gpdb,jmcatamney/gpdb,lisakowen/gpdb,xinzweb/gpdb,ashwinstar/gpdb,adam8157/gpdb,greenplum-db/gpdb,adam8157/gpdb,50wu/gpdb,jmcatamney/gpdb,ashwinstar/gpdb,greenplum-db/gpdb,adam8157/gpdb,jmcatamney/gpdb,ashwinstar/gpdb,greenplum-db/gpdb,ashwinstar/gpdb,xinzweb/gpdb,adam8157/gpdb,greenplum-db/gpdb,lisakowen/gpdb,50wu/gpdb,jmcatamney/gpdb