KUDU-2088: Synchronizer may not go out of scope with outstanding references
KUDU-2088: Synchronizer may not go out of scope with outstanding references

A common Kudu pattern is to declare a Synchronizer on the stack, create a
functor referencing it via AsStatusCallback, pass the functor into an
asynchronous function, wait until the functor is invoked (via Wait), then
allow the Synchronizer to go out of scope. This assumes that the async
function only invokes the StatusCallback once; a safe assumption in Kudu.

If using WaitFor instead of Wait, you must "fully wait" on the Synchronizer
before it goes out of scope. That is, WaitFor must be called repeatedly
until it no longer returns TimedOut, to ensure that whomever took the
reference has used it and will not use it again.

In this block of code, that invariant was violated. Namely, a failure in
SnoozeFailureDetector led to an early destruction of log_synchronizer,
causing a crash when the functor is invoked later. SnoozeFailureDetector
could fail if the node was elected leader while waiting on the Synchronizer.

Without the fix, raft_consensus-itest crashed in a particular way about one
or two percent of the time. With it, these crashes no longer manifest.

Change-Id: Icf9ce499616a872ad5ed4a27bd1e90e94fba5ff5
Reviewed-on: http://gerrit.cloudera.org:8080/7562
Reviewed-by: Todd Lipcon <2c763d726a8c2f6afda9be7e0e56a2caa6bb5e6b@apache.org>
Tested-by: Kudu Jenkins

InspurUSA/kudu,cloudera/kudu,EvilMcJerkface/kudu,andrwng/kudu,InspurUSA/kudu,cloudera/kudu,cloudera/kudu,EvilMcJerkface/kudu,EvilMcJerkface/kudu,andrwng/kudu,cloudera/kudu,helifu/kudu,InspurUSA/kudu,EvilMcJerkface/kudu,InspurUSA/kudu,andrwng/kudu,InspurUSA/kudu,cloudera/kudu,EvilMcJerkface/kudu,helifu/kudu,helifu/kudu,InspurUSA/kudu,andrwng/kudu,helifu/kudu,helifu/kudu,EvilMcJerkface/kudu,cloudera/kudu,EvilMcJerkface/kudu,InspurUSA/kudu,EvilMcJerkface/kudu,cloudera/kudu,helifu/kudu,InspurUSA/kudu,helifu/kudu,andrwng/kudu,helifu/kudu,EvilMcJerkface/kudu,andrwng/kudu,cloudera/kudu,cloudera/kudu,andrwng/kudu,andrwng/kudu,andrwng/kudu,helifu/kudu,InspurUSA/kudu