raft: Send messages in order
raft: Send messages in order

The integration tests found a problem when demoting the leader.
Sometimes the remaining members wouldn't know that the leader had been
demoted, and would lose quorum. This is despite the leader waiting until
the commit messages have been sent out to followers.

It turns out that all messages are sent concurrently, and this can
scramble their order. If a follower receives messages out of order, it
will reject the one that was supposed to come later. Normally this is
solved through retransmits, but that's not possible when the leader was
just demoted.

To solve this problem, and to avoid a lot of unnecessary retransmits in
general, only send one message at a time to any particular node. Do this
by setting up a chain of goroutines that each wait on the previous one
to finish. Arguably this isn't the best way to do this, and it may be
cleaner to have a long-lived goroutine for sending to each node that's
supplied messages from a channel. I tried this and found it was quite
complicated to manage the lifecycles of these goroutines, so I prefer to
go with the solution that requires fewer changes for now. We may want to
reconsider in the future as we refactor the code.

Also, TransferLeadership is the right way to handle leader demotion, so
in the future we should start using this instead of hoping that
followers receive our last messages.

Signed-off-by: Aaron Lehmann <8ecfc6017a87905413dcd7d63696a2a4c351b604@docker.com>

fate-grand-order/swarmkit,RenaudWasTaken/swarmkit,sungwonh/swarmkit,cezarsa/swarmkit,vasil-yordanov/swarmkit,alyyousuf7/swarmkit,elifa/swarmkit,alyyousuf7/swarmkit,aaronlehmann/swarmkit,elifa/swarmkit,deljuven/swarmkit,cheyang/swarmkit,vasil-yordanov/swarmkit,yongtang/swarmkit,allencloud/swarmkit,docker/swarmkit,aaronlehmann/swarmkit,deljuven/swarmkit,cheyang/swarmkit,RenaudWasTaken/swarmkit,jlhawn/swarmkit,yongtang/swarmkit,docker/swarmkit,AliyunContainerService/swarmkit,sungwonh/swarmkit,diogomonica/swarmkit,allencloud/swarmkit,diogomonica/swarmkit,AliyunContainerService/swarmkit,Noyan759/swarmkit,Noyan759/swarmkit,dperny/swarmkit-1,jlhawn/swarmkit,dperny/swarmkit-1,fate-grand-order/swarmkit,cezarsa/swarmkit