raft: Avoid returning errors from ProcessRaftMessage
raft: Avoid returning errors from ProcessRaftMessage

If the ProcessRaftMessage RPC returns an error, the client treats that
as a potential transport-level error, and tries to reestablish a
connection.

In some cases this can cause a feedback loop. If ProcessRaftMessage
can't successfully check the health of the sending node, it returns an
error. That causes the sending node to bounce its outgoing connection,
which results in another health check failure.

To solve this, only return an error from ProcessRaftMessage when it is
necessary to communicate to the client that it has been removed from the
cluster. Ideally, I would fix this by having the client check
specifically for a transport-level error before bouncing the connection,
but there doesn't seem to be a reliable way to do this. Transport errors
can end up with many different codes that are commonly returned by RPC
handlers, including Internal, Unavailable, FailedPrecondition,
DeadlineExceeded, and Cancelled.

Signed-off-by: Aaron Lehmann <8ecfc6017a87905413dcd7d63696a2a4c351b604@docker.com>

Noyan759/swarmkit,jlhawn/swarmkit,aaronlehmann/swarmkit,alyyousuf7/swarmkit,AliyunContainerService/swarmkit,sungwonh/swarmkit,deljuven/swarmkit,aaronlehmann/swarmkit,diogomonica/swarmkit,allencloud/swarmkit,sungwonh/swarmkit,Noyan759/swarmkit,AliyunContainerService/swarmkit,elifa/swarmkit,cheyang/swarmkit,RenaudWasTaken/swarmkit,deljuven/swarmkit,fate-grand-order/swarmkit,RenaudWasTaken/swarmkit,allencloud/swarmkit,docker/swarmkit,docker/swarmkit,diogomonica/swarmkit,dperny/swarmkit-1,cheyang/swarmkit,fate-grand-order/swarmkit,yongtang/swarmkit,jlhawn/swarmkit,yongtang/swarmkit,elifa/swarmkit,alyyousuf7/swarmkit,dperny/swarmkit-1