gms/gossiper: Synchronize endpoint state destruction
gms/gossiper: Synchronize endpoint state destruction

In gossiper::handle_major_state_change() we set the endpoint_state for
a particular endpoint and replicate the changes to other cores.

This is totally unsynchronized with the execution of
gossiper::evict_from_membership(), which can happen concurrently, and
can remove the very same endpoint from the map  (in all cores).

Replicating the changes to other cores in handle_major_state_change()
can interleave with replicating the changes to other cores in
evict_from_membership(), and result in an undefined final state.

Another issue happened in debug mode dtests, where a fiber executes
handle_major_state_change(), calls into the subscribers, of which
storage_service is one, and ultimately lands on
storage_service::update_peer_info(), which iterates over the
endpoint's application state with deferring points in between (to
update a system table). gossiper::evict_from_membership() was executed
concurrently by another fiber, which freed the state the first one is
iterating over.

Signed-off-by: Duarte Nunes <a60e8b7496f7715d71354fe4c06dd24017025bf1@scylladb.com>
Message-Id: <20180318123211.3366-1-a60e8b7496f7715d71354fe4c06dd24017025bf1@scylladb.com>

scylladb/scylla,scylladb/scylla,avikivity/scylla,scylladb/scylla,avikivity/scylla,avikivity/scylla,scylladb/scylla,duarten/scylla,duarten/scylla,duarten/scylla