osd: bind to new cluster address when wrongly marked down
osd: bind to new cluster address when wrongly marked down

If we come back up on the same address, there is a possible race.  Other
nodes will mark_down when they see us go down.  If we go up first, queue
some messages, and _then_ they see that we're down and mark_down, the
messages we queued will get lost.  Since it's stateful on the cluster
backend, we need to introduce an ordering so that closing out the _old_
session doesn't break the new session.  We do this by binding to a new
address (just a different port, actually) before marking ourselves back
up.

Fixes #592.

Signed-off-by: Sage Weil <6dd34506bbd5e58221bbb3e4732d97c91f02277b@newdream.net>

ajnelson/ceph,ajnelson/ceph,ajnelson/ceph,ajnelson/ceph,ajnelson/ceph,ajnelson/ceph