messaging_service: Fix send_message_timeout_and_retry
messaging_service: Fix send_message_timeout_and_retry

When a verb timeout, if we resend the message again, the peer could receive
the message more than once. This would confuse the receiver. Currently, only
the streaming code use the retry logic.

- In case of rpc:timeout_error:

Instead of doing timeout in a relatively short time and resending a few
times, we make the timeout big enough and let the tcp to do the resend.
Thus, we can avoid resending the message more than once, of course, the
receiver would not receive the message more than once.

- In case of rpc::closed_error:

There are two cases:
1) Failing to establish a connection.

For instance, the peer is down. It is safe to resend since we know for
sure the receiver hasn't received the message yet.

2) The connection is established.

We can not figure out if the remote peer have received the message
already or not upon receiving the rpc::closed_error exception.

Currently, we still sleep & resend the message again, so the receiver
might receive the message more than once. We do not have better choice
in this case, if we want the resend to recover the sending error due to
temporary network issue, since failing the whole stream_session due to
failing to send a single message is not wise.

NOTE: If the duplicated message is received when the stream_session is done,
it will be ignored since it can not find the stream_manager anymore.
For message like, STREAM_MUTATION, it is ok to receive twice (we apply the
mutation twice).

TODO: For other messages which uses the retry logic, we need
to make sure it is ok to receive more than once.

duarten/scylla,raphaelsc/scylla,raphaelsc/scylla,scylladb/scylla,scylladb/scylla,scylladb/scylla,raphaelsc/scylla,avikivity/scylla,kjniemi/scylla,avikivity/scylla,avikivity/scylla,kjniemi/scylla,duarten/scylla,scylladb/scylla,duarten/scylla,kjniemi/scylla