Fix msg leak with DC_EACH_SAFE_QUORUM
Fix msg leak with DC_EACH_SAFE_QUORUM

It turns out that the 'swallow' field in 'struct msg' is used in
more ways than just swallowing. It also is required to basically
free any cross rack and cross DC message even though we don't want
to swallow it. And alternate methods are used to check if we should
swallow a 'msg' or not. For example, checking in multiple places if
the response is from the same DC or not and swallowing accordingly
which attributes multiple purposes to the 'swallow' field and gets
very confusing.

This incorrect use of 'swallow' has made it hard to create any new
reasonable feature without breaking a bunch of things. This fix makes
sure we don't leak msgs while using DC_EACH_SAFE_QUORUM. Even
though we don't want to swallow the responses across DCs under
DC_EACH_SAFE_QUORUM, we have to mark it as 'swallow = true' just to
have it freed in the response path.

All of this needs to be rewritten since the codebase is basically a
house of cards.

Netflix/dynomite,Netflix/dynomite,Netflix/dynomite,Netflix/dynomite