storage_proxy: fix reconciliation with limits
storage_proxy: fix reconciliation with limits

Currently, if there is a disagreement between replicas we get mutations
from all of them, merge this mutations and send the result to the
client, difference between the result and the mutation sent by a
particular replica is sent back to repair it.
Unfortunately, that may not suffice to provide user with correct results
in case of disagreements.

Consider the following scenario:

create table cf(p int, c int, r int, primary key(p, c));

node1:
p=0, c=1, r=1 (timestamp = 1)
p=0, c=2, r=2 (timestamp = 2)

node2:
p=0, c=1, r=tombstone (timestamp = 2)
p=0, c=2, r=1 (timestamp = 1)

query:
select r from cf limit 1;

Let's assume there are no row markers. node1 will send only outdated
cell (p=0, c=1, r=1) while node2 will send both tombstone for c=1 and
outdated cell (p=0, c=2, r=1). A disagreement will be detected, the
replies will be merged and the coordinator will respond to the client
with result r=1, while the correct answer is r=2.

The solution proposed in this patch is to attempt to detect cases when
the problem may occur and retry queries with larger limit which result
in replicas providing more information.

The detection logic is simple: the partition key and clustering key of
the last row in the reconciled result are compared with the partition
keys and clustering keys of the last rows of replies from replicas
(except short reads). If the (pk, ck) of the replica last row is smaller
than the (pk, ck) of the reconciled result the query is retried.

Signed-off-by: Paweł Dziepak <e29bb3265626c60caf76d587cabdfcd8b379ef80@scylladb.com>

kjniemi/scylla,avikivity/scylla,avikivity/scylla,scylladb/scylla,kjniemi/scylla,scylladb/scylla,duarten/scylla,raphaelsc/scylla,raphaelsc/scylla,scylladb/scylla,duarten/scylla,avikivity/scylla,scylladb/scylla,raphaelsc/scylla,duarten/scylla,kjniemi/scylla