compaction: do resharding through an interposer
compaction: do resharding through an interposer

Our resharding code is complex, since the compaction object has to keep
track of many output SSTables, the current shard being written.

When implementing TWCS streaming writers, we ran away from such
write-side complexity by implementing an interposer: the interposer
consumes the flat_mutation_reader stream, creating many different writer
streams. We can do a similar thing for resharding SSTables and have each
writer be guaranteed to contain keys for only a specific source shard.

As we do that, we can move the SSTable and sstable_writer information
to the compacting_sstable_writer object. The compaction object will no
longer be responsible for it and can be simplified, paving the way for
TWCS-major, which will go through an interposer as well.

Note that the compaction_writer, which now holds both the SSTable
pointer and the sstable_writer still needs to be optional. This is
because LCS (and potentially others) still want to create more than
one SSTable per source stream. That is done to guarantee that each
SSTable complies with the max_sstable_size parameter, which is
information available in the sstable_writer that is not present at
the level of the flat_mutation_reader. We want to keep it in the writer
side.

Signed-off-by: Glauber Costa <7ba9037b0dd006f9408bd45b94707becd5cd4e87@scylladb.com>

scylladb/scylla,avikivity/scylla,scylladb/scylla,scylladb/scylla,avikivity/scylla,scylladb/scylla,avikivity/scylla