sstables: fix race condition when writing to the same sstable in parallel
sstables: fix race condition when writing to the same sstable in parallel

When we are about to write a new sstable, we check if the sstable exists
by checking if respective TOC exists. That check was added to handle a
possible attempt to write a new sstable with a generation being used.
Gleb was worried that a TOC could appear after the check, and that's indeed
possible if there is an ongoing sstable write that uses the same generation
(running in parallel).
If TOC appear after the check, we would again crap an existing sstable with
a temporary, and user wouldn't be to boot scylla anymore without manual
intervention.

Then Nadav proposed the following solution:
"We could do this by the following variant of Raphael's idea:

   1. create .txt.tmp unconditionally, as before the commit 031bf57c1
(if we can't create it, fail).
   2. Now confirm that .txt does not exist. If it does, delete the .txt.tmp
we just created and fail.
   3. continue as usual
   4. and at the end, as before, rename .txt.tmp to .txt.

The key to solving the race is step 1: Since we created .txt.tmp in step 1
and know this creation succeeded, we know that we cannot be running in
parallel with another writer - because such a writer too would have tried to
create the same file, and kept it existing until the very last step of its
work (step 4)."

This patch implements the solution described above.
Let me also say that the race is theoretical and scylla wasn't affected by
it so far.

Signed-off-by: Raphael S. Carvalho <1e2942cea40e3b4d2162dfcd9e53e41c05a84d22@scylladb.com>
Message-Id: <ef630f5ac1bd0d11632c343d9f77a5f6810d18c1.1457818331.git.1e2942cea40e3b4d2162dfcd9e53e41c05a84d22@scylladb.com>

scylladb/scylla,scylladb/scylla,avikivity/scylla,raphaelsc/scylla,kjniemi/scylla,avikivity/scylla,raphaelsc/scylla,duarten/scylla,avikivity/scylla,kjniemi/scylla,raphaelsc/scylla,scylladb/scylla,scylladb/scylla,duarten/scylla,kjniemi/scylla,duarten/scylla