An implementation of the compactor
An implementation of the compactor

In the scalability branch of Turnstile, buckets in the database are
stored as lists of updates.  To prevent these lists from growing
without bound, they need to be compacted from time to time into a
simple bucket record.  This is done by storing "summarize" records
in the list and placing the bucket key on a sorted set.

The compactor watches the sorted set for bucket keys of sufficient
age--bucket keys are allowed to "age" to ensure all actors see that
summarize records exist on the list, which helps to ensure that only
one compactor acts on a given bucket key.  The compactor then pops
the bucket key out of the sorted set (in an atomic fashion),
generates a compacted bucket, inserts it into the list of records in
the appropriate location, and eliminates the update records from
which it was generated.

This commit includes the implementation of the compactor.

klmitch/turnstile