fix deadlock during hash table expansion
fix deadlock during hash table expansion

The slab rebalancer was paused after the LRU maintainer thread during hash
table expansion. This was mostly fine, except in one case.

If an item is in the COLD LRU when it gets hit, it gets assigned an ACTIVE
bitflag and queued for an asynchronous bump. If the item already has an
ACTIVE bit, it does not queue a second time. Items in other LRU's are bumped
when they hit the tail, but COLD items have to bump relatively soon to avoid
accidentally evicting recently active items while under memory pressure.

Items queued for async bumps have an extra refcount.

The slab page mover cannot move a page until all memory within the page is
cleared and freed. If it runs into an item which is stuck, it will try to
delete it, then wait for all of its references to be removed.

However, if an item is in the bump queue when the LRU thread is stopped,
and the slab page mover is moving the page a queued item happens to be in,
the refcount will never drop to zero as the bump will never happen.

The page mover has no mechanism for "giving up" partway through, so it loops
forever. For most use cases, this degrades the instance but doesn't actually
break much:

- The COLD LRU's will eventually drain to zero.
- Allocations will all become "direct reclaims", which force items into COLD
  and then evicts them.
- This will degrade the LRU algorithm and some performance, but if the
  instance usage isn't very high most users may never notice.
- Normally, page moving is rare enough and hash table expansion is rare
  enough that it's an unlikely race.

However, with extstore, page moves tend to happen a lot more frequently.
Also, items are only ever written to extstore from the COLD LRU's, so if they
empty the process will stop. This makes the race more likely to happen and
the effects are more obvious.

The fix is to simply pause the page mover before the LRU thread. The LRU
thread does call the page mover, but it is via trylock() since it normally
expects to progress even if the page mover is busy. The pause of the page
mover won't succeed until it fully completes the current page, so it should
then be clear to pause the LRU thread.

Further improvements could be made by allowing the page mover to bail after N
unsuccessful loops. I am also considering removing the async LRU bump
process, and letting the LRU crawler handle that instead.

Much thanks to Shashi and Sridhar from Netflix for all of their help in
tracking down this bug!

dormando/memcached,dormando/memcached,dormando/memcached,dormando/memcached,memcached/memcached,memcached/memcached,memcached/memcached,dormando/memcached,memcached/memcached,memcached/memcached