MB-18426: Reduce compaction_number_of_kv_workers from 4 to 1
MB-18426: Reduce compaction_number_of_kv_workers from 4 to 1

There have been a number of instances where the IO cost of compaction
has had a detremental effect on normal "front-end" KV operations - see
MB-18426 for full details, but as one example:

  In a heavy write use case where a document is written to once, read
  once and then deleted. The documents are only stored in the bucket
  until they are processed, so in a normal state a document only
  exists for a few minutes. It was noted that when the 3 (see note)
  compactors ran the disk would become fully saturated and this would
  cause huge spikes in the disk queues up to 36 million. When the
  compaction_number_of_kv_workers was reduced, it had the following
  impacts:
  * Reduce the disk queue to ten of thousands from 36 million,
    which also reduce the amount of memory the bucket was using.
  * Reduce the disk IO.
  * Most surprisingly reducing the disk space required from 280GB to
    40GB. I suspect this was because the deletes were being blocked on
    the disk write queue.
  * The fragmentation percentages fluctuates a lot more.

Similar results have been reported in 2 other customer enviroments.

This patch reduces the default number down to 1 - it can still be
overridden if necessary via diag/eval.

  Note: While the value of compaction_number_of_kv_workers was
  previously 4, we would normally only see 3 compactors run in
  parallel in ep-engine due to not scheduling more than num_shards/2 +
  1 compactors at once - see EventuallyPersistentStore::compactDB.

Change-Id: I87bc03681f93f1c7df5535181c7fa1c640cb8e68
Reviewed-on: http://review.couchbase.org/64850
Reviewed-by: Trond Norbye <60edd2ef23891a753f231b0c6f161dc634079a93@gmail.com>
Reviewed-by: Artem Stemkovski <15a0730a27c69f62ab86f2b6c639fca202166c02@couchbase.com>
Well-Formed: buildbot <80754af91bfb6d1073585b046fe0a474ce868509@couchbase.com>
Tested-by: Dave Rigby <a09264da4832c7ff1d3bf1608a19f4b870f93750@couchbase.com>

membase/ns_server,ceejatec/ns_server,membase/ns_server,ceejatec/ns_server,ceejatec/ns_server,membase/ns_server,membase/ns_server,membase/ns_server,ceejatec/ns_server,membase/ns_server,ceejatec/ns_server,ceejatec/ns_server