DCP traffic added to ep-perfsuite stats tests
DCP traffic added to ep-perfsuite stats tests

A key stat used during rebalance is dcp-vbtakeover.  The execution path
for this stat differs depending on whether a DCP producer exists.

Therefore in addition to measuring the performance of the stat when sets
are executing on a separate thread a second thread performing DCP work
is also created.

This ensures that all the main code paths for the stat are measured.

An example of the fast stats test output is as follows:

Running [0013/0017]: Stat latency with 100 vbuckets. Also sets & DCP traffic on separate thread (couchstore)...Latency [With 100 vbuckets & background sets & DCP] - 100 items (µs) ==================

                                 Percentile
                         Median     95th     99th  Std Dev  Histogram of samples

Sets and DCP (bg)         3.716   18.174   30.343   10.447  ▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
diskinfo                  6.914    7.398   20.073    1.342  ▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
failovers_vb0             7.444    8.447   16.677    1.238  ▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
dcp-vbtakeover           20.955   63.741 1152.475  114.774  ▁▁█▅▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
tap-vbtakeover            8.580   29.674   36.222    7.543  ▁█▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
config                  204.646  300.238  450.719   38.287  ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅█▂▃▂▁▁▁▁▁▁
kvstore                 243.871  275.771  322.513   31.639  ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂█▅▃▁▁▁
prev-vbucket            116.446  135.676  199.296   10.624  ▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
vkey_vb0                 71.398   99.295  167.079   14.484  ▁▁▁▁▁▁▁▂█▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
key_vb0                   6.346    6.768   17.530    1.558  ▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
info                      1.720    1.799   11.437    0.975  ▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
uuid                      1.751    1.840    4.890    0.330  ▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
scheduler               121.684  132.489  170.064    7.119  ▁▁▁▁▁▁▁▁▁▁▁▁▁█▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
memory                   29.250   32.191   63.808    4.144  ▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
runtimes                128.936  133.461  152.130    4.471  ▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
workload                 34.149   65.209   77.258   12.884  ▁▁▁▁█▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
timings                 130.628  145.230  174.730    7.571  ▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
vb-details_vb0          105.789  132.880  155.987   13.395  ▁▁▁▁▁▁▁▁▁▁▁█▃▄▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
warmup                   19.495   20.928   36.114    2.020  ▁▁▂█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
checkpoint_vb0           56.809   69.943   89.800    7.005  ▁▁▁▁▁▁▇█▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
dcp                     134.295  167.340  193.301   17.510  ▁▁▁▁▁▁▁▁▁▁▁▁▃█▇▆▇▅▂▁▁▁▁▁▁▁▁▁▁▁▁▁
vb-seqno_vb0             31.422   57.883   71.353   15.070  ▁▁█▄▅▇▅▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
vbucket                  93.425   97.377  117.876    3.054  ▁▁▁▁▁▁▁▁▁▁█▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
tap                      25.932   29.192   65.570    5.122  ▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
dcpagg                   16.110   34.770   62.584    9.500  ▁▁█▃▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
tapagg                    0.534    0.589    1.022    0.061  ▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
                                                            0              µs            300

(0 sec) OK

An example of the slow stats test output is as follows:

Running [0016/0017]: Stat latency with 100 vbuckets. Also sets & DCP traffic on separate thread (couchstore)...Latency [With 100 vbuckets & background sets & DCP] - 10 items (µs) ===================

                                 Percentile
                         Median     95th     99th  Std Dev  Histogram of samples

Sets and DCP (bg)         2.272   10.962   24.360   63.813  ▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
diskinfo-detail (x1e3)    2.770    4.437    4.437    0.815  ▂▂▂▂▂▂▂█▃▂▃▄▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂
failovers               596.212  761.303  761.303   85.300  ▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
allocator       (x1e3)    1.718    1.821    1.821    0.301  ▁▁▁▄▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
checkpoint      (x1e3)    1.809    2.063    2.063    0.239  ▂▂▂▂▄█▅▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂
dispatcher      (x1e3)    7.511    9.595    9.595    0.673  ▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂██▃▂▂▂▃▂▂▂▂▂▂
engine                  447.907  607.613  607.613  105.918  ▂▆█▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂
kvtimings               155.353  251.856  251.856   38.742  ▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
vb-seqno                918.976 1163.314 1163.314  172.357  ▂▂▆█▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂
vb-details      (x1e3)    7.943    9.083    9.083    1.074  ▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▅▅▅█▃▅█▅▅▃▃▃▃▃▃▃
hash            (x1e3)   11.249   12.576   12.576    0.741  ▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▃█▃▃▃
                                                            2              µs          12575

(0 sec) OK

Change-Id: I891acc2cea7986963edba21c60a6546006ae0b39
Reviewed-on: http://review.couchbase.org/70077
Reviewed-by: Dave Rigby <a09264da4832c7ff1d3bf1608a19f4b870f93750@couchbase.com>
Tested-by: buildbot <80754af91bfb6d1073585b046fe0a474ce868509@couchbase.com>

couchbase/ep-engine,daverigby/ep-engine,daverigby/ep-engine,membase/ep-engine,couchbase/ep-engine,membase/ep-engine,daverigby/kv_engine,couchbase/ep-engine,daverigby/kv_engine,daverigby/ep-engine,couchbase/ep-engine,daverigby/kv_engine,daverigby/ep-engine,daverigby/kv_engine,membase/ep-engine,membase/ep-engine