KUDU-1177. blocking_queue: signal condition variable outside the lock
KUDU-1177. blocking_queue: signal condition variable outside the lock

Previously, we would signal the condition variable associated with a blocking
queue from within Put() while still holding the lock. This would wake the
waiting thread, which would then often attempt to acquire the lock before the
signaling thread had released it, causing an extra context switch.

This patch changes the behavior to be like Impala and signal the queue after
releasing the lock.

To test performance, I ran:

  tablet_server-stress-test --num_inserts_per_thread=50000

...under perf stat 5 times before and after the patch, using /dev/shm as my
TEST_TMPDIR.

Perf stat results
-----------
Before:

 Performance counter stats for './build/latest/tablet_server-stress-test --num_inserts_per_thread=50000' (5 runs):

      46354.819129 task-clock                #    4.283 CPUs utilized            ( +-  0.13% )
         2,434,563 context-switches          #    0.053 M/sec                    ( +-  0.38% )
           605,424 cpu-migrations            #    0.013 M/sec                    ( +-  0.31% )
            39,693 page-faults               #    0.856 K/sec                    ( +-  0.09% )
   142,616,125,796 cycles                    #    3.077 GHz                      ( +-  0.13% )
   <not supported> stalled-cycles-frontend
   <not supported> stalled-cycles-backend
    92,344,919,302 instructions              #    0.65  insns per cycle          ( +-  0.04% )
    19,476,274,082 branches                  #  420.156 M/sec                    ( +-  0.05% )
       351,148,995 branch-misses             #    1.80% of all branches          ( +-  0.25% )

      10.822731906 seconds time elapsed                                          ( +-  0.25% )

After:

 Performance counter stats for './build/latest/tablet_server-stress-test --num_inserts_per_thread=50000' (5 runs):

      43878.037037 task-clock                #    4.286 CPUs utilized            ( +-  0.07% )
         2,119,349 context-switches          #    0.048 M/sec                    ( +-  0.10% )
           554,759 cpu-migrations            #    0.013 M/sec                    ( +-  0.21% )
            39,756 page-faults               #    0.906 K/sec                    ( +-  0.03% )
   135,238,548,757 cycles                    #    3.082 GHz                      ( +-  0.08% )
   <not supported> stalled-cycles-frontend
   <not supported> stalled-cycles-backend
    88,238,960,873 instructions              #    0.65  insns per cycle          ( +-  0.02% )
    18,423,110,833 branches                  #  419.871 M/sec                    ( +-  0.03% )
       332,238,971 branch-misses             #    1.80% of all branches          ( +-  0.17% )

      10.238239583 seconds time elapsed                                          ( +-  0.19% )

Summary:
- 5-6% reduction in wall clock, task clock, branches, branch misses, CPU cycles
- 4.5% reduction in instructions
- 13% reduction in context switches
- 8.4% reduction in CPU migrations

Throughput results
----------------
Before:
  I1027 16:36:24.356654 28988 tablet_server-stress-test.cc:105] Throughput: 37391.1 rows/sec
  I1027 16:36:24.356683 28988 tablet_server-stress-test.cc:106] CPU efficiency: 14613.9 rows/cpusec
  I1027 16:36:35.266224 29080 tablet_server-stress-test.cc:105] Throughput: 37146.6 rows/sec
  I1027 16:36:35.266245 29080 tablet_server-stress-test.cc:106] CPU efficiency: 14405.3 rows/cpusec
  I1027 16:36:46.099784 29294 tablet_server-stress-test.cc:105] Throughput: 37415 rows/sec
  I1027 16:36:46.099802 29294 tablet_server-stress-test.cc:106] CPU efficiency: 14679.6 rows/cpusec
  I1027 16:36:56.856606 29395 tablet_server-stress-test.cc:105] Throughput: 37684.4 rows/sec
  I1027 16:36:56.856624 29395 tablet_server-stress-test.cc:106] CPU efficiency: 14411.3 rows/cpusec
  I1027 16:37:07.631279 29489 tablet_server-stress-test.cc:105] Throughput: 37620.7 rows/sec
  I1027 16:37:07.631314 29489 tablet_server-stress-test.cc:106] CPU efficiency: 14447.2 rows/cpusec

After:
  I1027 16:34:30.455397 28217 tablet_server-stress-test.cc:105] Throughput: 39401.5 rows/sec
  I1027 16:34:30.455415 28217 tablet_server-stress-test.cc:106] CPU efficiency: 14609.2 rows/cpusec
  I1027 16:34:40.645938 28309 tablet_server-stress-test.cc:105] Throughput: 39820.4 rows/sec
  I1027 16:34:40.645958 28309 tablet_server-stress-test.cc:106] CPU efficiency: 14889.7 rows/cpusec
  I1027 16:34:50.857866 28400 tablet_server-stress-test.cc:105] Throughput: 39724.4 rows/sec
  I1027 16:34:50.857885 28400 tablet_server-stress-test.cc:106] CPU efficiency: 14894.3 rows/cpusec
  I1027 16:35:01.083529 28492 tablet_server-stress-test.cc:105] Throughput: 39676.6 rows/sec
  I1027 16:35:01.083552 28492 tablet_server-stress-test.cc:106] CPU efficiency: 14674.3 rows/cpusec
  I1027 16:35:11.351135 28583 tablet_server-stress-test.cc:105] Throughput: 39503.8 rows/sec
  I1027 16:35:11.351153 28583 tablet_server-stress-test.cc:106] CPU efficiency: 14684.3 rows/cpusec

Summary:
  Throughput: 5.8% improvement
  CPU efficiency: 1.6% improvement

Latency numbers:
----------------
75th:
  /tmp/without-change.txt:    "percentile_75": 228,
  /tmp/without-change.txt:    "percentile_75": 229,
  /tmp/without-change.txt:    "percentile_75": 227,
  /tmp/without-change.txt:    "percentile_75": 226,
  /tmp/without-change.txt:    "percentile_75": 226,
  /tmp/with-queue-change.txt:    "percentile_75": 214,
  /tmp/with-queue-change.txt:    "percentile_75": 213,
  /tmp/with-queue-change.txt:    "percentile_75": 213,
  /tmp/with-queue-change.txt:    "percentile_75": 213,
  /tmp/with-queue-change.txt:    "percentile_75": 214,
(~5.8% reduction)

99th:
  /tmp/without-change.txt:    "percentile_99": 314,
  /tmp/without-change.txt:    "percentile_99": 326,
  /tmp/without-change.txt:    "percentile_99": 322,
  /tmp/without-change.txt:    "percentile_99": 312,
  /tmp/without-change.txt:    "percentile_99": 316,
  /tmp/with-queue-change.txt:    "percentile_99": 296,
  /tmp/with-queue-change.txt:    "percentile_99": 288,
  /tmp/with-queue-change.txt:    "percentile_99": 288,
  /tmp/with-queue-change.txt:    "percentile_99": 290,
  /tmp/with-queue-change.txt:    "percentile_99": 298,
~8% reduction

99.9th:
  /tmp/without-change.txt:    "percentile_99_9": 784,
  /tmp/without-change.txt:    "percentile_99_9": 1072,
  /tmp/without-change.txt:    "percentile_99_9": 848,
  /tmp/without-change.txt:    "percentile_99_9": 808,
  /tmp/without-change.txt:    "percentile_99_9": 808,
  /tmp/with-queue-change.txt:    "percentile_99_9": 884,
  /tmp/with-queue-change.txt:    "percentile_99_9": 704,
  /tmp/with-queue-change.txt:    "percentile_99_9": 716,
  /tmp/with-queue-change.txt:    "percentile_99_9": 744,
  /tmp/with-queue-change.txt:    "percentile_99_9": 868,
~9.4% reduction

Change-Id: I28d62e7b05b750b172be885ecd096b88e1db82a7
Reviewed-on: http://gerrit.cloudera.org:8080/1340
Reviewed-by: David Ribeiro Alves <33ea948168c114d220e0372a903be6ee60f6396e@cloudera.com>
Tested-by: Internal Jenkins

EvilMcJerkface/kudu,helifu/kudu,andrwng/kudu,InspurUSA/kudu,cloudera/kudu,andrwng/kudu,cloudera/kudu,EvilMcJerkface/kudu,helifu/kudu,cloudera/kudu,EvilMcJerkface/kudu,helifu/kudu,helifu/kudu,InspurUSA/kudu,InspurUSA/kudu,InspurUSA/kudu,andrwng/kudu,andrwng/kudu,helifu/kudu,andrwng/kudu,cloudera/kudu,EvilMcJerkface/kudu,cloudera/kudu,helifu/kudu,cloudera/kudu,helifu/kudu,InspurUSA/kudu,cloudera/kudu,EvilMcJerkface/kudu,EvilMcJerkface/kudu,andrwng/kudu,InspurUSA/kudu,andrwng/kudu,cloudera/kudu,InspurUSA/kudu,InspurUSA/kudu,helifu/kudu,EvilMcJerkface/kudu,helifu/kudu,EvilMcJerkface/kudu,cloudera/kudu,andrwng/kudu,InspurUSA/kudu,andrwng/kudu,EvilMcJerkface/kudu