cros_async: Fix underflow in BlockingPool worker threads
cros_async: Fix underflow in BlockingPool worker threads

The `num_idle` field of the shared state between BlockingPool worker
threads can underflow in the following case:

* state.num_idle == 2.
* We spawn 2 new tasks into the BlockingPool.
* Both idle worker threads are woken up.  `state.num_idle` goes to 0.
* The first worker thread wakes up and pulls a task from the queue.
  That task finishes very quickly so the worker thread pulls the second
  task from the queue before the second worker thread is scheduled.
* The second worker thread is scheduled.  It sees that
  `s.tasks.is_empty() == true` so it goes back to waiting on the
  Condvar.
* The second worker thread's wait times out and it tries to decrement
  `state.num_idle` leading to underflow.

Fix this by adding a `num_notified` field to the shared worker state.
This field acts like a counter for the number of idle worker threads
that have been woken up.

When an idle thread is waiting on a Condvar, rather than checking if the
task queue is empty, it will instead check if num_notified > 0.  When an
idle worker thread observes that num_notified > 0 it decrements it by 1
and then goes back to processing tasks from the queue.  num_idle is only
decremented when num_notified is 0.

Change the num_idle decrement to a checked_sub so that we can catch it
even when -Coverflow_checks=off.  Also add a test for this case.  This
test consistently panics without the num_notified changes.

BUG=none
TEST=unit tests

Change-Id: Ia1b348605e0d02415635cdd023db1c10201ab661
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3139159
Tested-by: kokoro <2ac7b1f3fa578934c95181d4272be0d3bca00121@google.com>
Reviewed-by: Richard Zhang <666ac13a3069b6c13f19fc70113e1242ca7ab9af@google.com>
Reviewed-by: Noah Gold <2d4550d8af93bfc6e6fbe1fb1b0cc97a9ee7c3ce@google.com>
Reviewed-by: Daniel Verkamp <72bc170b46ec491f7bdd4359a1c0bfed274de40c@chromium.org>
Commit-Queue: Chirantan Ekbote <35787c0e27de5c48735e07f8324823b765e8bcbc@chromium.org>

google/crosvm,google/crosvm,google/crosvm,google/crosvm