Remove the need for zmq in proper demand calcs.
Remove the need for zmq in proper demand calcs.

Where zmq is not used, and nodes are never marked as in use, nodepool
was unable to predict how large to grow the pool in advance. It
wouldn't launch new nodes until the current ones start to be deleted.

Previously the demands were calculated as:
start_demand = queue - running_jobs
ready = n_ready + n_building + n_test
actual_demand = start_demand - ready
extra_demand = start_demand + min_ready - ready

Expanded:
actual_demand =
    queue - running_jobs - n_ready - n_building - n_test
extra_demand =
    queue - running_jobs + min_ready - n_ready - n_building - n_test

If you make an assumption that a nodepool node will be used to do
a job for the gearman it is connected to, it becomes possible to
calculate the actual demand off just the gearman statistics instead.
Conversely this also assumes that a gearman job marked as 'running'
is being run by a nodepool managed node.

As such, count the total jobs in the queue as the demand (inclusive
of running jobs), then subtract the nodes considered to be USED. These
therefore account for the running jobs and lower the demand since
they aren't actually waiting for a node. This give us:

start_demand = queue
ready = n_ready + n_building + n_used + n_test
actual_demand = start_demand - ready
extra_demand = start_demand + min_ready - ready

Expanded:
actual_demand =
    queue - n_ready - n_building - n_used - n_test
extra_demand =
    queue + min_ready - n_ready - n_building - n_used - n_test

So assuming n_used == running_jobs (which it should in theory if
there are no foreign workers) then you can see the above two expanded
calculations are equal.

Nodes will not be able to enter the USED state where zmq isn't
enabled (since it is the zmq event that updates the db). However,
any node that would have been marked as USED will be in READY state
in this case. As such, since the demand consists of the sum of
READY+BUILDING+USED+TEST nodepool doesn't actually care which state
a particular node is in for its calculations.

In this case the demand could be thought of as "how many jobs am I
doing and need to do" (A) and "how many nodes am I managing" (B).
If we count jobs that are already in progress in (A) and all nodes,
including those actually running jobs, in (B) then the demand should
be covered if (B) > (A).

Removing the requirement of zmq makes it easier for other workers,
such as turbo-hipster, to be launched by nodepool without the need
of sending back an event stream.

Change-Id: I23089513ae4ff58388cfa82030b21f2ed41e51b9

Tesora/tesora-nodepool,seandst/nodepool,seandst/nodepool,Tesora/tesora-nodepool,citrix-openstack/nodepool,citrix-openstack/nodepool