Keep track of jobs assigned but not yet attended by a client in the scheduler. Fixes #513.
Keep track of jobs assigned but not yet attended by a client in the scheduler. Fixes #513.

We've found some situations where a server will retain jobs waiting for a
client to request they be run. In practice this means that a client that's
having trouble connecting to a server (due to #516 or another issue) can
take that server offline by consuming its capacity, which will be unavailable
indefinitely as far as the scheduler is concerned. This commit takes the step
of tracking jobs that have been assigned but have not yet been started.
When receiving a heartbeat from a server, the scheduler will check the unclaimed
jobs assigned to that server and de-allocate any that have been waiting to start
for too long. The threshold for now is 1 minute for jobs that start in the
"Ready" state and 5 minutes for jobs in the "Pending" state in case toolchain
packaging and transfer overrruns the smaller threshold.

mozilla/sccache,glandium/sccache,glandium/sccache,glandium/sccache,luser/sccache2,luser/sccache2,mozilla/sccache,mozilla/sccache,glandium/sccache,luser/sccache2,luser/sccache2