Annotate `check_idle_saturated` for Cythonization (#4289)
Annotate `check_idle_saturated` for Cythonization (#4289)

In our profiling of the scheduler, we identified to main transitions that took a good chunk of time. These are `transition_processing_memory` and `transition_waiting_processing`. The former takes slightly longer than the other, but both are easily 2x slower than any transition that follows them. Both of them directly or indirectly make a call to `check_idle_saturated`. While this is not necessarily the worst bottleneck for either of them, it does stick out on the callgraph and stands a good chance of improving both transitions runtimes at once. Additionally `check_idle_saturated` includes a fair bit of code that simply crunches numbers and does not touch Python objects as much. It also isn't as dependent on the Cythonization of other classes as other functions in the profile are. So this makes it easier to Cythonize this piece of code without needing to touch too much other code.

Here we go through and annotate the local variables with types. Also we assign non-local variables accessed through attributes to local variables, which we type. Additionally we changed default argument values to be more friendly with C-style types. Initially we tried to type `self.idle`. However as [`self.idle` is a `sortedcontainers.SortedSet`]( https://github.com/dask/distributed/blob/9460e3fe1e0bcdb2daf6ebafe5335d536fa4f492/distributed/scheduler.py#L1292 ), this didn't work (as we need an actual Python `set` to type it). So we left this as untyped.

Should add as a good chunk of the time in `check_idle_saturated` is just spent doing `ws.status == Status.closed`, we still need PR ( https://github.com/dask/distributed/pull/4270 ) to cutdown on the time spent in this method.



Combine assignments and separate them from `if` blocks.

* Change `occ` default to not be `None`

To make it easier to type `occ` later, define this as a non-`None`
value, which is also clearly bogus (namely `-1.0`). That way we can
still replace this value when it hasn't been otherwise set while also
including a type that includes the default value.

* Annotate `check_idle_saturated` for Cythonization

* Hackily type `set`s through local assignment

For now to just see what is possible, assign these attributes to local
variables typed as `set`s. This will allow Cython to use the
corresponding Python C APIs with these objects; thus, optimizing how
they are handled. This saves us temporarily from trying to more
generally Cythonize this class while exploring optimizations here.

* Drop typing of `self.idle`

As `self.idle` is actually a `SortedSet` instead of a `set`, we can't
type it. So revert the typing of `idle`. Though leave all other
`set`-based typing for other optimizations to be applied where possible.
Also leave the assignment of `self.idle` to `idle` as this generates the
attribute access code only once and we need this for both branches
anyways.
dask/distributed,dask/distributed,dask/distributed,dask/distributed