prometheus: collector: track busy sockets
prometheus: collector: track busy sockets

The current prometheus collector uses an unbounded amount of goroutines to
collect metrics from each VM.
Since the amount of VM is stable and slow-changing, this is not an
issue, unless a VM start to go rogue and becomes slow to answer to
calls, or unresponsive completely.

In this case blocked goroutines can add up trying to poll a stuck
domain. Existing code already takes care of not returning stale data,
but still goroutines are still thrown to a useless task.
Goroutines are cheap, and the go runtime can handle them in the order
of ten of thousands, but still the current approach is not good enough.

To avoid this, we add a map to track currently busy sockets. The flow
becomes:
- when iterating over the available sockets, query only the free ones
- mark busy a socket right before to poll it
- mark free a socket only after a succesfull query

Signed-off-by: Francesco Romani <c7100b580e6017ef67482ace46578cdd2d2e0dfe@redhat.com>

davidvossel/kubevirt,davidvossel/kubevirt,kubevirt/kubevirt,kubevirt/kubevirt,stu-gott/kubevirt,davidvossel/kubevirt,stu-gott/kubevirt,kubevirt/kubevirt,stu-gott/kubevirt,davidvossel/kubevirt,kubevirt/kubevirt,stu-gott/kubevirt