'RuntimeError: IOLoop is closing' thrown in Minion when using TCP transport
'RuntimeError: IOLoop is closing' thrown in Minion when using TCP transport

This issue was caused by the fix to Issue #26889. That fix caused the
io_loop to close in: 'salt.utils.async.SyncWrapper.__del__' before
closing the stream in 'salt.transport.SaltMessageClient.destroy'. The
side effect of this was that in tornado 4.2.1, tornado/ioloop.py lines
911-914:
    def add_callback(self, callback, *args, **kwargs):
        with self._callback_lock:
            if self._closing:
                raise RuntimeError("IOLoop is closing")

This RuntimeError exception would be raised. The reason why it is
raised is that the read future is still pending, and so it will invoke
set_exception(StreamClosedError()) on that future when closing the
stream. That set_exception will try to invoke a callback and run into
the code above which would throw the RuntimeError because the io_loop
is closed. The work-around is to mark the stream's '_read_future' as
None to prevent it from calling set_exception on it.

A better long term solution would be to rearchitect SaltMessageClient
to no longer use 'read_until' nor 'read_bytes' and instead use
'read_until_close' since 'read_until_close' is designed to have a read
active at the time of close whereas the others are not. This would
allow the removal of all the work-arounds in
'SaltMessageClient.destroy'.

Signed-off-by: Sergey Kizunov <dfddabaa9af05982b01940b837a87844148069c9@ni.com>

saltstack/salt,saltstack/salt,saltstack/salt,saltstack/salt,saltstack/salt