core: fix a race in execution stages
core: fix a race in execution stages

Execution stage has a threshold value - max_queue_length - after which
a flush will be forced. The flush procedure takes elements from the front
of the queue, processes them and returns. However, the exact order is:
1. Take a reference to the front element of the queue.
2. Launch the queued function.
3. Pop the element from the queue.

Unfortunately, if the applied function (2.) is inlined as ready and it also
triggers a flush (e.g. because the queue is full!), it will enter the
loop again and go to step 1., which will effectively take an element
from the top, which is a moved-from value, which leads to undefined behaviour.

Fix by moving the values and popping the queue front first and then proceeding,
which effectively changes the order above to 1->3->2, thus eliding the race.

Tested against an instantiation of this bug in ScyllaDB's issue
https://github.com/scylladb/scylla/issues/4856

Message-Id: <def79b55c35d08f06dc7596eddfdd4a5e0d534c5@scylladb.com>

avikivity/seastar,scylladb/seastar,syuu1228/seastar,avikivity/seastar,dreamsxin/seastar,scylladb/seastar,dreamsxin/seastar,scylladb/seastar,cloudius-systems/seastar,syuu1228/seastar,avikivity/seastar,dreamsxin/seastar,cloudius-systems/seastar,syuu1228/seastar,cloudius-systems/seastar