reactor: make sure a poll cycle always happens when later is called
reactor: make sure a poll cycle always happens when later is called

Some of our code that is expected to run for a long time relies on the later()
facility to defer parts of its own execution. We would like to give other tasks
a chance to run, and preferrably force the pollers to execute. However, as much
as this is a likely outcome, this is not guaranteed by the current incarnation
of the code.

To understand why, we must keep two things in mind:
1) schedule() will merely add a task to the list
2) run_tasks will loop through all the tasks until the list is empty.

Because of (2), if we just push a task to the queue from our current caller, it
will be executed after all existing tasks but it may not call any poller. In
particular, in the extreme case that the queue has no other tasks but the
current deferring task, it will just resume shortly after that.

Because we do want the pollers to run, we call force_poll(). But again, since we
set the future as ready right after we do that, we will add our continuation to
the current end of the queue, but any continuation generated by the pollers will
be queued after it (unless they are high priority, but that's a different issue)

So the likely result of this call is that it will force the pollers to be run
shortly at some point in the future, but none of the poller work will run before
the task continues.

That doesn't seem to be the goal of this function. To fix that, we call
force_poll() before the first call to schedule() instead of inside it. The
result is that we will now push the schedule()d task and bail to the pollers
immediately. They will have the chance to queue their continuations. That
schedule()d task will run before the pollers and make the actual continuation
ready - but now all the pollers' continuations will already be queued.

With this patch applied, I see a small but consistent (~ 1 %) improvement in a
ScyllaDB write workload with all values at default.

Signed-off-by: Glauber Costa <7ba9037b0dd006f9408bd45b94707becd5cd4e87@scylladb.com>
Message-Id: <1469126299-28344-1-git-send-email-7ba9037b0dd006f9408bd45b94707becd5cd4e87@scylladb.com>

dreamsxin/seastar,cloudius-systems/seastar,avikivity/seastar,scylladb/seastar,cloudius-systems/seastar,syuu1228/seastar,kjniemi/seastar,kjniemi/seastar,cloudius-systems/seastar,avikivity/seastar,dreamsxin/seastar,avikivity/seastar,scylladb/seastar,dreamsxin/seastar,kjniemi/seastar,syuu1228/seastar,syuu1228/seastar,scylladb/seastar