smp: register smp poller earlier so we can dispatch remote I/O earlier
smp: register smp poller earlier so we can dispatch remote I/O earlier

Pollers are run in the order they are registered - as they are executed
by looping through an array of registered pollers.

The SMP pollers run quite late. In particular, they run after the I/O
pollers are executed. This means that if in the beginning of a task
quota a remote shard sent a request to the I/O shard, the I/O shard
won't see it because the SMP queues weren't polled.

This causes the latencies of initiating remote requests to be 2 *
task_quota in the worst case, as opposed to 1 * task_quota for a local
request.

To remedy that, we can make sure we run the SMP pollers before we run
the I/O functions. The remote shard don't have to go through a poll
event to inject, as the events are put directly into its SMP Queues.
If we do that, the latency to initiate a remote request should be the
same as the latency to initiate a local request.

It is tempting to think that we could run io_pollfn, which checks for
ready requests, before we run the SMP pollers so that requests that are
ready can already be sent to their requesting CPUs. However, because
there is still a continuation in that path this doesn't really buy us
much.

By using io_tester with the shard that has the I/O Queue processing 5
CPU hog classes and no I/O, and a remote shard dispatching latency-bound
I/O into it with a 250usec task quota, we can see that the average
latencies are significantly improved (23 % better). A similar
improvement is seen all the way up to the 99 %.

Reads Before:
Class  0 (latency_reads: 1000 shares, 4096-byte RAND READ, 4 concurrent requests, NO think time)
  Throughput         :  32942.5 KB/s
  Lat average        :      481 usec
  Lat quantile=  0.5 :      425 usec
  Lat quantile= 0.95 :      621 usec
  Lat quantile= 0.99 :      688 usec
  Lat quantile=0.999 :     2309 usec
  Lat max            :   405644 usec

Reads After:
Class  0 (latency_reads: 1000 shares, 4096-byte RAND READ, 4 concurrent requests, NO think time)
  Throughput         :  42726.1 KB/s
  Lat average        :      370 usec
  Lat quantile=  0.5 :      318 usec
  Lat quantile= 0.95 :      470 usec
  Lat quantile= 0.99 :      605 usec
  Lat quantile=0.999 :      627 usec
  Lat max            :   405621 usec

Signed-off-by: Glauber Costa <7ba9037b0dd006f9408bd45b94707becd5cd4e87@scylladb.com>

syuu1228/seastar,syuu1228/seastar,avikivity/seastar,scylladb/seastar,cloudius-systems/seastar,scylladb/seastar,cloudius-systems/seastar,syuu1228/seastar,cloudius-systems/seastar,avikivity/seastar,dreamsxin/seastar,dreamsxin/seastar,avikivity/seastar,scylladb/seastar,dreamsxin/seastar