Track the pollsets of an FD in PO_MULTI mode for pollex.
Track the pollsets of an FD in PO_MULTI mode for pollex.

Each pollset in pollex has a lock, grabbed upon adding an FD
to the pollset. Since this is called on a per-call basis,
there is a flat array caching the FDs of the pollset, to
avoid unnecessarily calling epoll_ctl multiple times for
the same FD.

This has two problems:
1) When multiple threads add FDs to the same pollset,
   we will have contention on the pollset lock.
2) When we have many FDs we simply run out of cache storage,
   and call epoll_ctl().

This commit changes the caching strategy by simply storing
the pollsets of an FD inside that FD, when we are in PO_MULTI
mode. This results in address in both (1) and (2).

Moreover, this commit fixes another performance bug.
When we have a release FD callback, we do not call close().
That FD will remain in our epollset, until the new owner of
the FD actually call close(). This results in a lot of spurious
wake ups when we simply hand off gRPC FDs to other FDs.

vjpai/grpc,jtattermusch/grpc,muxi/grpc,carl-mastrangelo/grpc,grpc/grpc,jboeuf/grpc,grpc/grpc,ejona86/grpc,vjpai/grpc,carl-mastrangelo/grpc,pszemus/grpc,donnadionne/grpc,stanley-cheung/grpc,ctiller/grpc,donnadionne/grpc,muxi/grpc,ctiller/grpc,carl-mastrangelo/grpc,ejona86/grpc,firebase/grpc,nicolasnoble/grpc,donnadionne/grpc,jboeuf/grpc,sreecha/grpc,donnadionne/grpc,ctiller/grpc,donnadionne/grpc,muxi/grpc,firebase/grpc,donnadionne/grpc,grpc/grpc,ctiller/grpc,ctiller/grpc,carl-mastrangelo/grpc,jtattermusch/grpc,grpc/grpc,jtattermusch/grpc,jtattermusch/grpc,vjpai/grpc,carl-mastrangelo/grpc,nicolasnoble/grpc,jtattermusch/grpc,muxi/grpc,firebase/grpc,nicolasnoble/grpc,muxi/grpc,pszemus/grpc,pszemus/grpc,stanley-cheung/grpc,carl-mastrangelo/grpc,muxi/grpc,stanley-cheung/grpc,donnadionne/grpc,ejona86/grpc,grpc/grpc,nicolasnoble/grpc,vjpai/grpc,donnadionne/grpc,jboeuf/grpc,nicolasnoble/grpc,stanley-cheung/grpc,stanley-cheung/grpc,grpc/grpc,sreecha/grpc,ejona86/grpc,donnadionne/grpc,stanley-cheung/grpc,ctiller/grpc,grpc/grpc,nicolasnoble/grpc,donnadionne/grpc,ctiller/grpc,vjpai/grpc,nicolasnoble/grpc,stanley-cheung/grpc,sreecha/grpc,muxi/grpc,muxi/grpc,ctiller/grpc,firebase/grpc,nicolasnoble/grpc,vjpai/grpc,jtattermusch/grpc,sreecha/grpc,firebase/grpc,nicolasnoble/grpc,grpc/grpc,vjpai/grpc,grpc/grpc,nicolasnoble/grpc,grpc/grpc,pszemus/grpc,sreecha/grpc,pszemus/grpc,jtattermusch/grpc,vjpai/grpc,pszemus/grpc,nicolasnoble/grpc,ejona86/grpc,carl-mastrangelo/grpc,jtattermusch/grpc,sreecha/grpc,jboeuf/grpc,firebase/grpc,grpc/grpc,muxi/grpc,pszemus/grpc,ejona86/grpc,donnadionne/grpc,ejona86/grpc,ctiller/grpc,ejona86/grpc,carl-mastrangelo/grpc,jboeuf/grpc,jtattermusch/grpc,stanley-cheung/grpc,jboeuf/grpc,donnadionne/grpc,jtattermusch/grpc,jtattermusch/grpc,firebase/grpc,nicolasnoble/grpc,jboeuf/grpc,vjpai/grpc,jboeuf/grpc,ctiller/grpc,grpc/grpc,sreecha/grpc,ctiller/grpc,sreecha/grpc,carl-mastrangelo/grpc,muxi/grpc,jtattermusch/grpc,vjpai/grpc,pszemus/grpc,muxi/grpc,sreecha/grpc,jboeuf/grpc,firebase/grpc,jboeuf/grpc,ejona86/grpc,stanley-cheung/grpc,pszemus/grpc,carl-mastrangelo/grpc,sreecha/grpc,pszemus/grpc,jboeuf/grpc,firebase/grpc,carl-mastrangelo/grpc,jboeuf/grpc,stanley-cheung/grpc,vjpai/grpc,sreecha/grpc,carl-mastrangelo/grpc,pszemus/grpc,firebase/grpc,pszemus/grpc,stanley-cheung/grpc,ejona86/grpc,vjpai/grpc,muxi/grpc,stanley-cheung/grpc,firebase/grpc,sreecha/grpc,ejona86/grpc,ctiller/grpc,ejona86/grpc,firebase/grpc