epoll: explicitly delete sockets on close
epoll: explicitly delete sockets on close

The epoll-based event loop relies on the implicit removal of
closed FDs from the epoll set to do the housekeeping:
evloop_do_on_socket_close() is a no-op and the socket is
simply close(2)'d in do_dispose_socket(), while the event loop just
free()s the socket struct in the next iteration of update_status().

For the most part this works fine, however there is a caveat outlined in
epoll(7):

  Q6  Will closing a file descriptor cause it to be removed from all
      epoll sets automatically?

  A6  Yes,  but  be  aware of the following point.  A file descriptor is
      a reference to an open file description (see open(2)).  Whenever a
      file descriptor is duplicated via dup(2), dup2(2), fcntl(2)
      F_DUPFD, or fork(2),  a  new file  descriptor referring to the
      same open file description is created.  An open file description
      continues to exist until all file descriptors referring to it have
      been closed.  A file descriptor is removed from an  epoll set
      only after all the file descriptors referring to the underlying
      open file description have been closed (or before if the file
      descriptor is explicitly removed using epoll_ctl(2)
      EPOLL_CTL_DEL).  This  means  that  even after  a  file descriptor
      that  is part of an epoll set has been closed, events may be
      reported for that file descriptor if other file descriptors
      referring to the same underlying file description remain open.

While h2o never uses dup(2) and friends on sockets, it does
periodically fork(2) helper processes to handle OCSP updates. In that
tiny time window between fork(2) and having O_CLOEXEC kick-in,
all pollable sockets live in two processes and epoll_wait() on the
worker may actually wake up to process events from sockets it has
already closed. If such an event is processed after the relevant socket
has been free()d, a use-after-free abort or segfault might occur.

Since evloop_do_proceed() has no way of knowing if an event's data
pointer is supposed to be stale or not, we fix this by always removing
sockets from the epoll set explicitly before closing.
evloop_do_on_socket_export()'s error message is also modified to remove
ambiguity.

Signed-off-by: Apollon Oikonomopoulos <ed81ed0a45ccd5c8869bda52e38353a6299ee39e@debian.org>

lkwg82/h2o,yannick/h2o,h2o/h2o,h2o/h2o,devnexen/h2o,zlm2012/h2o,cwyang/h2o,deweerdt/h2o,zlm2012/h2o,zlm2012/h2o,zlm2012/h2o,deweerdt/h2o,deweerdt/h2o,deweerdt/h2o,yannick/h2o,cwyang/h2o,zlm2012/h2o,deweerdt/h2o,zlm2012/h2o,devnexen/h2o,deweerdt/h2o,lkwg82/h2o,lkwg82/h2o,h2o/h2o,cwyang/h2o,cwyang/h2o,yannick/h2o,devnexen/h2o,devnexen/h2o,lkwg82/h2o,yannick/h2o,lkwg82/h2o,deweerdt/h2o,deweerdt/h2o,devnexen/h2o,zlm2012/h2o,lkwg82/h2o,yannick/h2o,devnexen/h2o,yannick/h2o,yannick/h2o,lkwg82/h2o,zlm2012/h2o,cwyang/h2o,devnexen/h2o,h2o/h2o,devnexen/h2o,h2o/h2o,zlm2012/h2o,lkwg82/h2o,h2o/h2o,devnexen/h2o,cwyang/h2o,yannick/h2o,h2o/h2o,lkwg82/h2o,yannick/h2o,h2o/h2o,cwyang/h2o,h2o/h2o,cwyang/h2o,deweerdt/h2o