Decouple removing the fileWatcher from reading
Decouple removing the fileWatcher from reading

Fixes #27779

Currently `followLogs` can get into a deadlock if we receive an inotify
IN_MODIFY event while we are trying to close the `fileWatcher`. This is
due to the fact that closing the `fileWatcher` happens in the same block
as consumes events from the `fileWatcher`. We are trying to run
`fileWatcher.Close`, which is waiting for an IN_IGNORE event to come in
over inotify to confirm the watch was been removed. But, because an
IN_MODIFY event has appeared after `Close` was entered but before the
IN_IGNORE, the broadcast never comes. The IN_MODIFY cannot be consumed
as the events channel is unbuffered and the only `select` that reads
from it is busy waiting for the IN_IGNORE event.

In order to try and fix this race condition I've moved the removal of
the `fileWatcher` out to a separate go block that waits for a signal to
close, removes the watcher and then signals to the previous selects on
the close signal.

This has introduced a `fileWatcher.Remove` in the final case, but if we
try and remove a watcher that does not exist it will just return an
error saying so. We are not doing any checking on the return of `Remove`
so this shouldn't cause any side-effects.

Signed-off-by: Tom Booth <6e36c44ffc54c164e3d870bde661a38acace11ad@gmail.com>

diogomonica/docker,justincormack/docker,ripcurld0/moby,mikedougherty/docker,prertik/docker,unclejack/docker,darrenstahlmsft/docker,SvenDowideit/docker,miminar/docker,nawawi/docker,jthelin/docker,msabansal/docker,mahak/docker,nalind/docker,thaJeztah/docker,Syntaxide/docker,stevvooe/docker,catinthesky/docker,thaJeztah/docker,msabansal/docker,miminar/docker,haveatry/docker,flavio/docker,coreos/docker,qudongfang/docker,YuPengZTE/docker,johnstep/moby-moby,dperny/docker,johnstep/moby-moby,POLJM242/docker,KarthikNayak/docker,vdemeester/docker,madhanrm/docker,moby/moby,shishir-a412ed/docker,konstruktoid/docker-upstream,HideoYamauchi/docker,raja-sami-10p/moby,kolyshkin/moby,vdemeester/docker,nawawi/docker,jvanz/docker,bearstech/moby,noironetworks/docker,trebonian/docker,dmcgowan/docker,tonistiigi/docker,jimjag/docker,SvenDowideit/docker,resin-io/docker,laijs/docker,alfred-landrum/docker,johnstep/moby-moby,x1022as/docker,tklauser/docker,paralin/docker,nawawi/docker,cyphar/docker,KarthikNayak/docker,madhanrm/docker,moul/docker,mtrmac/docker,gnailuy/docker,michael-holzheu/docker,paralin/docker,unclejack/docker,mixja/docker,unclejack/docker,jlhawn/docker,prertik/docker,bonczj/moby-json-file-logger,sdurrheimer/docker,flavio/docker,kolyshkin/docker,akira-baruah/docker,cpuguy83/docker,AkihiroSuda/docker,darrenstahlmsft/docker,fcrisciani/docker,ibuildthecloud/docker,dccurtis/docker,mkumatag/docker,jimjag/docker,harche/docker,gschaetz/moby,tmp6154/docker,Syntaxide/docker,lixiaobing10051267/docker,alfred-landrum/docker,alfred-landrum/docker,stevvooe/docker,justincormack/docker,erikstmartin/docker,xuxinkun/docker,yasiramin10p/moby,yongtang/docker,noironetworks/docker,naveed-jamil-tenpearls/moby,shishir-a412ed/docker,gjaskiewicz/moby,duglin/docker,sarnautov/moby,tophj-ibm/moby,jameszhan/docker,mkumatag/docker,dgageot/docker,jlhawn/docker,whuwxl/docker,trebonian/docker,mstanleyjones/docker,dims/docker,pramodhkp/moby,tophj-ibm/docker,cpuguy83/docker,hqhq/moby,dgageot/docker,moby/moby,dsheets/moby,moul/docker,qudongfang/docker,konstruktoid/docker-1,shhsu/docker,kolyshkin/moby,PatrickLang/moby,diogomonica/docker,PatrickLang/moby,JMesser81/docker,amatsus/docker,allencloud/docker,flavio/docker,allencloud/docker,darrenstahlmsft/docker,dsheets/moby,rancher/docker,moypray/docker,thaJeztah/docker,rhatdan/docker,pradipd/moby,mikedougherty/docker,erikh/docker,tonistiigi/docker,coolljt0725/docker,naveed-jamil-tenpearls/moby,jthelin/docker,ndeloof/docker,ColinHebert/docker,mtesselH/docker,sterchelen/docker,stevvooe/docker,yosifkit/docker,erikh/docker,liaoqingwei/docker,kencochrane/docker,tianon/docker,nalind/docker,haveatry/docker,tonistiigi/docker,mkumatag/docker,hqhq/moby,dims/docker,srust/docker,laijs/moby,dgageot/docker,raja-sami-10p/moby,shhsu/docker,resin-io/docker,laijs/docker,coreos/docker,SvenDowideit/docker,darrenstahlmsft/moby,HideoYamauchi/docker,whuwxl/docker,laijs/moby,xuxinkun/docker,crosbymichael/docker,ffoysal/docker,sdurrheimer/docker,gnailuy/docker,cpuguy83/docker,amatsus/docker,imkin/moby,mike1729/docker,estesp/docker,moul/docker,kolyshkin/docker,tophj-ibm/docker,djs55/docker,lixiaobing10051267/docker,moul/docker,maxamillion/docker,lixiaobing10051267/docker,coolljt0725/docker,thtanaka/docker,scotttlin/docker,AkihiroSuda/docker,mtesselH/docker,catinthesky/docker,tmp6154/docker,yasiramin10p/moby,yummypeng/docker,bonczj/moby-json-file-logger,fcrisciani/docker,estesp/docker,rremer/moby,tophj-ibm/moby,prertik/docker,konstruktoid/docker-1,sterchelen/docker,yongtang/docker,ndeloof/docker,paralin/docker,srust/docker,charleszheng44/moby,mixja/docker,erikstmartin/docker,konstruktoid/docker-upstream,msabansal/docker,ColinHebert/docker,ffoysal/docker,justincormack/docker,rremer/moby,imkin/moby,dperny/docker,lixiaobing1/docker,flavio/docker,scotttlin/docker,srust/docker,jwhonce/docker,ibuildthecloud/docker,jvanz/docker,jameszhan/docker,Syntaxide/docker,AkihiroSuda/docker,docker/docker,PatrickLang/moby,bearstech/moby,POLJM242/docker,djs55/docker,sterchelen/docker,JMesser81/docker,madhanrm/docker,imkin/moby,scotttlin/docker,mstanleyjones/docker,x1022as/docker,michael-holzheu/docker,diogomonica/docker,coreos/docker,dmcgowan/docker,dccurtis/docker,darrenstahlmsft/moby,Mokolea/docker,shhsu/docker,danyalKhaliq10p/moby,estesp/docker,whuwxl/docker,yasiramin10p/moby,mikebrow/docker,moby/moby,maxamillion/docker,dmcgowan/docker,gnailuy/docker,harche/docker,thtanaka/docker,jlhawn/docker,rhatdan/docker,gschaetz/moby,kencochrane/docker,KingEmet/docker,crosbymichael/docker,erikh/docker,mikedougherty/docker,mahak/docker,moypray/docker,yummypeng/docker,q384566678/docker,KingEmet/docker,resin-io/docker,mikebrow/docker,Ramzec/docker,jimjag/docker,rancher/docker,konstruktoid/docker-1,mahak/docker,dims/docker,shishir-a412ed/docker,danyalKhaliq10p/moby,jimjag/docker,erikstmartin/docker,tophj-ibm/moby,ffoysal/docker,mike1729/docker,gnailuy/docker,michael-holzheu/docker,coolljt0725/docker,darrenstahlmsft/moby,rremer/moby,x1022as/docker,ibuildthecloud/docker,tophj-ibm/docker,KingEmet/docker,YuPengZTE/docker,AkihiroSuda/docker,KarthikNayak/docker,charleszheng44/moby,rancher/docker,thtanaka/docker,mstanleyjones/docker,ibuildthecloud/docker,Ramzec/docker,amatsus/docker,dccurtis/docker,Mokolea/docker,dperny/docker,tklauser/docker,cyphar/docker,mixja/docker,crosbymichael/docker,jvanz/docker,sarnautov/moby,akira-baruah/docker,catinthesky/docker,paralin/docker,duglin/docker,gschaetz/moby,JMesser81/docker,hqhq/moby,jwhonce/docker,gjaskiewicz/moby,lixiaobing1/docker,trebonian/docker,pramodhkp/moby,qudongfang/docker,rhatdan/docker,mike1729/docker,tmp6154/docker,miminar/docker,yosifkit/docker,q384566678/docker,mahak/docker,mikebrow/docker,YuPengZTE/docker,pradipd/moby,yosifkit/docker,yongtang/docker,yummypeng/docker,konstruktoid/docker-upstream,pramodhkp/moby,haveatry/docker,kojiromike/docker,ibuildthecloud/docker,nawawi/docker,trebonian/docker,jameszhan/docker,moypray/docker,cpuguy83/docker,bonczj/moby-json-file-logger,ripcurld0/moby,konstruktoid/docker-upstream,naveed-jamil-tenpearls/moby,charleszheng44/moby,Mokolea/docker,mtesselH/docker,mtrmac/docker,lixiaobing1/docker,laijs/docker,kolyshkin/moby,HideoYamauchi/docker,docker/docker,duglin/docker,Ramzec/docker,dsheets/moby,tianon/docker,danyalKhaliq10p/moby,liaoqingwei/docker,harche/docker,gjaskiewicz/moby,sarnautov/moby,liaoqingwei/docker,ColinHebert/docker,cyphar/docker,noironetworks/docker,maxamillion/docker,raja-sami-10p/moby,kencochrane/docker,laijs/moby,fcrisciani/docker,vdemeester/docker,nalind/docker,ripcurld0/moby,mtrmac/docker,Ramzec/docker,moby/moby,yosifkit/docker,jwhonce/docker,xuxinkun/docker,djs55/docker,nalind/docker,pradipd/moby,bearstech/moby,tklauser/docker,thaJeztah/docker,kojiromike/docker,ndeloof/docker,q384566678/docker,kojiromike/docker,Mokolea/docker,bearstech/moby,docker/docker,allencloud/docker,sdurrheimer/docker