Don't let flush preempt compaction in certain cases
Don't let flush preempt compaction in certain cases

Summary:
I have an application configured with 16 background threads. Write rates are high. L0->L1 compactions is very slow and it limits the concurrency of the system. While it's happening, other 15 threads are idle. However, when there is a need of a flush, that one thread busy with L0->L1 is doing flush, instead of any other 15 threads that are just sitting there.

This diff prevents that. If there are threads that are idle, we don't let flush preempt compaction.

Test Plan: Will run stress test

Reviewers: ljin, sdong, yhchiang

Reviewed By: sdong, yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D22299

wat-ze-hex/rocksdb,kaschaeffer/rocksdb,zhangpng/rocksdb,wenduo/rocksdb,fengshao0907/rocksdb,wskplho/rocksdb,NickCis/rocksdb,anagav/rocksdb,vashstorm/rocksdb,wat-ze-hex/rocksdb,virtdb/rocksdb,fengshao0907/rocksdb,fengshao0907/rocksdb,Andymic/rocksdb,kaschaeffer/rocksdb,jalexanderqed/rocksdb,NickCis/rocksdb,Andymic/rocksdb,JackLian/rocksdb,ylong/rocksdb,OverlordQ/rocksdb,Vaisman/rocksdb,skunkwerks/rocksdb,skunkwerks/rocksdb,JoeWoo/rocksdb,makelivedotnet/rocksdb,alihalabyah/rocksdb,jalexanderqed/rocksdb,tsheasha/rocksdb,Vaisman/rocksdb,wenduo/rocksdb,JoeWoo/rocksdb,dkorolev/rocksdb,flabby/rocksdb,JohnPJenkins/rocksdb,sorphi/rocksdb,caijieming-baidu/rocksdb,hobinyoon/rocksdb,lgscofield/rocksdb,vashstorm/rocksdb,vmx/rocksdb,JohnPJenkins/rocksdb,JackLian/rocksdb,Vaisman/rocksdb,siddhartharay007/rocksdb,biddyweb/rocksdb,rDSN-Projects/rocksdb.replicated,skunkwerks/rocksdb,NickCis/rocksdb,caijieming-baidu/rocksdb,Andymic/rocksdb,wlqGit/rocksdb,wlqGit/rocksdb,wlqGit/rocksdb,JoeWoo/rocksdb,wskplho/rocksdb,rDSN-Projects/rocksdb.replicated,siddhartharay007/rocksdb,biddyweb/rocksdb,luckywhu/rocksdb,JackLian/rocksdb,biddyweb/rocksdb,anagav/rocksdb,SunguckLee/RocksDB,vmx/rocksdb,JohnPJenkins/rocksdb,Applied-Duality/rocksdb,mbarbon/rocksdb,facebook/rocksdb,biddyweb/rocksdb,flabby/rocksdb,JoeWoo/rocksdb,siddhartharay007/rocksdb,wskplho/rocksdb,OverlordQ/rocksdb,bbiao/rocksdb,vmx/rocksdb,mbarbon/rocksdb,tsheasha/rocksdb,skunkwerks/rocksdb,rDSN-Projects/rocksdb.replicated,tschottdorf/rocksdb,amyvmiwei/rocksdb,bbiao/rocksdb,temicai/rocksdb,ylong/rocksdb,siddhartharay007/rocksdb,tschottdorf/rocksdb,norton/rocksdb,tizzybec/rocksdb,luckywhu/rocksdb,amyvmiwei/rocksdb,wat-ze-hex/rocksdb,makelivedotnet/rocksdb,IMCG/RcoksDB,norton/rocksdb,rDSN-Projects/rocksdb.replicated,wskplho/rocksdb,mbarbon/rocksdb,rDSN-Projects/rocksdb.replicated,vashstorm/rocksdb,wenduo/rocksdb,jalexanderqed/rocksdb,dkorolev/rocksdb,RyanTech/rocksdb,tizzybec/rocksdb,NickCis/rocksdb,temicai/rocksdb,wskplho/rocksdb,tschottdorf/rocksdb,SunguckLee/RocksDB,wenduo/rocksdb,wlqGit/rocksdb,RyanTech/rocksdb,norton/rocksdb,mbarbon/rocksdb,lgscofield/rocksdb,tsheasha/rocksdb,vmx/rocksdb,mbarbon/rocksdb,skunkwerks/rocksdb,hobinyoon/rocksdb,Applied-Duality/rocksdb,bbiao/rocksdb,biddyweb/rocksdb,hobinyoon/rocksdb,Applied-Duality/rocksdb,norton/rocksdb,geraldoandradee/rocksdb,caijieming-baidu/rocksdb,sorphi/rocksdb,jalexanderqed/rocksdb,wat-ze-hex/rocksdb,sorphi/rocksdb,SunguckLee/RocksDB,virtdb/rocksdb,ryneli/rocksdb,mbarbon/rocksdb,JackLian/rocksdb,ylong/rocksdb,ryneli/rocksdb,Applied-Duality/rocksdb,wlqGit/rocksdb,facebook/rocksdb,geraldoandradee/rocksdb,IMCG/RcoksDB,wenduo/rocksdb,lgscofield/rocksdb,hobinyoon/rocksdb,dkorolev/rocksdb,wat-ze-hex/rocksdb,geraldoandradee/rocksdb,alihalabyah/rocksdb,alihalabyah/rocksdb,luckywhu/rocksdb,vashstorm/rocksdb,tsheasha/rocksdb,amyvmiwei/rocksdb,hobinyoon/rocksdb,tizzybec/rocksdb,RyanTech/rocksdb,ylong/rocksdb,zhangpng/rocksdb,tsheasha/rocksdb,caijieming-baidu/rocksdb,fengshao0907/rocksdb,Andymic/rocksdb,JackLian/rocksdb,makelivedotnet/rocksdb,Applied-Duality/rocksdb,facebook/rocksdb,ylong/rocksdb,kaschaeffer/rocksdb,caijieming-baidu/rocksdb,luckywhu/rocksdb,anagav/rocksdb,lgscofield/rocksdb,temicai/rocksdb,SunguckLee/RocksDB,ylong/rocksdb,ryneli/rocksdb,facebook/rocksdb,flabby/rocksdb,luckywhu/rocksdb,Andymic/rocksdb,Andymic/rocksdb,tschottdorf/rocksdb,tizzybec/rocksdb,facebook/rocksdb,RyanTech/rocksdb,sorphi/rocksdb,JackLian/rocksdb,JoeWoo/rocksdb,OverlordQ/rocksdb,bbiao/rocksdb,lgscofield/rocksdb,vmx/rocksdb,Applied-Duality/rocksdb,fengshao0907/rocksdb,mbarbon/rocksdb,Applied-Duality/rocksdb,makelivedotnet/rocksdb,flabby/rocksdb,amyvmiwei/rocksdb,hobinyoon/rocksdb,facebook/rocksdb,wat-ze-hex/rocksdb,alihalabyah/rocksdb,dkorolev/rocksdb,hobinyoon/rocksdb,NickCis/rocksdb,geraldoandradee/rocksdb,vashstorm/rocksdb,geraldoandradee/rocksdb,vmx/rocksdb,rDSN-Projects/rocksdb.replicated,tizzybec/rocksdb,amyvmiwei/rocksdb,luckywhu/rocksdb,rDSN-Projects/rocksdb.replicated,zhangpng/rocksdb,wlqGit/rocksdb,sorphi/rocksdb,vmx/rocksdb,biddyweb/rocksdb,flabby/rocksdb,bbiao/rocksdb,IMCG/RcoksDB,SunguckLee/RocksDB,JoeWoo/rocksdb,JohnPJenkins/rocksdb,Andymic/rocksdb,virtdb/rocksdb,OverlordQ/rocksdb,zhangpng/rocksdb,wenduo/rocksdb,wlqGit/rocksdb,ryneli/rocksdb,bbiao/rocksdb,siddhartharay007/rocksdb,tizzybec/rocksdb,wenduo/rocksdb,SunguckLee/RocksDB,IMCG/RcoksDB,virtdb/rocksdb,norton/rocksdb,norton/rocksdb,ryneli/rocksdb,luckywhu/rocksdb,kaschaeffer/rocksdb,temicai/rocksdb,zhangpng/rocksdb,jalexanderqed/rocksdb,alihalabyah/rocksdb,temicai/rocksdb,tschottdorf/rocksdb,virtdb/rocksdb,ryneli/rocksdb,fengshao0907/rocksdb,amyvmiwei/rocksdb,tschottdorf/rocksdb,IMCG/RcoksDB,RyanTech/rocksdb,siddhartharay007/rocksdb,wskplho/rocksdb,skunkwerks/rocksdb,caijieming-baidu/rocksdb,anagav/rocksdb,sorphi/rocksdb,hobinyoon/rocksdb,ryneli/rocksdb,IMCG/RcoksDB,lgscofield/rocksdb,Vaisman/rocksdb,jalexanderqed/rocksdb,fengshao0907/rocksdb,bbiao/rocksdb,sorphi/rocksdb,norton/rocksdb,JackLian/rocksdb,Vaisman/rocksdb,norton/rocksdb,kaschaeffer/rocksdb,ylong/rocksdb,OverlordQ/rocksdb,virtdb/rocksdb,dkorolev/rocksdb,skunkwerks/rocksdb,wskplho/rocksdb,geraldoandradee/rocksdb,JohnPJenkins/rocksdb,amyvmiwei/rocksdb,alihalabyah/rocksdb,facebook/rocksdb,anagav/rocksdb,flabby/rocksdb,dkorolev/rocksdb,vashstorm/rocksdb,JoeWoo/rocksdb,zhangpng/rocksdb,facebook/rocksdb,bbiao/rocksdb,OverlordQ/rocksdb,RyanTech/rocksdb,tsheasha/rocksdb,virtdb/rocksdb,Vaisman/rocksdb,alihalabyah/rocksdb,wat-ze-hex/rocksdb,biddyweb/rocksdb,makelivedotnet/rocksdb,NickCis/rocksdb,jalexanderqed/rocksdb,NickCis/rocksdb,wat-ze-hex/rocksdb,wenduo/rocksdb,tizzybec/rocksdb,vmx/rocksdb,lgscofield/rocksdb,anagav/rocksdb,siddhartharay007/rocksdb,kaschaeffer/rocksdb,caijieming-baidu/rocksdb,JohnPJenkins/rocksdb,SunguckLee/RocksDB,dkorolev/rocksdb,temicai/rocksdb,kaschaeffer/rocksdb,flabby/rocksdb,JohnPJenkins/rocksdb,RyanTech/rocksdb,geraldoandradee/rocksdb,zhangpng/rocksdb,makelivedotnet/rocksdb,anagav/rocksdb,Andymic/rocksdb,tsheasha/rocksdb,tsheasha/rocksdb,vashstorm/rocksdb,temicai/rocksdb,makelivedotnet/rocksdb,IMCG/RcoksDB,SunguckLee/RocksDB,OverlordQ/rocksdb,Vaisman/rocksdb