[SPARK-3398] [EC2] Have spark-ec2 intelligently wait for specific cluster states
[SPARK-3398] [EC2] Have spark-ec2 intelligently wait for specific cluster states

Instead of waiting arbitrary amounts of time for the cluster to reach a specific state, this patch lets `spark-ec2` explicitly wait for a cluster to reach a desired state.

This is useful in a couple of situations:
* The cluster is launching and you want to wait until SSH is available before installing stuff.
* The cluster is being terminated and you want to wait until all the instances are terminated before trying to delete security groups.

This patch removes the need for the `--wait` option and removes some of the time-based retry logic that was being used.

Author: Nicholas Chammas <nicholas.chammas@gmail.com>

Closes #2339 from nchammas/spark-ec2-wait-properly and squashes the following commits:

43a69f0 [Nicholas Chammas] short-circuit SSH check; linear backoff
9a9e035 [Nicholas Chammas] remove extraneous comment
26c5ed0 [Nicholas Chammas] replace print with write()
bb67c06 [Nicholas Chammas] deprecate wait option; remove dead code
7969265 [Nicholas Chammas] fix long line (PEP 8)
126e4cf [Nicholas Chammas] wait for specific cluster states

szhem/spark,srowen/spark,bOOm-X/spark,lvdongr/spark,Aegeaner/spark,zero323/spark,aray/spark,1haodian/spark,mdespriee/spark,pgandhi999/spark,metamx/spark,gengliangwang/spark,jiangxb1987/spark,adrian-ionescu/apache-spark,setjet/spark,lxsmnv/spark,aosagie/spark,wangyum/spark,liyichao/spark,UndeadBaneGitHub/spark,aray/spark,sachintyagi22/spark,darionyaphet/spark,rikima/spark,alunarbeach/spark,dhruve/spark,mzl9039/spark,goldmedal/spark,bdrillard/spark,esi-mineset/spark,vinodkc/spark,ddna1021/spark,eyalfa/spark,brad-kaiser/spark,saturday-shi/spark,big-pegasus/spark,milliman/spark,patrick-nicholson/spark,wangyum/spark,hhbyyh/spark,big-pegasus/spark,loneknightpy/spark,shaneknapp/spark,jianran/spark,highfei2011/spark,ioana-delaney/spark,dbtsai/spark,patrick-nicholson/spark,dhruve/spark,holdenk/spark,jianran/spark,1haodian/spark,patrick-nicholson/spark,wangmiao1981/spark,sureshthalamati/spark,taroplus/spark,cin/spark,ahnqirage/spark,xflin/spark,gengliangwang/spark,liyichao/spark,haowu80s/spark,wangmiao1981/spark,kissmetrics/spark,shubhamchopra/spark,ueshin/apache-spark,ueshin/apache-spark,map222/spark,jianran/spark,hvanhovell/spark,SnappyDataInc/spark,pronix/spark,someorz/spark,brad-kaiser/spark,mdespriee/spark,ddna1021/spark,maropu/spark,sureshthalamati/spark,kiszk/spark,xuanyuanking/spark,taroplus/spark,witgo/spark,stanzhai/spark,esi-mineset/spark,JoshRosen/spark,JerryLead/spark,WeichenXu123/spark,MLnick/spark,witgo/spark,ibm-research-ireland/sparkoscope,SHASHANKB/spark,someorz/spark,tengpeng/spark,Aegeaner/spark,xuanyuanking/spark,HyukjinKwon/spark,mahak/spark,big-pegasus/spark,saturday-shi/spark,kissmetrics/spark,techaddict/spark,byakuinss/spark,janewangfb/spark,jrshust/spark,shaneknapp/spark,wzhfy/spark,ueshin/apache-spark,saturday-shi/spark,maropu/spark,brad-kaiser/spark,apache/spark,mdespriee/spark,lvdongr/spark,rednaxelafx/apache-spark,mike0sv/spark,ddna1021/spark,holdenk/spark,Aegeaner/spark,Aegeaner/spark,ron8hu/spark,shuangshuangwang/spark,apache/spark,zuotingbing/spark,srowen/spark,icexelloss/spark,actuaryzhang/spark,byakuinss/spark,gengliangwang/spark,sahilTakiar/spark,ConeyLiu/spark,stanzhai/spark,hvanhovell/spark,UndeadBaneGitHub/spark,aokolnychyi/spark,JoshRosen/spark,jiangxb1987/spark,panfengfeng/spark,akopich/spark,shubhamchopra/spark,ioana-delaney/spark,liutang123/spark,pgandhi999/spark,HyukjinKwon/spark,mahak/spark,milliman/spark,hhbyyh/spark,lvdongr/spark,aokolnychyi/spark,nlalevee/spark,zero323/spark,bravo-zhang/spark,goldmedal/spark,rekhajoshm/spark,wzhfy/spark,jkbradley/spark,mdespriee/spark,jianran/spark,apache-spark-on-k8s/spark,spark-test/spark,zuotingbing/spark,ptkool/spark,adrian-ionescu/apache-spark,milliman/spark,rednaxelafx/apache-spark,jrshust/spark,xuanyuanking/spark,ahnqirage/spark,aokolnychyi/spark,vinodkc/spark,tengpeng/spark,shuangshuangwang/spark,maropu/spark,darionyaphet/spark,actuaryzhang/spark,dhruve/spark,guoxiaolongzte/spark,zzcclp/spark,milliman/spark,minixalpha/spark,taroplus/spark,cin/spark,esi-mineset/spark,bravo-zhang/spark,facaiy/spark,mahak/spark,akopich/spark,pgandhi999/spark,markhamstra/spark,goldmedal/spark,JerryLead/spark,lxsmnv/spark,saltstar/spark,ibm-research-ireland/sparkoscope,andrewor14/iolap,janewangfb/spark,bravo-zhang/spark,spark-test/spark,xuanyuanking/spark,highfei2011/spark,kimoonkim/spark,kevinyu98/spark,pgandhi999/spark,saturday-shi/spark,HyukjinKwon/spark,sahilTakiar/spark,liyichao/spark,milliman/spark,zhouyejoe/spark,joseph-torres/spark,wangmiao1981/spark,debugger87/spark,dongjoon-hyun/spark,setjet/spark,guoxiaolongzte/spark,ron8hu/spark,panfengfeng/spark,akopich/spark,xflin/spark,publicRoman/spark,andrewor14/iolap,hhbyyh/spark,poffuomo/spark,icexelloss/spark,aokolnychyi/spark,metamx/spark,markhamstra/spark,HyukjinKwon/spark,szhem/spark,ibm-research-ireland/sparkoscope,zzcclp/spark,minixalpha/spark,ajaysaini725/spark,joseph-torres/spark,yanboliang/spark,janewangfb/spark,SHASHANKB/spark,lvdongr/spark,sachintyagi22/spark,ajaysaini725/spark,Panos-Bletsos/spark-cost-model-optimizer,techaddict/spark,kevinyu98/spark,aray/spark,jrshust/spark,aray/spark,UndeadBaneGitHub/spark,big-pegasus/spark,holdenk/spark,dbtsai/spark,dongjoon-hyun/spark,taroplus/spark,maropu/spark,liyichao/spark,akopich/spark,patrick-nicholson/spark,jkbradley/spark,wangyum/spark,tengpeng/spark,ddna1021/spark,narahari92/spark,icexelloss/spark,witgo/spark,narahari92/spark,someorz/spark,adrian-ionescu/apache-spark,yanboliang/spark,Aegeaner/spark,publicRoman/spark,aosagie/spark,ptkool/spark,narahari92/spark,apache/spark,someorz/spark,1haodian/spark,szhem/spark,apache/spark,stanzhai/spark,aosagie/spark,highfei2011/spark,kimoonkim/spark,cin/spark,ericvandenbergfb/spark,rednaxelafx/apache-spark,wangmiao1981/spark,pgandhi999/spark,bravo-zhang/spark,ioana-delaney/spark,dbtsai/spark,tengpeng/spark,SHASHANKB/spark,shuangshuangwang/spark,bOOm-X/spark,kevinyu98/spark,sureshthalamati/spark,joseph-torres/spark,zuotingbing/spark,ibm-research-ireland/sparkoscope,shuangshuangwang/spark,liyichao/spark,metamx/spark,SHASHANKB/spark,ericvandenbergfb/spark,SnappyDataInc/spark,wangyum/spark,sachintyagi22/spark,LantaoJin/spark,tengpeng/spark,MLnick/spark,andrewor14/spark,andrewor14/spark,hvanhovell/spark,nchammas/spark,debugger87/spark,ericvandenbergfb/spark,bdrillard/spark,michalsenkyr/spark,ahnqirage/spark,vinodkc/spark,jrshust/spark,shubhamchopra/spark,JerryLead/spark,JerryLead/spark,facaiy/spark,ron8hu/spark,tejasapatil/spark,bdrillard/spark,icexelloss/spark,zuotingbing/spark,lvdongr/spark,loneknightpy/spark,witgo/spark,huang1900/spark,aokolnychyi/spark,gengliangwang/spark,zero323/spark,kimoonkim/spark,rezasafi/spark,spark-test/spark,wangmiao1981/spark,andrewor14/iolap,caneGuy/spark,jlopezmalla/spark,bOOm-X/spark,poffuomo/spark,nlalevee/spark,rednaxelafx/apache-spark,jiangxb1987/spark,big-pegasus/spark,gioenn/xSpark,publicRoman/spark,dbtsai/spark,dhruve/spark,ahnqirage/spark,haowu80s/spark,nlalevee/spark,wzhfy/spark,mzl9039/spark,minixalpha/spark,eyalfa/spark,nchammas/spark,eyalfa/spark,sahilTakiar/spark,gioenn/xSpark,aokolnychyi/spark,wzhfy/spark,kimoonkim/spark,shaneknapp/spark,alunarbeach/spark,rikima/spark,alunarbeach/spark,rezasafi/spark,actuaryzhang/spark,panfengfeng/spark,cin/spark,andrewor14/spark,WindCanDie/spark,setjet/spark,actuaryzhang/spark,publicRoman/spark,bdrillard/spark,LantaoJin/spark,srowen/spark,jkbradley/spark,zhouyejoe/spark,facaiy/spark,mike0sv/spark,matthewfranglen/spark,wangyum/spark,wzhfy/spark,andrewor14/spark,wzhfy/spark,LantaoJin/spark,liutang123/spark,wangyum/spark,szhem/spark,sachintyagi22/spark,mike0sv/spark,saturday-shi/spark,kiszk/spark,poffuomo/spark,andrewor14/iolap,wzhfy/spark,nilsgrabbert/spark,gioenn/xSpark,setjet/spark,tejasapatil/spark,hvanhovell/spark,Panos-Bletsos/spark-cost-model-optimizer,bOOm-X/spark,spark-test/spark,mahak/spark,jlopezmalla/spark,taroplus/spark,tejasapatil/spark,mike0sv/spark,haowu80s/spark,pronix/spark,chuckchen/spark,kiszk/spark,michalsenkyr/spark,jrshust/spark,jrshust/spark,gioenn/xSpark,1haodian/spark,nchammas/spark,patrick-nicholson/spark,jlopezmalla/spark,gioenn/xSpark,sureshthalamati/spark,SHASHANKB/spark,rekhajoshm/spark,highfei2011/spark,jiangxb1987/spark,kissmetrics/spark,joseph-torres/spark,gioenn/xSpark,holdenk/spark,sureshthalamati/spark,zhouyejoe/spark,minixalpha/spark,holdenk/spark,ConeyLiu/spark,someorz/spark,map222/spark,maropu/spark,ioana-delaney/spark,pronix/spark,1haodian/spark,UndeadBaneGitHub/spark,andrewor14/iolap,ConeyLiu/spark,markhamstra/spark,cloud-fan/spark,highfei2011/spark,SnappyDataInc/spark,szhem/spark,ptkool/spark,JoshRosen/spark,eyalfa/spark,chuckchen/spark,loneknightpy/spark,techaddict/spark,maropu/spark,guoxiaolongzte/spark,ConeyLiu/spark,ajaysaini725/spark,JerryLead/spark,kiszk/spark,hhbyyh/spark,metamx/spark,cin/spark,kiszk/spark,zzcclp/spark,JerryLead/spark,nilsgrabbert/spark,kevinyu98/spark,ioana-delaney/spark,dongjoon-hyun/spark,ajaysaini725/spark,rikima/spark,nlalevee/spark,ddna1021/spark,WeichenXu123/spark,sachintyagi22/spark,zhouyejoe/spark,mzl9039/spark,sahilTakiar/spark,zero323/spark,pronix/spark,BryanCutler/spark,apache/spark,byakuinss/spark,jianran/spark,maropu/spark,rekhajoshm/spark,shuangshuangwang/spark,bOOm-X/spark,tejasapatil/spark,highfei2011/spark,poffuomo/spark,chuckchen/spark,stanzhai/spark,rezasafi/spark,ericvandenbergfb/spark,haowu80s/spark,vinodkc/spark,nchammas/spark,bravo-zhang/spark,techaddict/spark,LantaoJin/spark,skonto/spark,mzl9039/spark,byakuinss/spark,nilsgrabbert/spark,ptkool/spark,rekhajoshm/spark,dongjoon-hyun/spark,saturday-shi/spark,big-pegasus/spark,pronix/spark,dhruve/spark,nchammas/spark,nilsgrabbert/spark,hvanhovell/spark,shuangshuangwang/spark,BryanCutler/spark,publicRoman/spark,srowen/spark,ibm-research-ireland/sparkoscope,lxsmnv/spark,ConeyLiu/spark,ueshin/apache-spark,zhouyejoe/spark,shubhamchopra/spark,metamx/spark,BryanCutler/spark,jiangxb1987/spark,jlopezmalla/spark,brad-kaiser/spark,SnappyDataInc/spark,xflin/spark,chuckchen/spark,zero323/spark,stanzhai/spark,icexelloss/spark,saltstar/spark,loneknightpy/spark,techaddict/spark,cin/spark,lxsmnv/spark,tengpeng/spark,ConeyLiu/spark,brad-kaiser/spark,poffuomo/spark,darionyaphet/spark,caneGuy/spark,JoshRosen/spark,yanboliang/spark,zzcclp/spark,chuckchen/spark,alunarbeach/spark,WindCanDie/spark,JoshRosen/spark,janewangfb/spark,ptkool/spark,jlopezmalla/spark,taroplus/spark,hvanhovell/spark,1haodian/spark,loneknightpy/spark,skonto/spark,nlalevee/spark,facaiy/spark,caneGuy/spark,apache-spark-on-k8s/spark,ericvandenbergfb/spark,saltstar/spark,sahilTakiar/spark,ueshin/apache-spark,dhruve/spark,mahak/spark,ajaysaini725/spark,gengliangwang/spark,guoxiaolongzte/spark,kimoonkim/spark,HyukjinKwon/spark,ptkool/spark,eyalfa/spark,kevinyu98/spark,techaddict/spark,gengliangwang/spark,shubhamchopra/spark,shubhamchopra/spark,xuanyuanking/spark,rednaxelafx/apache-spark,brad-kaiser/spark,bdrillard/spark,WeichenXu123/spark,kiszk/spark,huang1900/spark,aokolnychyi/spark,michalsenkyr/spark,cloud-fan/spark,rekhajoshm/spark,kissmetrics/spark,haowu80s/spark,ahnqirage/spark,UndeadBaneGitHub/spark,Panos-Bletsos/spark-cost-model-optimizer,jianran/spark,jlopezmalla/spark,lvdongr/spark,esi-mineset/spark,publicRoman/spark,jkbradley/spark,debugger87/spark,pgandhi999/spark,srowen/spark,1haodian/spark,dbtsai/spark,alunarbeach/spark,lxsmnv/spark,witgo/spark,dbtsai/spark,mike0sv/spark,caneGuy/spark,byakuinss/spark,tengpeng/spark,BryanCutler/spark,milliman/spark,ron8hu/spark,ioana-delaney/spark,ahnqirage/spark,sureshthalamati/spark,bdrillard/spark,darionyaphet/spark,actuaryzhang/spark,aosagie/spark,rekhajoshm/spark,vinodkc/spark,kissmetrics/spark,rezasafi/spark,WeichenXu123/spark,apache-spark-on-k8s/spark,liutang123/spark,WeichenXu123/spark,UndeadBaneGitHub/spark,panfengfeng/spark,facaiy/spark,michalsenkyr/spark,rikima/spark,skonto/spark,facaiy/spark,ioana-delaney/spark,alunarbeach/spark,BryanCutler/spark,gioenn/xSpark,tejasapatil/spark,minixalpha/spark,ddna1021/spark,rezasafi/spark,cloud-fan/spark,loneknightpy/spark,hhbyyh/spark,caneGuy/spark,tejasapatil/spark,ueshin/apache-spark,UndeadBaneGitHub/spark,hvanhovell/spark,liyichao/spark,sahilTakiar/spark,esi-mineset/spark,huang1900/spark,mzl9039/spark,adrian-ionescu/apache-spark,liutang123/spark,map222/spark,WindCanDie/spark,panfengfeng/spark,sureshthalamati/spark,debugger87/spark,goldmedal/spark,narahari92/spark,rikima/spark,SHASHANKB/spark,sahilTakiar/spark,HyukjinKwon/spark,setjet/spark,ptkool/spark,szhem/spark,ibm-research-ireland/sparkoscope,guoxiaolongzte/spark,minixalpha/spark,esi-mineset/spark,WindCanDie/spark,shaneknapp/spark,debugger87/spark,mdespriee/spark,apache-spark-on-k8s/spark,cloud-fan/spark,setjet/spark,aosagie/spark,mike0sv/spark,shaneknapp/spark,ron8hu/spark,rikima/spark,srowen/spark,xuanyuanking/spark,akopich/spark,bOOm-X/spark,akopich/spark,minixalpha/spark,aray/spark,yanboliang/spark,mdespriee/spark,vinodkc/spark,janewangfb/spark,xflin/spark,skonto/spark,narahari92/spark,wangmiao1981/spark,ddna1021/spark,Panos-Bletsos/spark-cost-model-optimizer,ConeyLiu/spark,shaneknapp/spark,nchammas/spark,goldmedal/spark,Aegeaner/spark,andrewor14/spark,joseph-torres/spark,taroplus/spark,lxsmnv/spark,vinodkc/spark,huang1900/spark,michalsenkyr/spark,hhbyyh/spark,bravo-zhang/spark,debugger87/spark,michalsenkyr/spark,dbtsai/spark,MLnick/spark,nlalevee/spark,patrick-nicholson/spark,Panos-Bletsos/spark-cost-model-optimizer,yanboliang/spark,xflin/spark,zzcclp/spark,xflin/spark,rednaxelafx/apache-spark,saltstar/spark,zzcclp/spark,WindCanDie/spark,highfei2011/spark,rezasafi/spark,cloud-fan/spark,kiszk/spark,ericvandenbergfb/spark,facaiy/spark,guoxiaolongzte/spark,mdespriee/spark,map222/spark,narahari92/spark,jkbradley/spark,byakuinss/spark,jkbradley/spark,MLnick/spark,saltstar/spark,techaddict/spark,MLnick/spark,skonto/spark,ajaysaini725/spark,markhamstra/spark,rekhajoshm/spark,LantaoJin/spark,hhbyyh/spark,panfengfeng/spark,darionyaphet/spark,shuangshuangwang/spark,apache/spark,jiangxb1987/spark,jlopezmalla/spark,SHASHANKB/spark,michalsenkyr/spark,someorz/spark,icexelloss/spark,ericvandenbergfb/spark,markhamstra/spark,BryanCutler/spark,lxsmnv/spark,adrian-ionescu/apache-spark,szhem/spark,sachintyagi22/spark,stanzhai/spark,nilsgrabbert/spark,goldmedal/spark,mahak/spark,actuaryzhang/spark,patrick-nicholson/spark,Aegeaner/spark,actuaryzhang/spark,chuckchen/spark,caneGuy/spark,zero323/spark,sachintyagi22/spark,lvdongr/spark,joseph-torres/spark,liutang123/spark,eyalfa/spark,dhruve/spark,aosagie/spark,HyukjinKwon/spark,map222/spark,zuotingbing/spark,JoshRosen/spark,SnappyDataInc/spark,WindCanDie/spark,SnappyDataInc/spark,big-pegasus/spark,map222/spark,jianran/spark,skonto/spark,liutang123/spark,LantaoJin/spark,jrshust/spark,MLnick/spark,bdrillard/spark,kimoonkim/spark,ahnqirage/spark,jiangxb1987/spark,zhouyejoe/spark,saltstar/spark,dongjoon-hyun/spark,pronix/spark,holdenk/spark,spark-test/spark,metamx/spark,alunarbeach/spark,Panos-Bletsos/spark-cost-model-optimizer,publicRoman/spark,apache/spark,andrewor14/spark,rezasafi/spark,xflin/spark,narahari92/spark,kimoonkim/spark,stanzhai/spark,Panos-Bletsos/spark-cost-model-optimizer,mzl9039/spark,huang1900/spark,ibm-research-ireland/sparkoscope,loneknightpy/spark,ueshin/apache-spark,darionyaphet/spark,gengliangwang/spark,akopich/spark,byakuinss/spark,kissmetrics/spark,adrian-ionescu/apache-spark,nlalevee/spark,joseph-torres/spark,andrewor14/spark,witgo/spark,andrewor14/iolap,mzl9039/spark,BryanCutler/spark,brad-kaiser/spark,map222/spark,chuckchen/spark,srowen/spark,setjet/spark,apache-spark-on-k8s/spark,huang1900/spark,debugger87/spark,spark-test/spark,caneGuy/spark,bravo-zhang/spark,cloud-fan/spark,jkbradley/spark,LantaoJin/spark,apache-spark-on-k8s/spark,wangyum/spark,tejasapatil/spark,liyichao/spark,kevinyu98/spark,rednaxelafx/apache-spark,aosagie/spark,saltstar/spark,zuotingbing/spark,adrian-ionescu/apache-spark,someorz/spark,mike0sv/spark,shubhamchopra/spark,MLnick/spark,holdenk/spark,eyalfa/spark,zuotingbing/spark,icexelloss/spark,saturday-shi/spark,zhouyejoe/spark,ron8hu/spark,darionyaphet/spark,poffuomo/spark,dongjoon-hyun/spark,JerryLead/spark,pgandhi999/spark,wangmiao1981/spark,cloud-fan/spark,ajaysaini725/spark,zero323/spark,milliman/spark,kevinyu98/spark,guoxiaolongzte/spark,mahak/spark,janewangfb/spark,xuanyuanking/spark,huang1900/spark,JoshRosen/spark,cin/spark,aray/spark,markhamstra/spark,yanboliang/spark,SnappyDataInc/spark,nilsgrabbert/spark,panfengfeng/spark,andrewor14/iolap,WindCanDie/spark,nchammas/spark,poffuomo/spark,rikima/spark,shaneknapp/spark,apache-spark-on-k8s/spark,metamx/spark,zzcclp/spark,goldmedal/spark,WeichenXu123/spark,skonto/spark,dongjoon-hyun/spark,kissmetrics/spark,ron8hu/spark,nilsgrabbert/spark,janewangfb/spark,markhamstra/spark,bOOm-X/spark,aray/spark,witgo/spark,haowu80s/spark,spark-test/spark,yanboliang/spark,esi-mineset/spark,WeichenXu123/spark,liutang123/spark