[SPARK-8576] Add spark-ec2 options to set IAM roles and instance-initiated shutdown behavior
[SPARK-8576] Add spark-ec2 options to set IAM roles and instance-initiated shutdown behavior

Both of these options are useful when spark-ec2 is being used as part of an automated pipeline and the engineers want to minimize the need to pass around AWS keys for access to things like S3 (keys are replaced by the IAM role) and to be able to launch a cluster that can terminate itself cleanly.

Author: Nicholas Chammas <nicholas.chammas@gmail.com>

Closes #6962 from nchammas/additional-ec2-options and squashes the following commits:

fcf252e [Nicholas Chammas] PEP8 fixes
efba9ee [Nicholas Chammas] add help for --instance-initiated-shutdown-behavior
598aecf [Nicholas Chammas] option to launch instances into IAM role
2743632 [Nicholas Chammas] add option for instance initiated shutdown

ConeyLiu/spark,zzcclp/spark,kevinyu98/spark,hvanhovell/spark,guoxiaolongzte/spark,sureshthalamati/spark,lxsmnv/spark,joseph-torres/spark,xuanyuanking/spark,liyichao/spark,hvanhovell/spark,xflin/spark,shubhamchopra/spark,debugger87/spark,Aegeaner/spark,adrian-ionescu/apache-spark,loneknightpy/spark,narahari92/spark,eyalfa/spark,icexelloss/spark,facaiy/spark,poffuomo/spark,esi-mineset/spark,wzhfy/spark,haowu80s/spark,ericvandenbergfb/spark,UndeadBaneGitHub/spark,bdrillard/spark,nchammas/spark,bravo-zhang/spark,JerryLead/spark,spark-test/spark,apache-spark-on-k8s/spark,esi-mineset/spark,caneGuy/spark,apache/spark,gioenn/xSpark,byakuinss/spark,cin/spark,adrian-ionescu/apache-spark,pgandhi999/spark,liyichao/spark,mike0sv/spark,SnappyDataInc/spark,ptkool/spark,chuckchen/spark,jianran/spark,tengpeng/spark,hhbyyh/spark,yanboliang/spark,cin/spark,andrewor14/spark,rekhajoshm/spark,huang1900/spark,bravo-zhang/spark,akopich/spark,zuotingbing/spark,jlopezmalla/spark,Aegeaner/spark,rednaxelafx/apache-spark,rikima/spark,dongjoon-hyun/spark,ron8hu/spark,poffuomo/spark,jiangxb1987/spark,zhouyejoe/spark,stanzhai/spark,stanzhai/spark,tengpeng/spark,markhamstra/spark,HyukjinKwon/spark,mike0sv/spark,zhouyejoe/spark,aokolnychyi/spark,mzl9039/spark,nilsgrabbert/spark,rekhajoshm/spark,bdrillard/spark,gengliangwang/spark,guoxiaolongzte/spark,lxsmnv/spark,JerryLead/spark,JerryLead/spark,liutang123/spark,joseph-torres/spark,shuangshuangwang/spark,kiszk/spark,jrshust/spark,kevinyu98/spark,shuangshuangwang/spark,big-pegasus/spark,saturday-shi/spark,rednaxelafx/apache-spark,darionyaphet/spark,rednaxelafx/apache-spark,kevinyu98/spark,kimoonkim/spark,BryanCutler/spark,srowen/spark,zuotingbing/spark,xflin/spark,kissmetrics/spark,goldmedal/spark,jkbradley/spark,wangyum/spark,markhamstra/spark,ddna1021/spark,wangmiao1981/spark,setjet/spark,joseph-torres/spark,shaneknapp/spark,wangyum/spark,ConeyLiu/spark,ajaysaini725/spark,lvdongr/spark,tejasapatil/spark,byakuinss/spark,WeichenXu123/spark,ajaysaini725/spark,apache/spark,ueshin/apache-spark,Aegeaner/spark,shubhamchopra/spark,JoshRosen/spark,sureshthalamati/spark,BryanCutler/spark,ibm-research-ireland/sparkoscope,bravo-zhang/spark,lxsmnv/spark,stanzhai/spark,patrick-nicholson/spark,SnappyDataInc/spark,yanboliang/spark,ddna1021/spark,jianran/spark,ioana-delaney/spark,pgandhi999/spark,patrick-nicholson/spark,1haodian/spark,ericvandenbergfb/spark,mike0sv/spark,pgandhi999/spark,ajaysaini725/spark,taroplus/spark,techaddict/spark,adrian-ionescu/apache-spark,goldmedal/spark,ddna1021/spark,hvanhovell/spark,milliman/spark,big-pegasus/spark,guoxiaolongzte/spark,janewangfb/spark,map222/spark,jlopezmalla/spark,rednaxelafx/apache-spark,ahnqirage/spark,mahak/spark,haowu80s/spark,milliman/spark,Panos-Bletsos/spark-cost-model-optimizer,hvanhovell/spark,LantaoJin/spark,chuckchen/spark,alunarbeach/spark,rekhajoshm/spark,joseph-torres/spark,rikima/spark,witgo/spark,nlalevee/spark,gioenn/xSpark,tejasapatil/spark,andrewor14/spark,hhbyyh/spark,bOOm-X/spark,witgo/spark,ConeyLiu/spark,jlopezmalla/spark,zzcclp/spark,aray/spark,aosagie/spark,liutang123/spark,jiangxb1987/spark,lxsmnv/spark,huang1900/spark,metamx/spark,minixalpha/spark,huang1900/spark,xuanyuanking/spark,patrick-nicholson/spark,mike0sv/spark,JoshRosen/spark,debugger87/spark,aray/spark,skonto/spark,bdrillard/spark,map222/spark,cloud-fan/spark,shubhamchopra/spark,yanboliang/spark,lvdongr/spark,JerryLead/spark,UndeadBaneGitHub/spark,publicRoman/spark,mdespriee/spark,gioenn/xSpark,maropu/spark,panfengfeng/spark,hvanhovell/spark,saturday-shi/spark,byakuinss/spark,ConeyLiu/spark,lxsmnv/spark,narahari92/spark,alunarbeach/spark,metamx/spark,markhamstra/spark,eyalfa/spark,kiszk/spark,sachintyagi22/spark,xflin/spark,lvdongr/spark,srowen/spark,darionyaphet/spark,ron8hu/spark,pgandhi999/spark,ptkool/spark,wzhfy/spark,nilsgrabbert/spark,LantaoJin/spark,adrian-ionescu/apache-spark,liutang123/spark,wangyum/spark,panfengfeng/spark,poffuomo/spark,panfengfeng/spark,SnappyDataInc/spark,jlopezmalla/spark,jianran/spark,ahnqirage/spark,mdespriee/spark,minixalpha/spark,nchammas/spark,zhouyejoe/spark,gengliangwang/spark,ioana-delaney/spark,HyukjinKwon/spark,aosagie/spark,JerryLead/spark,maropu/spark,rednaxelafx/apache-spark,loneknightpy/spark,MLnick/spark,janewangfb/spark,esi-mineset/spark,joseph-torres/spark,mike0sv/spark,vinodkc/spark,kissmetrics/spark,tejasapatil/spark,alunarbeach/spark,minixalpha/spark,setjet/spark,dhruve/spark,WeichenXu123/spark,eyalfa/spark,bravo-zhang/spark,brad-kaiser/spark,rezasafi/spark,shaneknapp/spark,rikima/spark,MLnick/spark,nchammas/spark,stanzhai/spark,ioana-delaney/spark,jkbradley/spark,sureshthalamati/spark,jianran/spark,gengliangwang/spark,ueshin/apache-spark,vinodkc/spark,Aegeaner/spark,someorz/spark,nilsgrabbert/spark,vinodkc/spark,ahnqirage/spark,zero323/spark,BryanCutler/spark,jianran/spark,mzl9039/spark,apache-spark-on-k8s/spark,saltstar/spark,bdrillard/spark,1haodian/spark,zhouyejoe/spark,tejasapatil/spark,darionyaphet/spark,minixalpha/spark,publicRoman/spark,map222/spark,kissmetrics/spark,brad-kaiser/spark,shaneknapp/spark,sachintyagi22/spark,markhamstra/spark,shuangshuangwang/spark,gioenn/xSpark,apache-spark-on-k8s/spark,szhem/spark,ueshin/apache-spark,kevinyu98/spark,narahari92/spark,vinodkc/spark,1haodian/spark,maropu/spark,zuotingbing/spark,janewangfb/spark,milliman/spark,big-pegasus/spark,aosagie/spark,Panos-Bletsos/spark-cost-model-optimizer,bOOm-X/spark,actuaryzhang/spark,metamx/spark,poffuomo/spark,caneGuy/spark,MLnick/spark,byakuinss/spark,aosagie/spark,wangmiao1981/spark,BryanCutler/spark,huang1900/spark,publicRoman/spark,witgo/spark,matthewfranglen/spark,SnappyDataInc/spark,ron8hu/spark,mzl9039/spark,pronix/spark,markhamstra/spark,someorz/spark,big-pegasus/spark,wzhfy/spark,zero323/spark,ahnqirage/spark,bOOm-X/spark,ericvandenbergfb/spark,tengpeng/spark,zero323/spark,taroplus/spark,ron8hu/spark,hvanhovell/spark,goldmedal/spark,alunarbeach/spark,rikima/spark,srowen/spark,xflin/spark,sureshthalamati/spark,eyalfa/spark,liyichao/spark,wangmiao1981/spark,pronix/spark,dongjoon-hyun/spark,wzhfy/spark,dongjoon-hyun/spark,HyukjinKwon/spark,janewangfb/spark,highfei2011/spark,loneknightpy/spark,aosagie/spark,bravo-zhang/spark,chuckchen/spark,mike0sv/spark,panfengfeng/spark,pgandhi999/spark,someorz/spark,michalsenkyr/spark,kiszk/spark,mdespriee/spark,caneGuy/spark,brad-kaiser/spark,caneGuy/spark,bOOm-X/spark,ptkool/spark,UndeadBaneGitHub/spark,cin/spark,haowu80s/spark,pronix/spark,loneknightpy/spark,wangyum/spark,kiszk/spark,JoshRosen/spark,jkbradley/spark,Panos-Bletsos/spark-cost-model-optimizer,jrshust/spark,WindCanDie/spark,wangmiao1981/spark,huang1900/spark,Aegeaner/spark,andrewor14/spark,dbtsai/spark,saltstar/spark,map222/spark,panfengfeng/spark,akopich/spark,sachintyagi22/spark,ibm-research-ireland/sparkoscope,janewangfb/spark,mahak/spark,facaiy/spark,ericvandenbergfb/spark,apache-spark-on-k8s/spark,ajaysaini725/spark,ericvandenbergfb/spark,setjet/spark,goldmedal/spark,saltstar/spark,rezasafi/spark,minixalpha/spark,MLnick/spark,big-pegasus/spark,markhamstra/spark,LantaoJin/spark,icexelloss/spark,spark-test/spark,milliman/spark,szhem/spark,goldmedal/spark,techaddict/spark,cloud-fan/spark,SHASHANKB/spark,ericvandenbergfb/spark,eyalfa/spark,poffuomo/spark,brad-kaiser/spark,srowen/spark,highfei2011/spark,skonto/spark,icexelloss/spark,jlopezmalla/spark,jiangxb1987/spark,zero323/spark,publicRoman/spark,wzhfy/spark,liutang123/spark,Panos-Bletsos/spark-cost-model-optimizer,rikima/spark,zhouyejoe/spark,techaddict/spark,ibm-research-ireland/sparkoscope,zuotingbing/spark,zuotingbing/spark,dhruve/spark,esi-mineset/spark,sahilTakiar/spark,dhruve/spark,vinodkc/spark,ptkool/spark,poffuomo/spark,srowen/spark,nlalevee/spark,aokolnychyi/spark,xflin/spark,haowu80s/spark,saturday-shi/spark,WindCanDie/spark,alunarbeach/spark,WindCanDie/spark,rednaxelafx/apache-spark,pronix/spark,darionyaphet/spark,holdenk/spark,highfei2011/spark,hhbyyh/spark,sahilTakiar/spark,loneknightpy/spark,michalsenkyr/spark,nchammas/spark,saltstar/spark,someorz/spark,liutang123/spark,yanboliang/spark,highfei2011/spark,aray/spark,caneGuy/spark,ibm-research-ireland/sparkoscope,nilsgrabbert/spark,WindCanDie/spark,aosagie/spark,guoxiaolongzte/spark,WindCanDie/spark,kissmetrics/spark,holdenk/spark,stanzhai/spark,aokolnychyi/spark,techaddict/spark,aosagie/spark,tejasapatil/spark,cloud-fan/spark,milliman/spark,mzl9039/spark,SHASHANKB/spark,ajaysaini725/spark,janewangfb/spark,holdenk/spark,liyichao/spark,witgo/spark,jrshust/spark,kimoonkim/spark,tengpeng/spark,HyukjinKwon/spark,aray/spark,loneknightpy/spark,nilsgrabbert/spark,nchammas/spark,dhruve/spark,joseph-torres/spark,jiangxb1987/spark,wangmiao1981/spark,byakuinss/spark,panfengfeng/spark,icexelloss/spark,taroplus/spark,ddna1021/spark,cloud-fan/spark,minixalpha/spark,patrick-nicholson/spark,dhruve/spark,actuaryzhang/spark,bravo-zhang/spark,HyukjinKwon/spark,ConeyLiu/spark,witgo/spark,shaneknapp/spark,MLnick/spark,saturday-shi/spark,holdenk/spark,shubhamchopra/spark,publicRoman/spark,gioenn/xSpark,kissmetrics/spark,pronix/spark,dhruve/spark,eyalfa/spark,JoshRosen/spark,rikima/spark,skonto/spark,rezasafi/spark,rezasafi/spark,MLnick/spark,dbtsai/spark,xuanyuanking/spark,shuangshuangwang/spark,ddna1021/spark,dongjoon-hyun/spark,patrick-nicholson/spark,nlalevee/spark,mdespriee/spark,debugger87/spark,saturday-shi/spark,dbtsai/spark,hhbyyh/spark,xuanyuanking/spark,shaneknapp/spark,sachintyagi22/spark,bdrillard/spark,shubhamchopra/spark,ioana-delaney/spark,SnappyDataInc/spark,mzl9039/spark,eyalfa/spark,zero323/spark,techaddict/spark,janewangfb/spark,goldmedal/spark,nilsgrabbert/spark,setjet/spark,ptkool/spark,map222/spark,actuaryzhang/spark,tengpeng/spark,nlalevee/spark,pgandhi999/spark,ajaysaini725/spark,skonto/spark,dbtsai/spark,sahilTakiar/spark,brad-kaiser/spark,publicRoman/spark,mahak/spark,mzl9039/spark,guoxiaolongzte/spark,aokolnychyi/spark,ddna1021/spark,pronix/spark,lvdongr/spark,adrian-ionescu/apache-spark,SHASHANKB/spark,caneGuy/spark,nlalevee/spark,bOOm-X/spark,nlalevee/spark,andrewor14/spark,yanboliang/spark,HyukjinKwon/spark,minixalpha/spark,zzcclp/spark,szhem/spark,xflin/spark,saturday-shi/spark,WindCanDie/spark,dbtsai/spark,szhem/spark,debugger87/spark,akopich/spark,kimoonkim/spark,akopich/spark,szhem/spark,huang1900/spark,yanboliang/spark,cin/spark,milliman/spark,ConeyLiu/spark,ajaysaini725/spark,zhouyejoe/spark,rekhajoshm/spark,ibm-research-ireland/sparkoscope,nchammas/spark,UndeadBaneGitHub/spark,mahak/spark,holdenk/spark,liyichao/spark,JerryLead/spark,caneGuy/spark,JerryLead/spark,techaddict/spark,rikima/spark,big-pegasus/spark,guoxiaolongzte/spark,sachintyagi22/spark,facaiy/spark,andrewor14/spark,spark-test/spark,ahnqirage/spark,lxsmnv/spark,lvdongr/spark,michalsenkyr/spark,setjet/spark,JoshRosen/spark,chuckchen/spark,ueshin/apache-spark,sachintyagi22/spark,facaiy/spark,wangyum/spark,publicRoman/spark,chuckchen/spark,dongjoon-hyun/spark,ConeyLiu/spark,huang1900/spark,vinodkc/spark,ericvandenbergfb/spark,akopich/spark,kevinyu98/spark,big-pegasus/spark,ron8hu/spark,mahak/spark,debugger87/spark,1haodian/spark,bdrillard/spark,hvanhovell/spark,sahilTakiar/spark,jlopezmalla/spark,kiszk/spark,liutang123/spark,tengpeng/spark,cin/spark,rekhajoshm/spark,darionyaphet/spark,LantaoJin/spark,SHASHANKB/spark,WeichenXu123/spark,esi-mineset/spark,esi-mineset/spark,sahilTakiar/spark,rekhajoshm/spark,highfei2011/spark,gengliangwang/spark,cloud-fan/spark,saturday-shi/spark,dhruve/spark,icexelloss/spark,UndeadBaneGitHub/spark,apache/spark,spark-test/spark,jianran/spark,SnappyDataInc/spark,jianran/spark,jiangxb1987/spark,ron8hu/spark,cloud-fan/spark,zzcclp/spark,mzl9039/spark,icexelloss/spark,tejasapatil/spark,srowen/spark,SHASHANKB/spark,stanzhai/spark,ioana-delaney/spark,Aegeaner/spark,zhouyejoe/spark,zuotingbing/spark,mdespriee/spark,saltstar/spark,taroplus/spark,Panos-Bletsos/spark-cost-model-optimizer,spark-test/spark,setjet/spark,jkbradley/spark,shuangshuangwang/spark,kissmetrics/spark,jrshust/spark,saltstar/spark,michalsenkyr/spark,metamx/spark,jiangxb1987/spark,liyichao/spark,goldmedal/spark,tengpeng/spark,chuckchen/spark,1haodian/spark,spark-test/spark,UndeadBaneGitHub/spark,bdrillard/spark,ueshin/apache-spark,map222/spark,kimoonkim/spark,cin/spark,wangyum/spark,ptkool/spark,LantaoJin/spark,spark-test/spark,aokolnychyi/spark,skonto/spark,narahari92/spark,ahnqirage/spark,LantaoJin/spark,brad-kaiser/spark,bravo-zhang/spark,sahilTakiar/spark,alunarbeach/spark,wzhfy/spark,ibm-research-ireland/sparkoscope,aray/spark,apache/spark,cin/spark,wangyum/spark,nchammas/spark,apache-spark-on-k8s/spark,jkbradley/spark,map222/spark,milliman/spark,BryanCutler/spark,michalsenkyr/spark,xuanyuanking/spark,joseph-torres/spark,jkbradley/spark,narahari92/spark,metamx/spark,highfei2011/spark,sureshthalamati/spark,HyukjinKwon/spark,patrick-nicholson/spark,apache/spark,actuaryzhang/spark,holdenk/spark,bOOm-X/spark,apache/spark,shaneknapp/spark,sureshthalamati/spark,narahari92/spark,sureshthalamati/spark,zzcclp/spark,WeichenXu123/spark,SHASHANKB/spark,zero323/spark,yanboliang/spark,tejasapatil/spark,adrian-ionescu/apache-spark,darionyaphet/spark,nilsgrabbert/spark,hhbyyh/spark,ioana-delaney/spark,apache-spark-on-k8s/spark,WeichenXu123/spark,SHASHANKB/spark,rezasafi/spark,kevinyu98/spark,dbtsai/spark,MLnick/spark,someorz/spark,srowen/spark,sahilTakiar/spark,someorz/spark,vinodkc/spark,saltstar/spark,Aegeaner/spark,rezasafi/spark,gioenn/xSpark,skonto/spark,mahak/spark,ueshin/apache-spark,alunarbeach/spark,shuangshuangwang/spark,ron8hu/spark,michalsenkyr/spark,darionyaphet/spark,mdespriee/spark,cloud-fan/spark,debugger87/spark,1haodian/spark,ptkool/spark,taroplus/spark,xflin/spark,holdenk/spark,shuangshuangwang/spark,facaiy/spark,WeichenXu123/spark,maropu/spark,sachintyagi22/spark,Panos-Bletsos/spark-cost-model-optimizer,panfengfeng/spark,apache-spark-on-k8s/spark,kimoonkim/spark,gioenn/xSpark,adrian-ionescu/apache-spark,WindCanDie/spark,markhamstra/spark,esi-mineset/spark,byakuinss/spark,aray/spark,wzhfy/spark,byakuinss/spark,witgo/spark,jrshust/spark,mike0sv/spark,Panos-Bletsos/spark-cost-model-optimizer,gengliangwang/spark,dongjoon-hyun/spark,ddna1021/spark,BryanCutler/spark,metamx/spark,maropu/spark,shubhamchopra/spark,kimoonkim/spark,jkbradley/spark,BryanCutler/spark,jrshust/spark,andrewor14/spark,setjet/spark,LantaoJin/spark,zero323/spark,jlopezmalla/spark,aokolnychyi/spark,lvdongr/spark,jiangxb1987/spark,ahnqirage/spark,metamx/spark,poffuomo/spark,aray/spark,gengliangwang/spark,hhbyyh/spark,kimoonkim/spark,witgo/spark,andrewor14/spark,wangmiao1981/spark,haowu80s/spark,kiszk/spark,narahari92/spark,actuaryzhang/spark,michalsenkyr/spark,akopich/spark,shubhamchopra/spark,ueshin/apache-spark,nlalevee/spark,dbtsai/spark,rezasafi/spark,taroplus/spark,dongjoon-hyun/spark,ibm-research-ireland/sparkoscope,brad-kaiser/spark,1haodian/spark,rekhajoshm/spark,liyichao/spark,zzcclp/spark,taroplus/spark,mahak/spark,rednaxelafx/apache-spark,techaddict/spark,szhem/spark,liutang123/spark,JoshRosen/spark,kissmetrics/spark,skonto/spark,stanzhai/spark,debugger87/spark,xuanyuanking/spark,someorz/spark,icexelloss/spark,maropu/spark,pgandhi999/spark,guoxiaolongzte/spark,apache/spark,lvdongr/spark,facaiy/spark,WeichenXu123/spark,kiszk/spark,akopich/spark,SnappyDataInc/spark,aokolnychyi/spark,JoshRosen/spark,patrick-nicholson/spark,szhem/spark,bOOm-X/spark,chuckchen/spark,haowu80s/spark,facaiy/spark,hhbyyh/spark,actuaryzhang/spark,xuanyuanking/spark,loneknightpy/spark,maropu/spark,mdespriee/spark,zuotingbing/spark,UndeadBaneGitHub/spark,highfei2011/spark,ioana-delaney/spark,zzcclp/spark,kevinyu98/spark,lxsmnv/spark,shaneknapp/spark,jrshust/spark,gengliangwang/spark,wangmiao1981/spark,actuaryzhang/spark