[EC2] [SPARK-6188] Instance types can be mislabeled when re-starting cluster with default arguments
[EC2] [SPARK-6188] Instance types can be mislabeled when re-starting cluster with default arguments

As described in https://issues.apache.org/jira/browse/SPARK-6188 and discovered in https://issues.apache.org/jira/browse/SPARK-5838.

When re-starting a cluster, if the user does not provide the instance types, which is the recommended behavior in the docs currently, the instance will be assigned the default type m1.large. This then affects the setup of the machines.

This solves this by getting the instance types from the existing instances, and overwriting the default options.

EDIT: Further clarification of the issue:

In short, while the instances themselves are the same as launched, their setup is done assuming the default instance type, m1.large.

This means that the machines are assumed to have 2 disks, and that leads to problems that are described in in issue [5838](https://issues.apache.org/jira/browse/SPARK-5838), where machines that have one disk end up having shuffle spills in the in the small (8GB) snapshot partitions that quickly fills up and results in failing jobs due to "No space left on device" errors.

Other instance specific settings that are set in the spark_ec2.py script are likely to be wrong as well.

Author: Theodore Vasiloudis <thvasilo@users.noreply.github.com>
Author: Theodore Vasiloudis <tvas@sics.se>

Closes #4916 from thvasilo/SPARK-6188]-Instance-types-can-be-mislabeled-when-re-starting-cluster-with-default-arguments and squashes the following commits:

6705b98 [Theodore Vasiloudis] Added comment to clarify setting master instance type to the empty string.
a3d29fe [Theodore Vasiloudis] More trailing whitespace
7b32429 [Theodore Vasiloudis] Removed trailing whitespace
3ebd52a [Theodore Vasiloudis] Make sure that the instance type is correct when relaunching a cluster.

goldmedal/spark,ConeyLiu/spark,haowu80s/spark,WeichenXu123/spark,bravo-zhang/spark,haowu80s/spark,michalsenkyr/spark,esi-mineset/spark,rekhajoshm/spark,MLnick/spark,joseph-torres/spark,kiszk/spark,zuotingbing/spark,zero323/spark,pronix/spark,chuckchen/spark,shaneknapp/spark,bdrillard/spark,actuaryzhang/spark,ptkool/spark,liutang123/spark,patrick-nicholson/spark,wangyum/spark,patrick-nicholson/spark,zuotingbing/spark,kimoonkim/spark,jiangxb1987/spark,chuckchen/spark,spark-test/spark,shaneknapp/spark,shuangshuangwang/spark,darionyaphet/spark,rezasafi/spark,aray/spark,spark-test/spark,nchammas/spark,nilsgrabbert/spark,bOOm-X/spark,WindCanDie/spark,szhem/spark,holdenk/spark,spark-test/spark,panfengfeng/spark,xflin/spark,andrewor14/spark,JoshRosen/spark,jiangxb1987/spark,wangyum/spark,jiangxb1987/spark,ddna1021/spark,haowu80s/spark,gengliangwang/spark,milliman/spark,icexelloss/spark,LantaoJin/spark,vinodkc/spark,actuaryzhang/spark,xuanyuanking/spark,witgo/spark,saltstar/spark,lxsmnv/spark,someorz/spark,JoshRosen/spark,wzhfy/spark,rikima/spark,mdespriee/spark,skonto/spark,bOOm-X/spark,jrshust/spark,andrewor14/spark,kimoonkim/spark,debugger87/spark,eyalfa/spark,rezasafi/spark,HyukjinKwon/spark,janewangfb/spark,ahnqirage/spark,taroplus/spark,shuangshuangwang/spark,poffuomo/spark,ConeyLiu/spark,maropu/spark,zhouyejoe/spark,ConeyLiu/spark,kiszk/spark,sahilTakiar/spark,metamx/spark,JoshRosen/spark,tejasapatil/spark,guoxiaolongzte/spark,icexelloss/spark,cloud-fan/spark,wzhfy/spark,map222/spark,mahak/spark,setjet/spark,ptkool/spark,sachintyagi22/spark,dhruve/spark,rezasafi/spark,cloud-fan/spark,lvdongr/spark,liyichao/spark,dongjoon-hyun/spark,liutang123/spark,jianran/spark,ericvandenbergfb/spark,jkbradley/spark,ahnqirage/spark,janewangfb/spark,1haodian/spark,szhem/spark,cloud-fan/spark,ioana-delaney/spark,WindCanDie/spark,michalsenkyr/spark,bOOm-X/spark,shubhamchopra/spark,1haodian/spark,bravo-zhang/spark,panfengfeng/spark,caneGuy/spark,ahnqirage/spark,facaiy/spark,Aegeaner/spark,JerryLead/spark,apache-spark-on-k8s/spark,kimoonkim/spark,facaiy/spark,sureshthalamati/spark,zuotingbing/spark,wangmiao1981/spark,loneknightpy/spark,tejasapatil/spark,eyalfa/spark,aosagie/spark,cloud-fan/spark,ron8hu/spark,eyalfa/spark,minixalpha/spark,spark-test/spark,wangyum/spark,xflin/spark,maropu/spark,kevinyu98/spark,Aegeaner/spark,srowen/spark,1haodian/spark,andrewor14/iolap,shubhamchopra/spark,minixalpha/spark,kimoonkim/spark,xflin/spark,ibm-research-ireland/sparkoscope,ddna1021/spark,ericvandenbergfb/spark,gioenn/xSpark,narahari92/spark,tejasapatil/spark,jlopezmalla/spark,saturday-shi/spark,darionyaphet/spark,wangyum/spark,kevinyu98/spark,patrick-nicholson/spark,someorz/spark,goldmedal/spark,lxsmnv/spark,yanboliang/spark,hhbyyh/spark,loneknightpy/spark,joseph-torres/spark,nchammas/spark,xuanyuanking/spark,srowen/spark,andrewor14/spark,panfengfeng/spark,michalsenkyr/spark,WindCanDie/spark,pgandhi999/spark,rekhajoshm/spark,lxsmnv/spark,alunarbeach/spark,icexelloss/spark,map222/spark,nilsgrabbert/spark,sureshthalamati/spark,markhamstra/spark,zzcclp/spark,tengpeng/spark,brad-kaiser/spark,xuanyuanking/spark,kiszk/spark,huang1900/spark,xflin/spark,minixalpha/spark,eyalfa/spark,SHASHANKB/spark,ron8hu/spark,markhamstra/spark,mahak/spark,apache/spark,joseph-torres/spark,mike0sv/spark,jkbradley/spark,yanboliang/spark,ericvandenbergfb/spark,alunarbeach/spark,ibm-research-ireland/sparkoscope,gengliangwang/spark,saltstar/spark,aokolnychyi/spark,tejasapatil/spark,panfengfeng/spark,byakuinss/spark,jkbradley/spark,bravo-zhang/spark,zhouyejoe/spark,jlopezmalla/spark,mike0sv/spark,sureshthalamati/spark,facaiy/spark,techaddict/spark,dbtsai/spark,shubhamchopra/spark,akopich/spark,markhamstra/spark,poffuomo/spark,kevinyu98/spark,jlopezmalla/spark,nlalevee/spark,Panos-Bletsos/spark-cost-model-optimizer,cin/spark,pronix/spark,zero323/spark,zero323/spark,hvanhovell/spark,cin/spark,highfei2011/spark,andrewor14/iolap,Aegeaner/spark,setjet/spark,witgo/spark,SnappyDataInc/spark,SHASHANKB/spark,gioenn/xSpark,andrewor14/spark,srowen/spark,wangmiao1981/spark,big-pegasus/spark,HyukjinKwon/spark,aosagie/spark,sahilTakiar/spark,witgo/spark,ajaysaini725/spark,1haodian/spark,alunarbeach/spark,zuotingbing/spark,ajaysaini725/spark,ron8hu/spark,dhruve/spark,JerryLead/spark,rednaxelafx/apache-spark,alunarbeach/spark,jrshust/spark,techaddict/spark,cin/spark,zzcclp/spark,zzcclp/spark,zero323/spark,jianran/spark,sahilTakiar/spark,icexelloss/spark,wzhfy/spark,guoxiaolongzte/spark,dbtsai/spark,nilsgrabbert/spark,holdenk/spark,rezasafi/spark,sachintyagi22/spark,szhem/spark,ptkool/spark,jiangxb1987/spark,joseph-torres/spark,tengpeng/spark,holdenk/spark,wzhfy/spark,aray/spark,pgandhi999/spark,jkbradley/spark,brad-kaiser/spark,ibm-research-ireland/sparkoscope,andrewor14/spark,aokolnychyi/spark,LantaoJin/spark,bravo-zhang/spark,rikima/spark,zhouyejoe/spark,aokolnychyi/spark,saltstar/spark,adrian-ionescu/apache-spark,adrian-ionescu/apache-spark,haowu80s/spark,poffuomo/spark,sachintyagi22/spark,mdespriee/spark,nilsgrabbert/spark,yanboliang/spark,yanboliang/spark,sachintyagi22/spark,kissmetrics/spark,ibm-research-ireland/sparkoscope,vinodkc/spark,debugger87/spark,sachintyagi22/spark,metamx/spark,hvanhovell/spark,ioana-delaney/spark,1haodian/spark,minixalpha/spark,hvanhovell/spark,jkbradley/spark,big-pegasus/spark,jrshust/spark,byakuinss/spark,stanzhai/spark,techaddict/spark,wangyum/spark,byakuinss/spark,ptkool/spark,sahilTakiar/spark,esi-mineset/spark,nlalevee/spark,MLnick/spark,markhamstra/spark,zzcclp/spark,bdrillard/spark,darionyaphet/spark,pronix/spark,shuangshuangwang/spark,minixalpha/spark,shaneknapp/spark,zhouyejoe/spark,taroplus/spark,dbtsai/spark,jlopezmalla/spark,mzl9039/spark,janewangfb/spark,debugger87/spark,MLnick/spark,liyichao/spark,gioenn/xSpark,actuaryzhang/spark,nchammas/spark,kiszk/spark,shaneknapp/spark,someorz/spark,xuanyuanking/spark,mdespriee/spark,zuotingbing/spark,publicRoman/spark,wzhfy/spark,guoxiaolongzte/spark,stanzhai/spark,mahak/spark,patrick-nicholson/spark,ptkool/spark,LantaoJin/spark,szhem/spark,facaiy/spark,map222/spark,mzl9039/spark,ibm-research-ireland/sparkoscope,publicRoman/spark,ConeyLiu/spark,goldmedal/spark,akopich/spark,loneknightpy/spark,nlalevee/spark,publicRoman/spark,vinodkc/spark,matthewfranglen/spark,BryanCutler/spark,rednaxelafx/apache-spark,mzl9039/spark,shuangshuangwang/spark,debugger87/spark,esi-mineset/spark,saturday-shi/spark,mdespriee/spark,jianran/spark,UndeadBaneGitHub/spark,aokolnychyi/spark,BryanCutler/spark,wangmiao1981/spark,byakuinss/spark,ueshin/apache-spark,tejasapatil/spark,tengpeng/spark,apache/spark,szhem/spark,jrshust/spark,SnappyDataInc/spark,sachintyagi22/spark,gioenn/xSpark,actuaryzhang/spark,WeichenXu123/spark,BryanCutler/spark,BryanCutler/spark,jrshust/spark,gengliangwang/spark,sureshthalamati/spark,apache/spark,aokolnychyi/spark,taroplus/spark,saltstar/spark,map222/spark,zero323/spark,JoshRosen/spark,hhbyyh/spark,spark-test/spark,rekhajoshm/spark,apache-spark-on-k8s/spark,lxsmnv/spark,mdespriee/spark,dhruve/spark,rikima/spark,tengpeng/spark,ericvandenbergfb/spark,liyichao/spark,cloud-fan/spark,jiangxb1987/spark,publicRoman/spark,chuckchen/spark,narahari92/spark,rekhajoshm/spark,sureshthalamati/spark,witgo/spark,darionyaphet/spark,milliman/spark,WindCanDie/spark,ueshin/apache-spark,aray/spark,kimoonkim/spark,SnappyDataInc/spark,lxsmnv/spark,yanboliang/spark,metamx/spark,bdrillard/spark,JerryLead/spark,skonto/spark,alunarbeach/spark,hvanhovell/spark,xuanyuanking/spark,guoxiaolongzte/spark,wangyum/spark,dongjoon-hyun/spark,rikima/spark,zhouyejoe/spark,michalsenkyr/spark,kissmetrics/spark,zhouyejoe/spark,1haodian/spark,HyukjinKwon/spark,saltstar/spark,nilsgrabbert/spark,skonto/spark,gioenn/xSpark,andrewor14/spark,dhruve/spark,aray/spark,gioenn/xSpark,huang1900/spark,yanboliang/spark,poffuomo/spark,hhbyyh/spark,ddna1021/spark,ioana-delaney/spark,kissmetrics/spark,kiszk/spark,narahari92/spark,nlalevee/spark,patrick-nicholson/spark,facaiy/spark,bdrillard/spark,caneGuy/spark,liutang123/spark,nlalevee/spark,apache/spark,ddna1021/spark,cin/spark,ajaysaini725/spark,HyukjinKwon/spark,metamx/spark,wangmiao1981/spark,WeichenXu123/spark,holdenk/spark,pronix/spark,goldmedal/spark,highfei2011/spark,techaddict/spark,jkbradley/spark,lvdongr/spark,BryanCutler/spark,ConeyLiu/spark,someorz/spark,kissmetrics/spark,hhbyyh/spark,huang1900/spark,markhamstra/spark,milliman/spark,HyukjinKwon/spark,rednaxelafx/apache-spark,jrshust/spark,vinodkc/spark,yanboliang/spark,publicRoman/spark,loneknightpy/spark,MLnick/spark,markhamstra/spark,nilsgrabbert/spark,loneknightpy/spark,jiangxb1987/spark,kissmetrics/spark,guoxiaolongzte/spark,bOOm-X/spark,JerryLead/spark,bdrillard/spark,michalsenkyr/spark,narahari92/spark,ajaysaini725/spark,holdenk/spark,huang1900/spark,srowen/spark,SnappyDataInc/spark,lvdongr/spark,ConeyLiu/spark,kevinyu98/spark,jianran/spark,zzcclp/spark,wangmiao1981/spark,metamx/spark,stanzhai/spark,eyalfa/spark,pgandhi999/spark,ajaysaini725/spark,setjet/spark,chuckchen/spark,caneGuy/spark,aosagie/spark,vinodkc/spark,lvdongr/spark,rednaxelafx/apache-spark,aosagie/spark,kissmetrics/spark,facaiy/spark,kiszk/spark,mzl9039/spark,rezasafi/spark,sahilTakiar/spark,kiszk/spark,jkbradley/spark,apache/spark,jlopezmalla/spark,dbtsai/spark,michalsenkyr/spark,gengliangwang/spark,shuangshuangwang/spark,vinodkc/spark,kimoonkim/spark,1haodian/spark,stanzhai/spark,gioenn/xSpark,kimoonkim/spark,mahak/spark,LantaoJin/spark,debugger87/spark,dbtsai/spark,goldmedal/spark,mike0sv/spark,map222/spark,SnappyDataInc/spark,andrewor14/iolap,akopich/spark,brad-kaiser/spark,saturday-shi/spark,alunarbeach/spark,markhamstra/spark,tengpeng/spark,techaddict/spark,caneGuy/spark,WindCanDie/spark,zuotingbing/spark,setjet/spark,Panos-Bletsos/spark-cost-model-optimizer,saltstar/spark,JerryLead/spark,andrewor14/iolap,ioana-delaney/spark,brad-kaiser/spark,tejasapatil/spark,UndeadBaneGitHub/spark,saturday-shi/spark,metamx/spark,apache-spark-on-k8s/spark,aray/spark,apache-spark-on-k8s/spark,cloud-fan/spark,Aegeaner/spark,maropu/spark,dbtsai/spark,SnappyDataInc/spark,SnappyDataInc/spark,aosagie/spark,rikima/spark,Panos-Bletsos/spark-cost-model-optimizer,saltstar/spark,dongjoon-hyun/spark,eyalfa/spark,dhruve/spark,guoxiaolongzte/spark,aokolnychyi/spark,cin/spark,ahnqirage/spark,hhbyyh/spark,mzl9039/spark,Panos-Bletsos/spark-cost-model-optimizer,WeichenXu123/spark,ConeyLiu/spark,gengliangwang/spark,ueshin/apache-spark,eyalfa/spark,brad-kaiser/spark,ron8hu/spark,bdrillard/spark,liyichao/spark,janewangfb/spark,mahak/spark,joseph-torres/spark,pgandhi999/spark,taroplus/spark,srowen/spark,hvanhovell/spark,map222/spark,esi-mineset/spark,kevinyu98/spark,srowen/spark,rednaxelafx/apache-spark,aosagie/spark,dbtsai/spark,pgandhi999/spark,panfengfeng/spark,pronix/spark,ron8hu/spark,cin/spark,narahari92/spark,wzhfy/spark,guoxiaolongzte/spark,poffuomo/spark,milliman/spark,apache/spark,mahak/spark,ptkool/spark,liutang123/spark,debugger87/spark,poffuomo/spark,icexelloss/spark,esi-mineset/spark,mahak/spark,brad-kaiser/spark,janewangfb/spark,taroplus/spark,jianran/spark,rednaxelafx/apache-spark,andrewor14/iolap,BryanCutler/spark,big-pegasus/spark,sachintyagi22/spark,publicRoman/spark,pgandhi999/spark,kevinyu98/spark,shubhamchopra/spark,chuckchen/spark,nilsgrabbert/spark,goldmedal/spark,SHASHANKB/spark,huang1900/spark,hhbyyh/spark,jlopezmalla/spark,ron8hu/spark,rezasafi/spark,WeichenXu123/spark,setjet/spark,LantaoJin/spark,sahilTakiar/spark,map222/spark,huang1900/spark,narahari92/spark,byakuinss/spark,ajaysaini725/spark,shubhamchopra/spark,HyukjinKwon/spark,nchammas/spark,mzl9039/spark,chuckchen/spark,SHASHANKB/spark,ioana-delaney/spark,michalsenkyr/spark,icexelloss/spark,ron8hu/spark,highfei2011/spark,mzl9039/spark,liyichao/spark,JerryLead/spark,JoshRosen/spark,WindCanDie/spark,xuanyuanking/spark,UndeadBaneGitHub/spark,skonto/spark,janewangfb/spark,andrewor14/iolap,saturday-shi/spark,Panos-Bletsos/spark-cost-model-optimizer,apache-spark-on-k8s/spark,HyukjinKwon/spark,dhruve/spark,actuaryzhang/spark,ibm-research-ireland/sparkoscope,maropu/spark,shaneknapp/spark,dongjoon-hyun/spark,sahilTakiar/spark,wangmiao1981/spark,sureshthalamati/spark,wangmiao1981/spark,facaiy/spark,stanzhai/spark,ajaysaini725/spark,ahnqirage/spark,bOOm-X/spark,ddna1021/spark,minixalpha/spark,highfei2011/spark,adrian-ionescu/apache-spark,byakuinss/spark,shuangshuangwang/spark,saturday-shi/spark,UndeadBaneGitHub/spark,witgo/spark,Aegeaner/spark,ddna1021/spark,ahnqirage/spark,rikima/spark,apache/spark,mike0sv/spark,ptkool/spark,highfei2011/spark,rekhajoshm/spark,esi-mineset/spark,mike0sv/spark,spark-test/spark,maropu/spark,debugger87/spark,gengliangwang/spark,huang1900/spark,SHASHANKB/spark,andrewor14/iolap,dongjoon-hyun/spark,aosagie/spark,MLnick/spark,skonto/spark,jlopezmalla/spark,techaddict/spark,MLnick/spark,ibm-research-ireland/sparkoscope,shubhamchopra/spark,wangyum/spark,minixalpha/spark,JoshRosen/spark,bOOm-X/spark,tengpeng/spark,akopich/spark,panfengfeng/spark,loneknightpy/spark,hvanhovell/spark,lxsmnv/spark,adrian-ionescu/apache-spark,jianran/spark,witgo/spark,UndeadBaneGitHub/spark,xflin/spark,mike0sv/spark,SHASHANKB/spark,milliman/spark,holdenk/spark,holdenk/spark,rikima/spark,xflin/spark,mike0sv/spark,SHASHANKB/spark,haowu80s/spark,WeichenXu123/spark,big-pegasus/spark,zero323/spark,LantaoJin/spark,ioana-delaney/spark,rezasafi/spark,stanzhai/spark,big-pegasus/spark,liyichao/spark,JoshRosen/spark,caneGuy/spark,dhruve/spark,big-pegasus/spark,ahnqirage/spark,lvdongr/spark,alunarbeach/spark,liutang123/spark,narahari92/spark,WeichenXu123/spark,poffuomo/spark,xflin/spark,caneGuy/spark,janewangfb/spark,rekhajoshm/spark,darionyaphet/spark,joseph-torres/spark,ueshin/apache-spark,pronix/spark,ericvandenbergfb/spark,Panos-Bletsos/spark-cost-model-optimizer,UndeadBaneGitHub/spark,xuanyuanking/spark,kevinyu98/spark,srowen/spark,dongjoon-hyun/spark,shubhamchopra/spark,dongjoon-hyun/spark,bravo-zhang/spark,andrewor14/spark,esi-mineset/spark,ddna1021/spark,setjet/spark,rednaxelafx/apache-spark,zuotingbing/spark,zero323/spark,techaddict/spark,maropu/spark,vinodkc/spark,akopich/spark,setjet/spark,szhem/spark,adrian-ionescu/apache-spark,bravo-zhang/spark,mdespriee/spark,rekhajoshm/spark,bravo-zhang/spark,cloud-fan/spark,liutang123/spark,bOOm-X/spark,sureshthalamati/spark,ioana-delaney/spark,shaneknapp/spark,aray/spark,ueshin/apache-spark,Aegeaner/spark,lxsmnv/spark,panfengfeng/spark,maropu/spark,darionyaphet/spark,stanzhai/spark,loneknightpy/spark,icexelloss/spark,someorz/spark,haowu80s/spark,hhbyyh/spark,apache-spark-on-k8s/spark,ueshin/apache-spark,darionyaphet/spark,taroplus/spark,kissmetrics/spark,milliman/spark,taroplus/spark,someorz/spark,nlalevee/spark,joseph-torres/spark,brad-kaiser/spark,Panos-Bletsos/spark-cost-model-optimizer,adrian-ionescu/apache-spark,gengliangwang/spark,actuaryzhang/spark,nchammas/spark,shuangshuangwang/spark,jianran/spark,zzcclp/spark,LantaoJin/spark,UndeadBaneGitHub/spark,shaneknapp/spark,szhem/spark,MLnick/spark,skonto/spark,nlalevee/spark,skonto/spark,nchammas/spark,lvdongr/spark,witgo/spark,liyichao/spark,JerryLead/spark,apache-spark-on-k8s/spark,ericvandenbergfb/spark,ericvandenbergfb/spark,zzcclp/spark,milliman/spark,caneGuy/spark,cin/spark,jrshust/spark,mdespriee/spark,goldmedal/spark,aokolnychyi/spark,akopich/spark,highfei2011/spark,hvanhovell/spark,chuckchen/spark,big-pegasus/spark,pgandhi999/spark,liutang123/spark,byakuinss/spark,spark-test/spark,BryanCutler/spark,nchammas/spark,publicRoman/spark,bdrillard/spark,metamx/spark,zhouyejoe/spark,ueshin/apache-spark,wzhfy/spark,lvdongr/spark,someorz/spark,Aegeaner/spark,akopich/spark,patrick-nicholson/spark,WindCanDie/spark,tengpeng/spark,patrick-nicholson/spark,saturday-shi/spark,highfei2011/spark,aray/spark,jiangxb1987/spark,actuaryzhang/spark,adrian-ionescu/apache-spark,tejasapatil/spark