[SPARK-2747] git diff --dirstat can miss sql changes and not run Hive tests
[SPARK-2747] git diff --dirstat can miss sql changes and not run Hive tests

dev/run-tests use "git diff --dirstat master" to check whether sql is changed. However, --dirstat won't show sql if sql's change is negligible (e.g. 1k loc change in core, and only 1 loc change in hive).

We should use "git diff --name-only master" instead.

Author: Reynold Xin <rxin@apache.org>

Closes #1656 from rxin/hiveTest and squashes the following commits:

f5eab9f [Reynold Xin] [SPARK-2747] git diff --dirstat can miss sql changes and not run Hive tests.

rikima/spark,shaneknapp/spark,sureshthalamati/spark,wangyum/spark,aray/spark,HyukjinKwon/spark,zzcclp/spark,ron8hu/spark,ibm-research-ireland/sparkoscope,kissmetrics/spark,WeichenXu123/spark,cloud-fan/spark,saltstar/spark,zuotingbing/spark,mahak/spark,szhem/spark,publicRoman/spark,rikima/spark,HyukjinKwon/spark,dongjoon-hyun/spark,highfei2011/spark,cloud-fan/spark,dhruve/spark,guoxiaolongzte/spark,sahilTakiar/spark,mzl9039/spark,saltstar/spark,sahilTakiar/spark,SnappyDataInc/spark,brad-kaiser/spark,debugger87/spark,wangmiao1981/spark,stanzhai/spark,narahari92/spark,alunarbeach/spark,witgo/spark,BryanCutler/spark,michalsenkyr/spark,haowu80s/spark,setjet/spark,UndeadBaneGitHub/spark,UndeadBaneGitHub/spark,zuotingbing/spark,aray/spark,panfengfeng/spark,setjet/spark,narahari92/spark,tejasapatil/spark,dhruve/spark,publicRoman/spark,nchammas/spark,bOOm-X/spark,loneknightpy/spark,ueshin/apache-spark,jkbradley/spark,ahnqirage/spark,tejasapatil/spark,Aegeaner/spark,liutang123/spark,ioana-delaney/spark,mike0sv/spark,big-pegasus/spark,hhbyyh/spark,hvanhovell/spark,icexelloss/spark,stanzhai/spark,goldmedal/spark,spark-test/spark,ueshin/apache-spark,xflin/spark,andrewor14/spark,ibm-research-ireland/sparkoscope,kiszk/spark,saturday-shi/spark,LantaoJin/spark,SnappyDataInc/spark,JoshRosen/spark,ptkool/spark,ericvandenbergfb/spark,publicRoman/spark,Panos-Bletsos/spark-cost-model-optimizer,huang1900/spark,michalsenkyr/spark,stanzhai/spark,setjet/spark,hhbyyh/spark,highfei2011/spark,ericvandenbergfb/spark,zzcclp/spark,tejasapatil/spark,metamx/spark,guoxiaolongzte/spark,sahilTakiar/spark,facaiy/spark,ConeyLiu/spark,WindCanDie/spark,1haodian/spark,rezasafi/spark,lxsmnv/spark,someorz/spark,rezasafi/spark,skonto/spark,pgandhi999/spark,poffuomo/spark,eyalfa/spark,vinodkc/spark,markhamstra/spark,cloud-fan/spark,bdrillard/spark,caneGuy/spark,WeichenXu123/spark,LantaoJin/spark,michalsenkyr/spark,ueshin/apache-spark,patrick-nicholson/spark,ddna1021/spark,janewangfb/spark,ptkool/spark,xuanyuanking/spark,aokolnychyi/spark,taroplus/spark,patrick-nicholson/spark,rednaxelafx/apache-spark,kissmetrics/spark,bravo-zhang/spark,janewangfb/spark,dbtsai/spark,ibm-research-ireland/sparkoscope,highfei2011/spark,jianran/spark,rekhajoshm/spark,rikima/spark,ddna1021/spark,saltstar/spark,brad-kaiser/spark,lxsmnv/spark,highfei2011/spark,liutang123/spark,markhamstra/spark,WindCanDie/spark,jkbradley/spark,Aegeaner/spark,apache/spark,UndeadBaneGitHub/spark,goldmedal/spark,techaddict/spark,xuanyuanking/spark,nilsgrabbert/spark,WindCanDie/spark,maropu/spark,bdrillard/spark,narahari92/spark,sahilTakiar/spark,panfengfeng/spark,pgandhi999/spark,jiangxb1987/spark,sureshthalamati/spark,kimoonkim/spark,ajaysaini725/spark,BryanCutler/spark,alunarbeach/spark,LantaoJin/spark,zhouyejoe/spark,metamx/spark,mike0sv/spark,mdespriee/spark,zero323/spark,JerryLead/spark,markhamstra/spark,zuotingbing/spark,MLnick/spark,adrian-ionescu/apache-spark,gengliangwang/spark,rekhajoshm/spark,cin/spark,zero323/spark,lvdongr/spark,kiszk/spark,loneknightpy/spark,highfei2011/spark,dhruve/spark,xuanyuanking/spark,UndeadBaneGitHub/spark,skonto/spark,kimoonkim/spark,zero323/spark,dongjoon-hyun/spark,sureshthalamati/spark,kevinyu98/spark,jianran/spark,tengpeng/spark,joseph-torres/spark,dongjoon-hyun/spark,eyalfa/spark,setjet/spark,goldmedal/spark,poffuomo/spark,pronix/spark,cloud-fan/spark,debugger87/spark,saturday-shi/spark,JoshRosen/spark,wzhfy/spark,zzcclp/spark,map222/spark,rekhajoshm/spark,mike0sv/spark,hhbyyh/spark,holdenk/spark,shaneknapp/spark,shuangshuangwang/spark,someorz/spark,actuaryzhang/spark,narahari92/spark,liyichao/spark,ioana-delaney/spark,shubhamchopra/spark,BryanCutler/spark,rezasafi/spark,dhruve/spark,apache-spark-on-k8s/spark,UndeadBaneGitHub/spark,WindCanDie/spark,joseph-torres/spark,tengpeng/spark,gioenn/xSpark,aosagie/spark,vinodkc/spark,shuangshuangwang/spark,wangyum/spark,facaiy/spark,aokolnychyi/spark,cin/spark,publicRoman/spark,cloud-fan/spark,witgo/spark,matthewfranglen/spark,zhouyejoe/spark,patrick-nicholson/spark,haowu80s/spark,cloud-fan/spark,yanboliang/spark,icexelloss/spark,ron8hu/spark,alunarbeach/spark,sureshthalamati/spark,wangyum/spark,dbtsai/spark,liutang123/spark,sahilTakiar/spark,big-pegasus/spark,witgo/spark,sahilTakiar/spark,BryanCutler/spark,sahilTakiar/spark,rezasafi/spark,wzhfy/spark,nilsgrabbert/spark,andrewor14/spark,srowen/spark,LantaoJin/spark,ibm-research-ireland/sparkoscope,dhruve/spark,bravo-zhang/spark,kevinyu98/spark,1haodian/spark,kissmetrics/spark,pgandhi999/spark,jkbradley/spark,MLnick/spark,highfei2011/spark,janewangfb/spark,adrian-ionescu/apache-spark,nilsgrabbert/spark,setjet/spark,andrewor14/iolap,szhem/spark,rednaxelafx/apache-spark,ddna1021/spark,markhamstra/spark,michalsenkyr/spark,WeichenXu123/spark,tengpeng/spark,mdespriee/spark,gioenn/xSpark,caneGuy/spark,janewangfb/spark,wangmiao1981/spark,JoshRosen/spark,bdrillard/spark,akopich/spark,akopich/spark,panfengfeng/spark,maropu/spark,caneGuy/spark,WeichenXu123/spark,akopich/spark,ConeyLiu/spark,panfengfeng/spark,xuanyuanking/spark,actuaryzhang/spark,liutang123/spark,rikima/spark,adrian-ionescu/apache-spark,HyukjinKwon/spark,bdrillard/spark,dongjoon-hyun/spark,guoxiaolongzte/spark,zuotingbing/spark,stanzhai/spark,wzhfy/spark,apache/spark,esi-mineset/spark,sachintyagi22/spark,kiszk/spark,skonto/spark,kissmetrics/spark,nilsgrabbert/spark,metamx/spark,alunarbeach/spark,kissmetrics/spark,big-pegasus/spark,caneGuy/spark,jlopezmalla/spark,chuckchen/spark,techaddict/spark,ajaysaini725/spark,patrick-nicholson/spark,vinodkc/spark,aray/spark,jlopezmalla/spark,debugger87/spark,wzhfy/spark,ibm-research-ireland/sparkoscope,gengliangwang/spark,wzhfy/spark,markhamstra/spark,shubhamchopra/spark,yanboliang/spark,akopich/spark,jianran/spark,tejasapatil/spark,dhruve/spark,SHASHANKB/spark,milliman/spark,eyalfa/spark,Aegeaner/spark,narahari92/spark,gioenn/xSpark,bdrillard/spark,joseph-torres/spark,bravo-zhang/spark,SHASHANKB/spark,ahnqirage/spark,ahnqirage/spark,lvdongr/spark,pgandhi999/spark,dbtsai/spark,WeichenXu123/spark,byakuinss/spark,huang1900/spark,skonto/spark,MLnick/spark,aray/spark,SHASHANKB/spark,zhouyejoe/spark,xuanyuanking/spark,eyalfa/spark,pgandhi999/spark,setjet/spark,tengpeng/spark,pgandhi999/spark,sachintyagi22/spark,minixalpha/spark,chuckchen/spark,techaddict/spark,ConeyLiu/spark,lxsmnv/spark,maropu/spark,jrshust/spark,eyalfa/spark,facaiy/spark,eyalfa/spark,skonto/spark,debugger87/spark,nlalevee/spark,dbtsai/spark,sachintyagi22/spark,mdespriee/spark,aosagie/spark,gioenn/xSpark,apache-spark-on-k8s/spark,lvdongr/spark,ptkool/spark,ioana-delaney/spark,kissmetrics/spark,yanboliang/spark,ahnqirage/spark,apache/spark,patrick-nicholson/spark,gengliangwang/spark,publicRoman/spark,aokolnychyi/spark,shuangshuangwang/spark,shubhamchopra/spark,wangyum/spark,shuangshuangwang/spark,minixalpha/spark,kevinyu98/spark,bdrillard/spark,JerryLead/spark,LantaoJin/spark,jianran/spark,huang1900/spark,ericvandenbergfb/spark,ConeyLiu/spark,saltstar/spark,ioana-delaney/spark,big-pegasus/spark,kevinyu98/spark,ajaysaini725/spark,map222/spark,brad-kaiser/spark,dongjoon-hyun/spark,ConeyLiu/spark,aokolnychyi/spark,saturday-shi/spark,ahnqirage/spark,ibm-research-ireland/sparkoscope,rekhajoshm/spark,bravo-zhang/spark,eyalfa/spark,debugger87/spark,nchammas/spark,big-pegasus/spark,poffuomo/spark,taroplus/spark,witgo/spark,debugger87/spark,ddna1021/spark,shubhamchopra/spark,srowen/spark,ron8hu/spark,nilsgrabbert/spark,dhruve/spark,shaneknapp/spark,huang1900/spark,jrshust/spark,gioenn/xSpark,Panos-Bletsos/spark-cost-model-optimizer,chuckchen/spark,yanboliang/spark,big-pegasus/spark,Panos-Bletsos/spark-cost-model-optimizer,gengliangwang/spark,SnappyDataInc/spark,ahnqirage/spark,darionyaphet/spark,tengpeng/spark,adrian-ionescu/apache-spark,wangyum/spark,JerryLead/spark,ConeyLiu/spark,srowen/spark,lvdongr/spark,holdenk/spark,narahari92/spark,huang1900/spark,rednaxelafx/apache-spark,jiangxb1987/spark,witgo/spark,shaneknapp/spark,zuotingbing/spark,mahak/spark,UndeadBaneGitHub/spark,aray/spark,setjet/spark,publicRoman/spark,loneknightpy/spark,xflin/spark,byakuinss/spark,darionyaphet/spark,spark-test/spark,pronix/spark,zzcclp/spark,Aegeaner/spark,gioenn/xSpark,joseph-torres/spark,facaiy/spark,stanzhai/spark,ptkool/spark,apache/spark,lxsmnv/spark,aosagie/spark,ptkool/spark,caneGuy/spark,JoshRosen/spark,JoshRosen/spark,cin/spark,kevinyu98/spark,SHASHANKB/spark,darionyaphet/spark,patrick-nicholson/spark,skonto/spark,zero323/spark,ericvandenbergfb/spark,mike0sv/spark,Panos-Bletsos/spark-cost-model-optimizer,JerryLead/spark,shaneknapp/spark,jkbradley/spark,jianran/spark,hhbyyh/spark,SHASHANKB/spark,szhem/spark,alunarbeach/spark,big-pegasus/spark,joseph-torres/spark,akopich/spark,adrian-ionescu/apache-spark,aokolnychyi/spark,icexelloss/spark,maropu/spark,liyichao/spark,bOOm-X/spark,liutang123/spark,rednaxelafx/apache-spark,vinodkc/spark,ueshin/apache-spark,kimoonkim/spark,sachintyagi22/spark,shuangshuangwang/spark,liutang123/spark,caneGuy/spark,jrshust/spark,ajaysaini725/spark,zhouyejoe/spark,mahak/spark,mdespriee/spark,bravo-zhang/spark,WeichenXu123/spark,map222/spark,SHASHANKB/spark,WindCanDie/spark,goldmedal/spark,JoshRosen/spark,saturday-shi/spark,UndeadBaneGitHub/spark,MLnick/spark,ron8hu/spark,JerryLead/spark,WindCanDie/spark,panfengfeng/spark,ueshin/apache-spark,saturday-shi/spark,zzcclp/spark,holdenk/spark,hhbyyh/spark,kimoonkim/spark,someorz/spark,mdespriee/spark,andrewor14/spark,apache-spark-on-k8s/spark,bravo-zhang/spark,andrewor14/spark,zhouyejoe/spark,nchammas/spark,esi-mineset/spark,poffuomo/spark,jkbradley/spark,maropu/spark,jlopezmalla/spark,jiangxb1987/spark,ericvandenbergfb/spark,brad-kaiser/spark,patrick-nicholson/spark,darionyaphet/spark,nlalevee/spark,ron8hu/spark,loneknightpy/spark,mzl9039/spark,poffuomo/spark,mdespriee/spark,jrshust/spark,kimoonkim/spark,tengpeng/spark,SHASHANKB/spark,ioana-delaney/spark,xuanyuanking/spark,bOOm-X/spark,metamx/spark,someorz/spark,saturday-shi/spark,dongjoon-hyun/spark,wzhfy/spark,someorz/spark,publicRoman/spark,BryanCutler/spark,sureshthalamati/spark,loneknightpy/spark,zzcclp/spark,holdenk/spark,ajaysaini725/spark,nchammas/spark,alunarbeach/spark,Panos-Bletsos/spark-cost-model-optimizer,chuckchen/spark,hvanhovell/spark,taroplus/spark,bravo-zhang/spark,yanboliang/spark,milliman/spark,jiangxb1987/spark,SnappyDataInc/spark,shubhamchopra/spark,dbtsai/spark,HyukjinKwon/spark,brad-kaiser/spark,goldmedal/spark,gioenn/xSpark,icexelloss/spark,akopich/spark,adrian-ionescu/apache-spark,nlalevee/spark,mahak/spark,milliman/spark,kimoonkim/spark,liyichao/spark,hvanhovell/spark,haowu80s/spark,minixalpha/spark,1haodian/spark,rekhajoshm/spark,spark-test/spark,yanboliang/spark,metamx/spark,Aegeaner/spark,aokolnychyi/spark,shaneknapp/spark,ajaysaini725/spark,mzl9039/spark,srowen/spark,yanboliang/spark,metamx/spark,debugger87/spark,metamx/spark,milliman/spark,holdenk/spark,apache-spark-on-k8s/spark,zhouyejoe/spark,nchammas/spark,andrewor14/iolap,ueshin/apache-spark,nchammas/spark,ron8hu/spark,maropu/spark,szhem/spark,pronix/spark,aray/spark,huang1900/spark,rednaxelafx/apache-spark,jkbradley/spark,nchammas/spark,apache/spark,zero323/spark,ron8hu/spark,minixalpha/spark,tengpeng/spark,apache/spark,wangmiao1981/spark,goldmedal/spark,esi-mineset/spark,kevinyu98/spark,pronix/spark,spark-test/spark,byakuinss/spark,1haodian/spark,techaddict/spark,apache/spark,WindCanDie/spark,jiangxb1987/spark,highfei2011/spark,spark-test/spark,hvanhovell/spark,tejasapatil/spark,xuanyuanking/spark,rekhajoshm/spark,esi-mineset/spark,jrshust/spark,liyichao/spark,actuaryzhang/spark,adrian-ionescu/apache-spark,cin/spark,wangmiao1981/spark,someorz/spark,vinodkc/spark,srowen/spark,byakuinss/spark,sachintyagi22/spark,alunarbeach/spark,nlalevee/spark,hvanhovell/spark,lvdongr/spark,HyukjinKwon/spark,ioana-delaney/spark,JoshRosen/spark,sureshthalamati/spark,jlopezmalla/spark,actuaryzhang/spark,andrewor14/iolap,rednaxelafx/apache-spark,map222/spark,janewangfb/spark,dbtsai/spark,haowu80s/spark,kimoonkim/spark,facaiy/spark,1haodian/spark,jlopezmalla/spark,cin/spark,gengliangwang/spark,saltstar/spark,hhbyyh/spark,mike0sv/spark,rezasafi/spark,mzl9039/spark,kiszk/spark,BryanCutler/spark,xflin/spark,minixalpha/spark,michalsenkyr/spark,andrewor14/iolap,jianran/spark,taroplus/spark,zero323/spark,stanzhai/spark,mzl9039/spark,nlalevee/spark,holdenk/spark,icexelloss/spark,actuaryzhang/spark,srowen/spark,dbtsai/spark,cin/spark,chuckchen/spark,rikima/spark,shubhamchopra/spark,rednaxelafx/apache-spark,gengliangwang/spark,taroplus/spark,JerryLead/spark,xflin/spark,dongjoon-hyun/spark,hvanhovell/spark,shubhamchopra/spark,lxsmnv/spark,loneknightpy/spark,jkbradley/spark,mzl9039/spark,vinodkc/spark,SnappyDataInc/spark,actuaryzhang/spark,jrshust/spark,holdenk/spark,bOOm-X/spark,mahak/spark,srowen/spark,zero323/spark,HyukjinKwon/spark,taroplus/spark,aosagie/spark,markhamstra/spark,apache-spark-on-k8s/spark,byakuinss/spark,MLnick/spark,Panos-Bletsos/spark-cost-model-optimizer,loneknightpy/spark,guoxiaolongzte/spark,actuaryzhang/spark,guoxiaolongzte/spark,minixalpha/spark,shuangshuangwang/spark,ueshin/apache-spark,xflin/spark,poffuomo/spark,LantaoJin/spark,ptkool/spark,aosagie/spark,haowu80s/spark,aray/spark,huang1900/spark,ptkool/spark,wangmiao1981/spark,maropu/spark,andrewor14/iolap,jiangxb1987/spark,saltstar/spark,cin/spark,nlalevee/spark,narahari92/spark,szhem/spark,ericvandenbergfb/spark,ConeyLiu/spark,brad-kaiser/spark,darionyaphet/spark,Aegeaner/spark,HyukjinKwon/spark,LantaoJin/spark,ddna1021/spark,ajaysaini725/spark,liyichao/spark,wangmiao1981/spark,stanzhai/spark,kissmetrics/spark,cloud-fan/spark,kiszk/spark,rekhajoshm/spark,Aegeaner/spark,aosagie/spark,milliman/spark,lvdongr/spark,chuckchen/spark,shuangshuangwang/spark,lxsmnv/spark,minixalpha/spark,sureshthalamati/spark,BryanCutler/spark,milliman/spark,esi-mineset/spark,caneGuy/spark,xflin/spark,nlalevee/spark,darionyaphet/spark,icexelloss/spark,ddna1021/spark,tejasapatil/spark,wzhfy/spark,haowu80s/spark,liyichao/spark,esi-mineset/spark,panfengfeng/spark,janewangfb/spark,ahnqirage/spark,bdrillard/spark,guoxiaolongzte/spark,sachintyagi22/spark,xflin/spark,liutang123/spark,liyichao/spark,mike0sv/spark,techaddict/spark,zuotingbing/spark,sachintyagi22/spark,hhbyyh/spark,apache-spark-on-k8s/spark,pronix/spark,witgo/spark,rezasafi/spark,techaddict/spark,kiszk/spark,saturday-shi/spark,kevinyu98/spark,nilsgrabbert/spark,akopich/spark,ioana-delaney/spark,pgandhi999/spark,JerryLead/spark,jlopezmalla/spark,gengliangwang/spark,mahak/spark,wangyum/spark,poffuomo/spark,tejasapatil/spark,zzcclp/spark,map222/spark,byakuinss/spark,saltstar/spark,jianran/spark,szhem/spark,markhamstra/spark,facaiy/spark,spark-test/spark,ibm-research-ireland/sparkoscope,michalsenkyr/spark,rikima/spark,mahak/spark,jrshust/spark,mzl9039/spark,andrewor14/iolap,lxsmnv/spark,joseph-torres/spark,facaiy/spark,map222/spark,kiszk/spark,aosagie/spark,spark-test/spark,apache-spark-on-k8s/spark,ericvandenbergfb/spark,MLnick/spark,michalsenkyr/spark,mike0sv/spark,1haodian/spark,byakuinss/spark,nilsgrabbert/spark,andrewor14/spark,esi-mineset/spark,andrewor14/spark,aokolnychyi/spark,icexelloss/spark,darionyaphet/spark,Panos-Bletsos/spark-cost-model-optimizer,panfengfeng/spark,rikima/spark,mdespriee/spark,andrewor14/iolap,vinodkc/spark,wangmiao1981/spark,jiangxb1987/spark,shaneknapp/spark,map222/spark,ddna1021/spark,andrewor14/spark,guoxiaolongzte/spark,rezasafi/spark,bOOm-X/spark,joseph-torres/spark,witgo/spark,taroplus/spark,SnappyDataInc/spark,MLnick/spark,milliman/spark,wangyum/spark,goldmedal/spark,WeichenXu123/spark,someorz/spark,janewangfb/spark,techaddict/spark,pronix/spark,bOOm-X/spark,SnappyDataInc/spark,1haodian/spark,hvanhovell/spark,lvdongr/spark,chuckchen/spark,skonto/spark,bOOm-X/spark,szhem/spark,brad-kaiser/spark,jlopezmalla/spark,zhouyejoe/spark,zuotingbing/spark