[SPARK-5808] [build] Package pyspark files in sbt assembly.
[SPARK-5808] [build] Package pyspark files in sbt assembly.

This turned out to be more complicated than I wanted because the
layout of python/ doesn't really follow the usual maven conventions.
So some extra code is needed to copy just the right things.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #5461 from vanzin/SPARK-5808 and squashes the following commits:

7153dac [Marcelo Vanzin] Only try to create resource dir if it doesn't already exist.
ee90e84 [Marcelo Vanzin] [SPARK-5808] [build] Package pyspark files in sbt assembly.

JoshRosen/spark,cloud-fan/spark,akopich/spark,wangyum/spark,WeichenXu123/spark,milliman/spark,saltstar/spark,kevinyu98/spark,facaiy/spark,darionyaphet/spark,hvanhovell/spark,WeichenXu123/spark,nilsgrabbert/spark,darionyaphet/spark,actuaryzhang/spark,zhouyejoe/spark,milliman/spark,vinodkc/spark,akopich/spark,sureshthalamati/spark,LantaoJin/spark,ahnqirage/spark,highfei2011/spark,zhouyejoe/spark,hhbyyh/spark,facaiy/spark,narahari92/spark,byakuinss/spark,saturday-shi/spark,bdrillard/spark,gengliangwang/spark,tejasapatil/spark,ioana-delaney/spark,bdrillard/spark,WeichenXu123/spark,nchammas/spark,icexelloss/spark,Aegeaner/spark,xflin/spark,andrewor14/iolap,alunarbeach/spark,dbtsai/spark,aray/spark,goldmedal/spark,techaddict/spark,liyichao/spark,tengpeng/spark,shubhamchopra/spark,esi-mineset/spark,jrshust/spark,icexelloss/spark,gioenn/xSpark,witgo/spark,jianran/spark,brad-kaiser/spark,guoxiaolongzte/spark,poffuomo/spark,markhamstra/spark,nilsgrabbert/spark,ConeyLiu/spark,cloud-fan/spark,shaneknapp/spark,xflin/spark,nchammas/spark,ioana-delaney/spark,apache/spark,gioenn/xSpark,michalsenkyr/spark,skonto/spark,goldmedal/spark,vinodkc/spark,metamx/spark,michalsenkyr/spark,narahari92/spark,big-pegasus/spark,SHASHANKB/spark,rezasafi/spark,zzcclp/spark,lvdongr/spark,taroplus/spark,sureshthalamati/spark,akopich/spark,szhem/spark,akopich/spark,publicRoman/spark,shubhamchopra/spark,wzhfy/spark,yanboliang/spark,wzhfy/spark,cloud-fan/spark,saturday-shi/spark,publicRoman/spark,hvanhovell/spark,kissmetrics/spark,skonto/spark,haowu80s/spark,mzl9039/spark,SnappyDataInc/spark,andrewor14/iolap,rednaxelafx/apache-spark,pgandhi999/spark,jlopezmalla/spark,darionyaphet/spark,nchammas/spark,spark-test/spark,jrshust/spark,hhbyyh/spark,lxsmnv/spark,alunarbeach/spark,minixalpha/spark,adrian-ionescu/apache-spark,WindCanDie/spark,rekhajoshm/spark,WindCanDie/spark,saltstar/spark,ueshin/apache-spark,zero323/spark,rezasafi/spark,markhamstra/spark,nchammas/spark,ron8hu/spark,jkbradley/spark,rezasafi/spark,haowu80s/spark,esi-mineset/spark,bdrillard/spark,setjet/spark,ConeyLiu/spark,saturday-shi/spark,jiangxb1987/spark,SnappyDataInc/spark,1haodian/spark,sachintyagi22/spark,ibm-research-ireland/sparkoscope,poffuomo/spark,MLnick/spark,highfei2011/spark,JoshRosen/spark,shuangshuangwang/spark,kimoonkim/spark,kimoonkim/spark,ibm-research-ireland/sparkoscope,skonto/spark,panfengfeng/spark,nilsgrabbert/spark,publicRoman/spark,ajaysaini725/spark,chuckchen/spark,panfengfeng/spark,shaneknapp/spark,ptkool/spark,ajaysaini725/spark,aosagie/spark,HyukjinKwon/spark,liyichao/spark,zuotingbing/spark,srowen/spark,map222/spark,jiangxb1987/spark,BryanCutler/spark,esi-mineset/spark,maropu/spark,rednaxelafx/apache-spark,dongjoon-hyun/spark,Panos-Bletsos/spark-cost-model-optimizer,vinodkc/spark,ron8hu/spark,dbtsai/spark,debugger87/spark,bravo-zhang/spark,1haodian/spark,eyalfa/spark,Panos-Bletsos/spark-cost-model-optimizer,hvanhovell/spark,ajaysaini725/spark,dhruve/spark,setjet/spark,rikima/spark,sureshthalamati/spark,facaiy/spark,aray/spark,mzl9039/spark,MLnick/spark,mike0sv/spark,joseph-torres/spark,alunarbeach/spark,ericvandenbergfb/spark,eyalfa/spark,xflin/spark,xuanyuanking/spark,jianran/spark,maropu/spark,LantaoJin/spark,SnappyDataInc/spark,bOOm-X/spark,someorz/spark,poffuomo/spark,SHASHANKB/spark,apache-spark-on-k8s/spark,minixalpha/spark,hhbyyh/spark,mzl9039/spark,debugger87/spark,sachintyagi22/spark,JerryLead/spark,ioana-delaney/spark,UndeadBaneGitHub/spark,jkbradley/spark,kiszk/spark,apache-spark-on-k8s/spark,bOOm-X/spark,cin/spark,SnappyDataInc/spark,yanboliang/spark,map222/spark,adrian-ionescu/apache-spark,zero323/spark,wangmiao1981/spark,dhruve/spark,dongjoon-hyun/spark,tengpeng/spark,aosagie/spark,mahak/spark,shubhamchopra/spark,srowen/spark,pronix/spark,gengliangwang/spark,Aegeaner/spark,holdenk/spark,patrick-nicholson/spark,andrewor14/spark,spark-test/spark,zhouyejoe/spark,SnappyDataInc/spark,mahak/spark,ericvandenbergfb/spark,saturday-shi/spark,skonto/spark,shaneknapp/spark,ptkool/spark,minixalpha/spark,andrewor14/spark,gioenn/xSpark,sahilTakiar/spark,kevinyu98/spark,sureshthalamati/spark,1haodian/spark,metamx/spark,nlalevee/spark,jlopezmalla/spark,MLnick/spark,UndeadBaneGitHub/spark,ibm-research-ireland/sparkoscope,aokolnychyi/spark,sahilTakiar/spark,pgandhi999/spark,byakuinss/spark,apache-spark-on-k8s/spark,holdenk/spark,liutang123/spark,ConeyLiu/spark,hvanhovell/spark,loneknightpy/spark,gengliangwang/spark,xflin/spark,Aegeaner/spark,Panos-Bletsos/spark-cost-model-optimizer,JoshRosen/spark,holdenk/spark,maropu/spark,yanboliang/spark,highfei2011/spark,saturday-shi/spark,mzl9039/spark,andrewor14/iolap,ericvandenbergfb/spark,techaddict/spark,ericvandenbergfb/spark,maropu/spark,milliman/spark,tejasapatil/spark,apache/spark,guoxiaolongzte/spark,setjet/spark,facaiy/spark,guoxiaolongzte/spark,alunarbeach/spark,wzhfy/spark,wzhfy/spark,nilsgrabbert/spark,big-pegasus/spark,bdrillard/spark,liutang123/spark,shuangshuangwang/spark,mdespriee/spark,dhruve/spark,ahnqirage/spark,skonto/spark,liutang123/spark,HyukjinKwon/spark,huang1900/spark,darionyaphet/spark,jiangxb1987/spark,loneknightpy/spark,vinodkc/spark,kissmetrics/spark,cloud-fan/spark,joseph-torres/spark,lvdongr/spark,JerryLead/spark,aray/spark,Aegeaner/spark,lxsmnv/spark,actuaryzhang/spark,publicRoman/spark,zuotingbing/spark,big-pegasus/spark,bdrillard/spark,techaddict/spark,adrian-ionescu/apache-spark,hhbyyh/spark,rednaxelafx/apache-spark,big-pegasus/spark,ioana-delaney/spark,jlopezmalla/spark,guoxiaolongzte/spark,matthewfranglen/spark,publicRoman/spark,janewangfb/spark,jlopezmalla/spark,jiangxb1987/spark,wangmiao1981/spark,wzhfy/spark,sahilTakiar/spark,kimoonkim/spark,debugger87/spark,UndeadBaneGitHub/spark,narahari92/spark,kimoonkim/spark,huang1900/spark,LantaoJin/spark,dbtsai/spark,SHASHANKB/spark,nlalevee/spark,goldmedal/spark,SHASHANKB/spark,cin/spark,ron8hu/spark,gengliangwang/spark,narahari92/spark,tejasapatil/spark,tengpeng/spark,WeichenXu123/spark,lxsmnv/spark,caneGuy/spark,LantaoJin/spark,zhouyejoe/spark,lvdongr/spark,milliman/spark,shuangshuangwang/spark,brad-kaiser/spark,aokolnychyi/spark,BryanCutler/spark,sureshthalamati/spark,apache-spark-on-k8s/spark,srowen/spark,tejasapatil/spark,ueshin/apache-spark,nchammas/spark,mahak/spark,mdespriee/spark,rednaxelafx/apache-spark,mdespriee/spark,janewangfb/spark,chuckchen/spark,esi-mineset/spark,taroplus/spark,SHASHANKB/spark,ddna1021/spark,chuckchen/spark,gioenn/xSpark,mahak/spark,apache/spark,MLnick/spark,wzhfy/spark,tengpeng/spark,witgo/spark,goldmedal/spark,jrshust/spark,tejasapatil/spark,wangmiao1981/spark,bravo-zhang/spark,JerryLead/spark,dhruve/spark,kissmetrics/spark,wzhfy/spark,kissmetrics/spark,jrshust/spark,pronix/spark,yanboliang/spark,esi-mineset/spark,sachintyagi22/spark,zuotingbing/spark,actuaryzhang/spark,saltstar/spark,loneknightpy/spark,shuangshuangwang/spark,cloud-fan/spark,guoxiaolongzte/spark,MLnick/spark,ptkool/spark,byakuinss/spark,icexelloss/spark,rikima/spark,wangmiao1981/spark,brad-kaiser/spark,taroplus/spark,JerryLead/spark,akopich/spark,minixalpha/spark,alunarbeach/spark,aray/spark,xuanyuanking/spark,yanboliang/spark,aray/spark,haowu80s/spark,HyukjinKwon/spark,brad-kaiser/spark,zuotingbing/spark,adrian-ionescu/apache-spark,kimoonkim/spark,joseph-torres/spark,holdenk/spark,highfei2011/spark,facaiy/spark,rezasafi/spark,maropu/spark,goldmedal/spark,hhbyyh/spark,liyichao/spark,brad-kaiser/spark,byakuinss/spark,szhem/spark,nlalevee/spark,aray/spark,markhamstra/spark,holdenk/spark,pgandhi999/spark,jkbradley/spark,patrick-nicholson/spark,andrewor14/spark,big-pegasus/spark,UndeadBaneGitHub/spark,someorz/spark,WindCanDie/spark,akopich/spark,andrewor14/iolap,rekhajoshm/spark,metamx/spark,caneGuy/spark,aosagie/spark,nlalevee/spark,loneknightpy/spark,publicRoman/spark,WeichenXu123/spark,LantaoJin/spark,panfengfeng/spark,lxsmnv/spark,someorz/spark,milliman/spark,JoshRosen/spark,techaddict/spark,someorz/spark,shubhamchopra/spark,ericvandenbergfb/spark,markhamstra/spark,minixalpha/spark,ConeyLiu/spark,tengpeng/spark,ueshin/apache-spark,liyichao/spark,wangyum/spark,zzcclp/spark,mahak/spark,kevinyu98/spark,rekhajoshm/spark,xuanyuanking/spark,MLnick/spark,panfengfeng/spark,zzcclp/spark,zero323/spark,zero323/spark,SnappyDataInc/spark,markhamstra/spark,poffuomo/spark,ibm-research-ireland/sparkoscope,markhamstra/spark,wangmiao1981/spark,WeichenXu123/spark,wangyum/spark,poffuomo/spark,apache-spark-on-k8s/spark,ueshin/apache-spark,huang1900/spark,huang1900/spark,WeichenXu123/spark,ajaysaini725/spark,chuckchen/spark,caneGuy/spark,szhem/spark,ajaysaini725/spark,pronix/spark,jianran/spark,rikima/spark,skonto/spark,liutang123/spark,apache/spark,kiszk/spark,byakuinss/spark,nilsgrabbert/spark,ddna1021/spark,pgandhi999/spark,WindCanDie/spark,kissmetrics/spark,sureshthalamati/spark,ueshin/apache-spark,gengliangwang/spark,wangyum/spark,narahari92/spark,kiszk/spark,JoshRosen/spark,UndeadBaneGitHub/spark,stanzhai/spark,caneGuy/spark,sahilTakiar/spark,minixalpha/spark,janewangfb/spark,aokolnychyi/spark,szhem/spark,debugger87/spark,dhruve/spark,rekhajoshm/spark,BryanCutler/spark,saltstar/spark,gioenn/xSpark,adrian-ionescu/apache-spark,szhem/spark,taroplus/spark,andrewor14/iolap,taroplus/spark,michalsenkyr/spark,wangyum/spark,wangyum/spark,setjet/spark,lxsmnv/spark,xflin/spark,zero323/spark,JerryLead/spark,aokolnychyi/spark,facaiy/spark,witgo/spark,ioana-delaney/spark,UndeadBaneGitHub/spark,techaddict/spark,caneGuy/spark,mdespriee/spark,rikima/spark,liutang123/spark,ron8hu/spark,tejasapatil/spark,ddna1021/spark,caneGuy/spark,setjet/spark,metamx/spark,shuangshuangwang/spark,darionyaphet/spark,Aegeaner/spark,wangyum/spark,zuotingbing/spark,stanzhai/spark,milliman/spark,nlalevee/spark,stanzhai/spark,BryanCutler/spark,liyichao/spark,ibm-research-ireland/sparkoscope,bravo-zhang/spark,SHASHANKB/spark,saltstar/spark,adrian-ionescu/apache-spark,rednaxelafx/apache-spark,lvdongr/spark,jkbradley/spark,huang1900/spark,goldmedal/spark,guoxiaolongzte/spark,spark-test/spark,kimoonkim/spark,andrewor14/iolap,tengpeng/spark,actuaryzhang/spark,michalsenkyr/spark,LantaoJin/spark,jlopezmalla/spark,ptkool/spark,szhem/spark,patrick-nicholson/spark,mahak/spark,witgo/spark,chuckchen/spark,ueshin/apache-spark,spark-test/spark,nlalevee/spark,ajaysaini725/spark,janewangfb/spark,narahari92/spark,mzl9039/spark,rezasafi/spark,gioenn/xSpark,joseph-torres/spark,poffuomo/spark,milliman/spark,map222/spark,shaneknapp/spark,kevinyu98/spark,metamx/spark,ahnqirage/spark,ron8hu/spark,setjet/spark,caneGuy/spark,darionyaphet/spark,ConeyLiu/spark,rikima/spark,cin/spark,liyichao/spark,1haodian/spark,zzcclp/spark,metamx/spark,Panos-Bletsos/spark-cost-model-optimizer,1haodian/spark,xuanyuanking/spark,kiszk/spark,shubhamchopra/spark,facaiy/spark,mahak/spark,icexelloss/spark,mike0sv/spark,rikima/spark,actuaryzhang/spark,zzcclp/spark,saturday-shi/spark,szhem/spark,xflin/spark,BryanCutler/spark,kissmetrics/spark,ueshin/apache-spark,shaneknapp/spark,aosagie/spark,holdenk/spark,JerryLead/spark,aokolnychyi/spark,HyukjinKwon/spark,ddna1021/spark,yanboliang/spark,nilsgrabbert/spark,pronix/spark,eyalfa/spark,bravo-zhang/spark,brad-kaiser/spark,pgandhi999/spark,highfei2011/spark,janewangfb/spark,goldmedal/spark,andrewor14/spark,someorz/spark,someorz/spark,apache/spark,mike0sv/spark,pgandhi999/spark,aosagie/spark,andrewor14/spark,dhruve/spark,lxsmnv/spark,dongjoon-hyun/spark,map222/spark,actuaryzhang/spark,shaneknapp/spark,sachintyagi22/spark,bOOm-X/spark,brad-kaiser/spark,cloud-fan/spark,apache-spark-on-k8s/spark,mzl9039/spark,ConeyLiu/spark,holdenk/spark,actuaryzhang/spark,patrick-nicholson/spark,dbtsai/spark,sachintyagi22/spark,JoshRosen/spark,jlopezmalla/spark,apache/spark,panfengfeng/spark,michalsenkyr/spark,zzcclp/spark,JoshRosen/spark,ioana-delaney/spark,eyalfa/spark,dongjoon-hyun/spark,ptkool/spark,bOOm-X/spark,andrewor14/spark,witgo/spark,BryanCutler/spark,mike0sv/spark,shaneknapp/spark,lvdongr/spark,ajaysaini725/spark,debugger87/spark,michalsenkyr/spark,highfei2011/spark,aokolnychyi/spark,pronix/spark,kevinyu98/spark,tejasapatil/spark,huang1900/spark,yanboliang/spark,techaddict/spark,zuotingbing/spark,gioenn/xSpark,nchammas/spark,kiszk/spark,xuanyuanking/spark,dbtsai/spark,ahnqirage/spark,loneknightpy/spark,stanzhai/spark,ahnqirage/spark,icexelloss/spark,patrick-nicholson/spark,haowu80s/spark,wangmiao1981/spark,hvanhovell/spark,apache-spark-on-k8s/spark,HyukjinKwon/spark,zuotingbing/spark,big-pegasus/spark,ddna1021/spark,map222/spark,byakuinss/spark,patrick-nicholson/spark,joseph-torres/spark,jkbradley/spark,jianran/spark,highfei2011/spark,vinodkc/spark,xuanyuanking/spark,ptkool/spark,hhbyyh/spark,saltstar/spark,guoxiaolongzte/spark,setjet/spark,nlalevee/spark,huang1900/spark,haowu80s/spark,loneknightpy/spark,stanzhai/spark,sachintyagi22/spark,jianran/spark,metamx/spark,dbtsai/spark,ibm-research-ireland/sparkoscope,esi-mineset/spark,maropu/spark,bravo-zhang/spark,aosagie/spark,dongjoon-hyun/spark,ron8hu/spark,zero323/spark,srowen/spark,ericvandenbergfb/spark,rednaxelafx/apache-spark,eyalfa/spark,patrick-nicholson/spark,hvanhovell/spark,jianran/spark,skonto/spark,byakuinss/spark,sureshthalamati/spark,sachintyagi22/spark,janewangfb/spark,dhruve/spark,esi-mineset/spark,WindCanDie/spark,map222/spark,sahilTakiar/spark,taroplus/spark,ericvandenbergfb/spark,liyichao/spark,spark-test/spark,alunarbeach/spark,ahnqirage/spark,hhbyyh/spark,rednaxelafx/apache-spark,dongjoon-hyun/spark,kiszk/spark,Aegeaner/spark,ron8hu/spark,narahari92/spark,andrewor14/iolap,vinodkc/spark,poffuomo/spark,aray/spark,darionyaphet/spark,jkbradley/spark,shubhamchopra/spark,jrshust/spark,WindCanDie/spark,BryanCutler/spark,gengliangwang/spark,mdespriee/spark,zero323/spark,UndeadBaneGitHub/spark,jiangxb1987/spark,aokolnychyi/spark,spark-test/spark,panfengfeng/spark,bOOm-X/spark,wangmiao1981/spark,shuangshuangwang/spark,lvdongr/spark,shuangshuangwang/spark,JerryLead/spark,tengpeng/spark,jlopezmalla/spark,zhouyejoe/spark,mdespriee/spark,mdespriee/spark,techaddict/spark,HyukjinKwon/spark,debugger87/spark,SHASHANKB/spark,bOOm-X/spark,joseph-torres/spark,sahilTakiar/spark,ConeyLiu/spark,1haodian/spark,jkbradley/spark,jiangxb1987/spark,lvdongr/spark,kimoonkim/spark,MLnick/spark,shubhamchopra/spark,kevinyu98/spark,mike0sv/spark,kevinyu98/spark,witgo/spark,hvanhovell/spark,LantaoJin/spark,pgandhi999/spark,bdrillard/spark,rekhajoshm/spark,cin/spark,mike0sv/spark,big-pegasus/spark,Aegeaner/spark,jianran/spark,joseph-torres/spark,xflin/spark,bravo-zhang/spark,zzcclp/spark,jiangxb1987/spark,SnappyDataInc/spark,nchammas/spark,cin/spark,xuanyuanking/spark,HyukjinKwon/spark,nilsgrabbert/spark,cin/spark,map222/spark,jrshust/spark,icexelloss/spark,ioana-delaney/spark,adrian-ionescu/apache-spark,bravo-zhang/spark,janewangfb/spark,michalsenkyr/spark,Panos-Bletsos/spark-cost-model-optimizer,cin/spark,eyalfa/spark,markhamstra/spark,kissmetrics/spark,ptkool/spark,spark-test/spark,dbtsai/spark,liutang123/spark,publicRoman/spark,cloud-fan/spark,loneknightpy/spark,stanzhai/spark,saltstar/spark,gengliangwang/spark,Panos-Bletsos/spark-cost-model-optimizer,apache/spark,dongjoon-hyun/spark,rezasafi/spark,sahilTakiar/spark,bOOm-X/spark,WindCanDie/spark,srowen/spark,mike0sv/spark,srowen/spark,andrewor14/spark,ddna1021/spark,vinodkc/spark,chuckchen/spark,saturday-shi/spark,ddna1021/spark,Panos-Bletsos/spark-cost-model-optimizer,someorz/spark,kiszk/spark,icexelloss/spark,zhouyejoe/spark,taroplus/spark,bdrillard/spark,jrshust/spark,1haodian/spark,ahnqirage/spark,debugger87/spark,maropu/spark,eyalfa/spark,witgo/spark,haowu80s/spark,ibm-research-ireland/sparkoscope,pronix/spark,akopich/spark,rikima/spark,stanzhai/spark,aosagie/spark,minixalpha/spark,lxsmnv/spark,srowen/spark,mzl9039/spark,liutang123/spark,zhouyejoe/spark,chuckchen/spark,rekhajoshm/spark,panfengfeng/spark,rekhajoshm/spark,rezasafi/spark,alunarbeach/spark