[SPARK-1468] Modify the partition function used by partitionBy.
[SPARK-1468] Modify the partition function used by partitionBy.

Make partitionBy use a tweaked version of hash as its default partition function
since the python hash function does not consistently assign the same value
to None across python processes.

Associated JIRA at https://issues.apache.org/jira/browse/SPARK-1468

Author: Erik Selin <erik.selin@jadedpixel.com>

Closes #371 from tyro89/consistent_hashing and squashes the following commits:

201c301 [Erik Selin] Make partitionBy use a tweaked version of hash as its default partition function since the python hash function does not consistently assign the same value to None across python processes.

loneknightpy/spark,mike0sv/spark,dhruve/spark,cin/spark,sureshthalamati/spark,aokolnychyi/spark,zero323/spark,hvanhovell/spark,jiangxb1987/spark,apache/spark,saltstar/spark,WeichenXu123/spark,saturday-shi/spark,facaiy/spark,rikima/spark,maropu/spark,xuanyuanking/spark,shuangshuangwang/spark,dhruve/spark,techaddict/spark,pgandhi999/spark,wangmiao1981/spark,pronix/spark,markhamstra/spark,liyichao/spark,sureshthalamati/spark,gioenn/xSpark,szhem/spark,aray/spark,MLnick/spark,shubhamchopra/spark,publicRoman/spark,sureshthalamati/spark,chuckchen/spark,ron8hu/spark,goldmedal/spark,shubhamchopra/spark,vinodkc/spark,lxsmnv/spark,wangyum/spark,sachintyagi22/spark,kimoonkim/spark,1haodian/spark,goldmedal/spark,rednaxelafx/apache-spark,BryanCutler/spark,stanzhai/spark,ibm-research-ireland/sparkoscope,dongjoon-hyun/spark,xuanyuanking/spark,wzhfy/spark,akopich/spark,liutang123/spark,guoxiaolongzte/spark,andrewor14/iolap,minixalpha/spark,apache/spark,setjet/spark,andrewor14/spark,adrian-ionescu/apache-spark,holdenk/spark,maropu/spark,BryanCutler/spark,witgo/spark,zhouyejoe/spark,loneknightpy/spark,andrewor14/spark,mzl9039/spark,michalsenkyr/spark,shaneknapp/spark,pronix/spark,skonto/spark,xflin/spark,shuangshuangwang/spark,JoshRosen/spark,pronix/spark,szhem/spark,lvdongr/spark,nilsgrabbert/spark,lvdongr/spark,cin/spark,Panos-Bletsos/spark-cost-model-optimizer,mike0sv/spark,yanboliang/spark,JerryLead/spark,bravo-zhang/spark,guoxiaolongzte/spark,zhouyejoe/spark,kevinyu98/spark,ptkool/spark,dbtsai/spark,panfengfeng/spark,sahilTakiar/spark,alunarbeach/spark,holdenk/spark,xuanyuanking/spark,wzhfy/spark,pronix/spark,Aegeaner/spark,shubhamchopra/spark,bOOm-X/spark,bravo-zhang/spark,SnappyDataInc/spark,sachintyagi22/spark,big-pegasus/spark,spark-test/spark,darionyaphet/spark,publicRoman/spark,rekhajoshm/spark,joseph-torres/spark,shubhamchopra/spark,maropu/spark,saturday-shi/spark,facaiy/spark,zuotingbing/spark,zzcclp/spark,metamx/spark,pgandhi999/spark,zzcclp/spark,jkbradley/spark,cloud-fan/spark,witgo/spark,facaiy/spark,maropu/spark,patrick-nicholson/spark,jianran/spark,JoshRosen/spark,jianran/spark,hvanhovell/spark,szhem/spark,bOOm-X/spark,pgandhi999/spark,jkbradley/spark,jianran/spark,BryanCutler/spark,wzhfy/spark,LantaoJin/spark,vinodkc/spark,apache/spark,shaneknapp/spark,actuaryzhang/spark,rezasafi/spark,tejasapatil/spark,poffuomo/spark,ddna1021/spark,publicRoman/spark,joseph-torres/spark,zero323/spark,ioana-delaney/spark,bdrillard/spark,setjet/spark,facaiy/spark,ueshin/apache-spark,narahari92/spark,wangyum/spark,panfengfeng/spark,icexelloss/spark,spark-test/spark,patrick-nicholson/spark,adrian-ionescu/apache-spark,bravo-zhang/spark,actuaryzhang/spark,yanboliang/spark,ConeyLiu/spark,lxsmnv/spark,ioana-delaney/spark,rednaxelafx/apache-spark,mahak/spark,vinodkc/spark,jrshust/spark,apache-spark-on-k8s/spark,brad-kaiser/spark,kissmetrics/spark,huang1900/spark,metamx/spark,saltstar/spark,jkbradley/spark,sureshthalamati/spark,jrshust/spark,akopich/spark,UndeadBaneGitHub/spark,tengpeng/spark,byakuinss/spark,vinodkc/spark,jrshust/spark,zero323/spark,shaneknapp/spark,chuckchen/spark,bdrillard/spark,lxsmnv/spark,dongjoon-hyun/spark,stanzhai/spark,MLnick/spark,SnappyDataInc/spark,rikima/spark,janewangfb/spark,caneGuy/spark,sureshthalamati/spark,techaddict/spark,witgo/spark,andrewor14/iolap,rezasafi/spark,caneGuy/spark,ron8hu/spark,zero323/spark,witgo/spark,JoshRosen/spark,nlalevee/spark,SHASHANKB/spark,map222/spark,taroplus/spark,SnappyDataInc/spark,WindCanDie/spark,nchammas/spark,milliman/spark,patrick-nicholson/spark,kimoonkim/spark,gengliangwang/spark,ibm-research-ireland/sparkoscope,patrick-nicholson/spark,adrian-ionescu/apache-spark,cin/spark,zuotingbing/spark,rednaxelafx/apache-spark,kiszk/spark,MLnick/spark,ioana-delaney/spark,xuanyuanking/spark,shuangshuangwang/spark,loneknightpy/spark,goldmedal/spark,goldmedal/spark,guoxiaolongzte/spark,szhem/spark,witgo/spark,narahari92/spark,dhruve/spark,bdrillard/spark,UndeadBaneGitHub/spark,mike0sv/spark,wangyum/spark,jrshust/spark,cin/spark,icexelloss/spark,big-pegasus/spark,alunarbeach/spark,WeichenXu123/spark,ptkool/spark,akopich/spark,xuanyuanking/spark,nchammas/spark,kevinyu98/spark,jkbradley/spark,milliman/spark,gioenn/xSpark,brad-kaiser/spark,WindCanDie/spark,MLnick/spark,aray/spark,alunarbeach/spark,icexelloss/spark,andrewor14/spark,xflin/spark,shaneknapp/spark,janewangfb/spark,kiszk/spark,HyukjinKwon/spark,someorz/spark,ptkool/spark,gioenn/xSpark,1haodian/spark,srowen/spark,zzcclp/spark,shaneknapp/spark,esi-mineset/spark,rikima/spark,JoshRosen/spark,yanboliang/spark,LantaoJin/spark,haowu80s/spark,sahilTakiar/spark,narahari92/spark,Panos-Bletsos/spark-cost-model-optimizer,mzl9039/spark,darionyaphet/spark,kiszk/spark,jiangxb1987/spark,techaddict/spark,rikima/spark,lvdongr/spark,zuotingbing/spark,wzhfy/spark,alunarbeach/spark,akopich/spark,darionyaphet/spark,mike0sv/spark,zero323/spark,wangmiao1981/spark,joseph-torres/spark,dongjoon-hyun/spark,skonto/spark,aosagie/spark,milliman/spark,icexelloss/spark,bOOm-X/spark,hvanhovell/spark,1haodian/spark,vinodkc/spark,pronix/spark,rezasafi/spark,bravo-zhang/spark,apache-spark-on-k8s/spark,ConeyLiu/spark,ioana-delaney/spark,alunarbeach/spark,brad-kaiser/spark,nchammas/spark,mdespriee/spark,szhem/spark,wangyum/spark,apache-spark-on-k8s/spark,map222/spark,nlalevee/spark,joseph-torres/spark,publicRoman/spark,spark-test/spark,sahilTakiar/spark,guoxiaolongzte/spark,minixalpha/spark,tejasapatil/spark,ericvandenbergfb/spark,JerryLead/spark,cloud-fan/spark,hvanhovell/spark,jlopezmalla/spark,icexelloss/spark,mzl9039/spark,srowen/spark,mike0sv/spark,haowu80s/spark,spark-test/spark,alunarbeach/spark,darionyaphet/spark,ueshin/apache-spark,holdenk/spark,minixalpha/spark,markhamstra/spark,UndeadBaneGitHub/spark,WindCanDie/spark,ioana-delaney/spark,hhbyyh/spark,hvanhovell/spark,kimoonkim/spark,sureshthalamati/spark,saltstar/spark,ahnqirage/spark,zhouyejoe/spark,byakuinss/spark,kevinyu98/spark,zuotingbing/spark,taroplus/spark,WeichenXu123/spark,aray/spark,andrewor14/iolap,rikima/spark,facaiy/spark,srowen/spark,hhbyyh/spark,Aegeaner/spark,gengliangwang/spark,lxsmnv/spark,mike0sv/spark,mdespriee/spark,hhbyyh/spark,liyichao/spark,big-pegasus/spark,mzl9039/spark,skonto/spark,nchammas/spark,ron8hu/spark,cin/spark,tengpeng/spark,poffuomo/spark,BryanCutler/spark,chuckchen/spark,cin/spark,loneknightpy/spark,akopich/spark,xflin/spark,ConeyLiu/spark,liyichao/spark,kimoonkim/spark,ddna1021/spark,skonto/spark,pronix/spark,tengpeng/spark,caneGuy/spark,ptkool/spark,andrewor14/iolap,vinodkc/spark,goldmedal/spark,andrewor14/spark,bdrillard/spark,JoshRosen/spark,BryanCutler/spark,jrshust/spark,shubhamchopra/spark,adrian-ionescu/apache-spark,rekhajoshm/spark,janewangfb/spark,chuckchen/spark,taroplus/spark,shubhamchopra/spark,chuckchen/spark,aosagie/spark,ptkool/spark,haowu80s/spark,huang1900/spark,bOOm-X/spark,ron8hu/spark,eyalfa/spark,darionyaphet/spark,LantaoJin/spark,sachintyagi22/spark,xflin/spark,nilsgrabbert/spark,andrewor14/spark,spark-test/spark,esi-mineset/spark,Aegeaner/spark,WindCanDie/spark,SHASHANKB/spark,kiszk/spark,setjet/spark,HyukjinKwon/spark,Panos-Bletsos/spark-cost-model-optimizer,loneknightpy/spark,dbtsai/spark,nlalevee/spark,ConeyLiu/spark,ajaysaini725/spark,kiszk/spark,jianran/spark,skonto/spark,LantaoJin/spark,milliman/spark,caneGuy/spark,big-pegasus/spark,techaddict/spark,zuotingbing/spark,ahnqirage/spark,mdespriee/spark,ddna1021/spark,wangmiao1981/spark,markhamstra/spark,andrewor14/spark,michalsenkyr/spark,huang1900/spark,Aegeaner/spark,SHASHANKB/spark,srowen/spark,michalsenkyr/spark,panfengfeng/spark,ajaysaini725/spark,srowen/spark,jianran/spark,publicRoman/spark,bdrillard/spark,hhbyyh/spark,highfei2011/spark,michalsenkyr/spark,ahnqirage/spark,zuotingbing/spark,eyalfa/spark,debugger87/spark,map222/spark,mdespriee/spark,wzhfy/spark,shuangshuangwang/spark,Aegeaner/spark,janewangfb/spark,wangyum/spark,esi-mineset/spark,nlalevee/spark,nilsgrabbert/spark,aray/spark,techaddict/spark,caneGuy/spark,lxsmnv/spark,mzl9039/spark,markhamstra/spark,liutang123/spark,adrian-ionescu/apache-spark,ConeyLiu/spark,wzhfy/spark,caneGuy/spark,rezasafi/spark,debugger87/spark,byakuinss/spark,jiangxb1987/spark,WindCanDie/spark,byakuinss/spark,goldmedal/spark,SnappyDataInc/spark,zhouyejoe/spark,kevinyu98/spark,andrewor14/iolap,spark-test/spark,mahak/spark,rednaxelafx/apache-spark,MLnick/spark,aray/spark,cloud-fan/spark,wangyum/spark,HyukjinKwon/spark,matthewfranglen/spark,kissmetrics/spark,aosagie/spark,SHASHANKB/spark,lvdongr/spark,yanboliang/spark,zzcclp/spark,guoxiaolongzte/spark,hhbyyh/spark,hhbyyh/spark,LantaoJin/spark,lxsmnv/spark,nlalevee/spark,Aegeaner/spark,poffuomo/spark,tejasapatil/spark,pgandhi999/spark,byakuinss/spark,aosagie/spark,SHASHANKB/spark,sureshthalamati/spark,loneknightpy/spark,wangmiao1981/spark,sachintyagi22/spark,HyukjinKwon/spark,holdenk/spark,sahilTakiar/spark,kimoonkim/spark,nilsgrabbert/spark,shuangshuangwang/spark,loneknightpy/spark,ahnqirage/spark,ajaysaini725/spark,big-pegasus/spark,adrian-ionescu/apache-spark,jiangxb1987/spark,ueshin/apache-spark,jrshust/spark,LantaoJin/spark,WeichenXu123/spark,actuaryzhang/spark,apache/spark,facaiy/spark,bdrillard/spark,big-pegasus/spark,nchammas/spark,setjet/spark,mzl9039/spark,gioenn/xSpark,gengliangwang/spark,lvdongr/spark,dongjoon-hyun/spark,narahari92/spark,nilsgrabbert/spark,taroplus/spark,huang1900/spark,skonto/spark,JerryLead/spark,ptkool/spark,brad-kaiser/spark,apache-spark-on-k8s/spark,UndeadBaneGitHub/spark,zhouyejoe/spark,dbtsai/spark,ajaysaini725/spark,aray/spark,debugger87/spark,map222/spark,rikima/spark,apache-spark-on-k8s/spark,xuanyuanking/spark,jiangxb1987/spark,apache/spark,HyukjinKwon/spark,stanzhai/spark,WindCanDie/spark,gioenn/xSpark,saltstar/spark,kissmetrics/spark,dongjoon-hyun/spark,kissmetrics/spark,metamx/spark,JoshRosen/spark,metamx/spark,goldmedal/spark,rekhajoshm/spark,big-pegasus/spark,Panos-Bletsos/spark-cost-model-optimizer,caneGuy/spark,tengpeng/spark,wangmiao1981/spark,kevinyu98/spark,highfei2011/spark,sahilTakiar/spark,chuckchen/spark,ptkool/spark,bravo-zhang/spark,jkbradley/spark,SnappyDataInc/spark,aokolnychyi/spark,taroplus/spark,publicRoman/spark,WeichenXu123/spark,ahnqirage/spark,ioana-delaney/spark,map222/spark,milliman/spark,aosagie/spark,zero323/spark,lvdongr/spark,ueshin/apache-spark,zhouyejoe/spark,rednaxelafx/apache-spark,narahari92/spark,kimoonkim/spark,michalsenkyr/spark,mzl9039/spark,jlopezmalla/spark,rednaxelafx/apache-spark,nlalevee/spark,huang1900/spark,jianran/spark,SHASHANKB/spark,aray/spark,adrian-ionescu/apache-spark,UndeadBaneGitHub/spark,haowu80s/spark,dongjoon-hyun/spark,dbtsai/spark,jrshust/spark,kiszk/spark,someorz/spark,darionyaphet/spark,map222/spark,cin/spark,ConeyLiu/spark,liutang123/spark,shuangshuangwang/spark,UndeadBaneGitHub/spark,saturday-shi/spark,stanzhai/spark,liyichao/spark,ueshin/apache-spark,hvanhovell/spark,saltstar/spark,apache-spark-on-k8s/spark,ahnqirage/spark,lvdongr/spark,gioenn/xSpark,stanzhai/spark,apache-spark-on-k8s/spark,JerryLead/spark,ueshin/apache-spark,dhruve/spark,rikima/spark,maropu/spark,akopich/spark,kissmetrics/spark,bravo-zhang/spark,esi-mineset/spark,gengliangwang/spark,jlopezmalla/spark,BryanCutler/spark,highfei2011/spark,mahak/spark,panfengfeng/spark,aosagie/spark,eyalfa/spark,kissmetrics/spark,skonto/spark,liutang123/spark,holdenk/spark,witgo/spark,markhamstra/spark,vinodkc/spark,map222/spark,Panos-Bletsos/spark-cost-model-optimizer,techaddict/spark,metamx/spark,jlopezmalla/spark,pgandhi999/spark,someorz/spark,facaiy/spark,dbtsai/spark,aosagie/spark,eyalfa/spark,JerryLead/spark,mahak/spark,bOOm-X/spark,ericvandenbergfb/spark,poffuomo/spark,1haodian/spark,someorz/spark,minixalpha/spark,WeichenXu123/spark,guoxiaolongzte/spark,patrick-nicholson/spark,srowen/spark,saturday-shi/spark,ddna1021/spark,ron8hu/spark,bravo-zhang/spark,1haodian/spark,WindCanDie/spark,metamx/spark,byakuinss/spark,zzcclp/spark,MLnick/spark,gengliangwang/spark,ericvandenbergfb/spark,aokolnychyi/spark,hvanhovell/spark,spark-test/spark,jiangxb1987/spark,byakuinss/spark,taroplus/spark,jlopezmalla/spark,michalsenkyr/spark,setjet/spark,eyalfa/spark,bdrillard/spark,holdenk/spark,pgandhi999/spark,bOOm-X/spark,highfei2011/spark,nchammas/spark,icexelloss/spark,minixalpha/spark,mdespriee/spark,someorz/spark,ajaysaini725/spark,SnappyDataInc/spark,rezasafi/spark,cloud-fan/spark,ibm-research-ireland/sparkoscope,shuangshuangwang/spark,kissmetrics/spark,actuaryzhang/spark,lxsmnv/spark,jiangxb1987/spark,yanboliang/spark,srowen/spark,rezasafi/spark,rekhajoshm/spark,pgandhi999/spark,markhamstra/spark,taroplus/spark,brad-kaiser/spark,shubhamchopra/spark,1haodian/spark,hhbyyh/spark,xflin/spark,saturday-shi/spark,maropu/spark,techaddict/spark,sahilTakiar/spark,haowu80s/spark,JerryLead/spark,actuaryzhang/spark,jlopezmalla/spark,JerryLead/spark,patrick-nicholson/spark,milliman/spark,huang1900/spark,icexelloss/spark,kevinyu98/spark,janewangfb/spark,maropu/spark,saturday-shi/spark,andrewor14/spark,guoxiaolongzte/spark,minixalpha/spark,dbtsai/spark,eyalfa/spark,MLnick/spark,cloud-fan/spark,gioenn/xSpark,nilsgrabbert/spark,metamx/spark,ddna1021/spark,publicRoman/spark,rednaxelafx/apache-spark,gengliangwang/spark,joseph-torres/spark,panfengfeng/spark,ibm-research-ireland/sparkoscope,Aegeaner/spark,liyichao/spark,ibm-research-ireland/sparkoscope,brad-kaiser/spark,zuotingbing/spark,witgo/spark,zhouyejoe/spark,mdespriee/spark,janewangfb/spark,narahari92/spark,someorz/spark,bOOm-X/spark,LantaoJin/spark,apache/spark,panfengfeng/spark,Panos-Bletsos/spark-cost-model-optimizer,zero323/spark,sachintyagi22/spark,aokolnychyi/spark,haowu80s/spark,nlalevee/spark,ericvandenbergfb/spark,shaneknapp/spark,ericvandenbergfb/spark,panfengfeng/spark,mahak/spark,chuckchen/spark,jianran/spark,xuanyuanking/spark,kevinyu98/spark,nilsgrabbert/spark,huang1900/spark,stanzhai/spark,holdenk/spark,jkbradley/spark,poffuomo/spark,mahak/spark,andrewor14/iolap,ioana-delaney/spark,liutang123/spark,cloud-fan/spark,ron8hu/spark,debugger87/spark,jlopezmalla/spark,actuaryzhang/spark,highfei2011/spark,ericvandenbergfb/spark,HyukjinKwon/spark,aokolnychyi/spark,sahilTakiar/spark,rekhajoshm/spark,shaneknapp/spark,apache/spark,WeichenXu123/spark,markhamstra/spark,saturday-shi/spark,esi-mineset/spark,HyukjinKwon/spark,xflin/spark,stanzhai/spark,rezasafi/spark,liyichao/spark,esi-mineset/spark,tejasapatil/spark,liutang123/spark,ddna1021/spark,darionyaphet/spark,joseph-torres/spark,milliman/spark,ueshin/apache-spark,saltstar/spark,someorz/spark,nchammas/spark,szhem/spark,yanboliang/spark,wangmiao1981/spark,tejasapatil/spark,UndeadBaneGitHub/spark,ibm-research-ireland/sparkoscope,dbtsai/spark,andrewor14/iolap,wzhfy/spark,gengliangwang/spark,ericvandenbergfb/spark,zzcclp/spark,jkbradley/spark,esi-mineset/spark,tejasapatil/spark,dhruve/spark,highfei2011/spark,joseph-torres/spark,poffuomo/spark,brad-kaiser/spark,liutang123/spark,mahak/spark,michalsenkyr/spark,alunarbeach/spark,JoshRosen/spark,liyichao/spark,highfei2011/spark,kimoonkim/spark,dongjoon-hyun/spark,aokolnychyi/spark,setjet/spark,rekhajoshm/spark,yanboliang/spark,ajaysaini725/spark,rekhajoshm/spark,xflin/spark,ddna1021/spark,Panos-Bletsos/spark-cost-model-optimizer,sachintyagi22/spark,zzcclp/spark,dhruve/spark,mike0sv/spark,ron8hu/spark,wangmiao1981/spark,debugger87/spark,saltstar/spark,narahari92/spark,szhem/spark,ibm-research-ireland/sparkoscope,SHASHANKB/spark,dhruve/spark,debugger87/spark,cloud-fan/spark,1haodian/spark,BryanCutler/spark,ahnqirage/spark,ConeyLiu/spark,kiszk/spark,poffuomo/spark,tengpeng/spark,akopich/spark,minixalpha/spark,actuaryzhang/spark,debugger87/spark,sachintyagi22/spark,tejasapatil/spark,eyalfa/spark,patrick-nicholson/spark,tengpeng/spark,janewangfb/spark,wangyum/spark,ajaysaini725/spark,tengpeng/spark,mdespriee/spark,SnappyDataInc/spark,setjet/spark,aokolnychyi/spark