SPARK-2203: PySpark defaults to use same num reduce partitions as map side
SPARK-2203: PySpark defaults to use same num reduce partitions as map side

For shuffle-based operators, such as rdd.groupBy() or rdd.sortByKey(), PySpark will always assume that the default parallelism to use for the reduce side is ctx.defaultParallelism, which is a constant typically determined by the number of cores in cluster.

In contrast, Spark's Partitioner#defaultPartitioner will use the same number of reduce partitions as map partitions unless the defaultParallelism config is explicitly set. This tends to be a better default in order to avoid OOMs, and should also be the behavior of PySpark.

JIRA: https://issues.apache.org/jira/browse/SPARK-2203

Author: Aaron Davidson <aaron@databricks.com>

Closes #1138 from aarondav/pyfix and squashes the following commits:

1bd5751 [Aaron Davidson] SPARK-2203: PySpark defaults to use same num reduce partitions as map partitions

caneGuy/spark,wangyum/spark,someorz/spark,haowu80s/spark,hhbyyh/spark,guoxiaolongzte/spark,SnappyDataInc/spark,publicRoman/spark,skonto/spark,cloud-fan/spark,ddna1021/spark,nilsgrabbert/spark,narahari92/spark,panfengfeng/spark,kiszk/spark,maropu/spark,eyalfa/spark,someorz/spark,wangyum/spark,zero323/spark,bdrillard/spark,actuaryzhang/spark,pgandhi999/spark,ajaysaini725/spark,wzhfy/spark,nlalevee/spark,mahak/spark,dongjoon-hyun/spark,WeichenXu123/spark,loneknightpy/spark,hvanhovell/spark,shubhamchopra/spark,mahak/spark,someorz/spark,taroplus/spark,matthewfranglen/spark,xflin/spark,nilsgrabbert/spark,tejasapatil/spark,sureshthalamati/spark,ueshin/apache-spark,rednaxelafx/apache-spark,joseph-torres/spark,publicRoman/spark,mdespriee/spark,hhbyyh/spark,WindCanDie/spark,cin/spark,ron8hu/spark,rednaxelafx/apache-spark,bOOm-X/spark,narahari92/spark,spark-test/spark,rikima/spark,nchammas/spark,dbtsai/spark,tengpeng/spark,akopich/spark,ericvandenbergfb/spark,wzhfy/spark,apache/spark,rikima/spark,kissmetrics/spark,icexelloss/spark,pgandhi999/spark,dbtsai/spark,JerryLead/spark,minixalpha/spark,ajaysaini725/spark,wzhfy/spark,liutang123/spark,pgandhi999/spark,gioenn/xSpark,dhruve/spark,dbtsai/spark,jiangxb1987/spark,wangyum/spark,nlalevee/spark,poffuomo/spark,nilsgrabbert/spark,kimoonkim/spark,mdespriee/spark,jianran/spark,alunarbeach/spark,liutang123/spark,icexelloss/spark,jlopezmalla/spark,ueshin/apache-spark,nchammas/spark,1haodian/spark,pgandhi999/spark,icexelloss/spark,big-pegasus/spark,MLnick/spark,bravo-zhang/spark,shuangshuangwang/spark,bravo-zhang/spark,big-pegasus/spark,minixalpha/spark,adrian-ionescu/apache-spark,bOOm-X/spark,metamx/spark,ConeyLiu/spark,xflin/spark,LantaoJin/spark,ueshin/apache-spark,michalsenkyr/spark,zzcclp/spark,highfei2011/spark,UndeadBaneGitHub/spark,jianran/spark,WindCanDie/spark,milliman/spark,vinodkc/spark,eyalfa/spark,gengliangwang/spark,jlopezmalla/spark,ptkool/spark,byakuinss/spark,caneGuy/spark,jkbradley/spark,haowu80s/spark,haowu80s/spark,SnappyDataInc/spark,huang1900/spark,akopich/spark,huang1900/spark,jkbradley/spark,saturday-shi/spark,aray/spark,JoshRosen/spark,chuckchen/spark,mike0sv/spark,big-pegasus/spark,rekhajoshm/spark,maropu/spark,chuckchen/spark,JerryLead/spark,wangyum/spark,saltstar/spark,narahari92/spark,metamx/spark,SHASHANKB/spark,kevinyu98/spark,ConeyLiu/spark,shuangshuangwang/spark,publicRoman/spark,apache-spark-on-k8s/spark,goldmedal/spark,WindCanDie/spark,SHASHANKB/spark,dongjoon-hyun/spark,pronix/spark,zuotingbing/spark,gengliangwang/spark,rekhajoshm/spark,haowu80s/spark,byakuinss/spark,zzcclp/spark,holdenk/spark,haowu80s/spark,akopich/spark,UndeadBaneGitHub/spark,ibm-research-ireland/sparkoscope,maropu/spark,akopich/spark,srowen/spark,facaiy/spark,markhamstra/spark,ericvandenbergfb/spark,pronix/spark,michalsenkyr/spark,lvdongr/spark,ajaysaini725/spark,zero323/spark,saturday-shi/spark,actuaryzhang/spark,wangmiao1981/spark,ddna1021/spark,publicRoman/spark,someorz/spark,tengpeng/spark,taroplus/spark,lvdongr/spark,ptkool/spark,loneknightpy/spark,MLnick/spark,shuangshuangwang/spark,apache-spark-on-k8s/spark,cloud-fan/spark,WindCanDie/spark,someorz/spark,ddna1021/spark,narahari92/spark,lxsmnv/spark,chuckchen/spark,cloud-fan/spark,eyalfa/spark,narahari92/spark,holdenk/spark,metamx/spark,icexelloss/spark,bravo-zhang/spark,panfengfeng/spark,andrewor14/iolap,hvanhovell/spark,kevinyu98/spark,esi-mineset/spark,metamx/spark,vinodkc/spark,shaneknapp/spark,aosagie/spark,techaddict/spark,ibm-research-ireland/sparkoscope,cin/spark,mzl9039/spark,huang1900/spark,aray/spark,guoxiaolongzte/spark,kimoonkim/spark,hhbyyh/spark,milliman/spark,chuckchen/spark,LantaoJin/spark,apache-spark-on-k8s/spark,BryanCutler/spark,icexelloss/spark,yanboliang/spark,milliman/spark,setjet/spark,kissmetrics/spark,sureshthalamati/spark,cloud-fan/spark,tejasapatil/spark,mzl9039/spark,gioenn/xSpark,srowen/spark,UndeadBaneGitHub/spark,yanboliang/spark,gioenn/xSpark,dhruve/spark,jrshust/spark,aosagie/spark,gioenn/xSpark,kevinyu98/spark,zzcclp/spark,jianran/spark,rezasafi/spark,wangyum/spark,mike0sv/spark,guoxiaolongzte/spark,markhamstra/spark,mahak/spark,ahnqirage/spark,kimoonkim/spark,ron8hu/spark,BryanCutler/spark,rikima/spark,andrewor14/spark,ahnqirage/spark,milliman/spark,sachintyagi22/spark,zhouyejoe/spark,shuangshuangwang/spark,bravo-zhang/spark,kevinyu98/spark,UndeadBaneGitHub/spark,aray/spark,liyichao/spark,loneknightpy/spark,bOOm-X/spark,loneknightpy/spark,jiangxb1987/spark,srowen/spark,tengpeng/spark,rikima/spark,jkbradley/spark,SHASHANKB/spark,sachintyagi22/spark,kevinyu98/spark,adrian-ionescu/apache-spark,kissmetrics/spark,rednaxelafx/apache-spark,debugger87/spark,esi-mineset/spark,alunarbeach/spark,markhamstra/spark,jianran/spark,dbtsai/spark,darionyaphet/spark,byakuinss/spark,zhouyejoe/spark,andrewor14/iolap,gengliangwang/spark,ptkool/spark,andrewor14/iolap,sahilTakiar/spark,ueshin/apache-spark,BryanCutler/spark,shubhamchopra/spark,sachintyagi22/spark,eyalfa/spark,rekhajoshm/spark,ibm-research-ireland/sparkoscope,techaddict/spark,BryanCutler/spark,nchammas/spark,ajaysaini725/spark,caneGuy/spark,actuaryzhang/spark,joseph-torres/spark,apache/spark,sureshthalamati/spark,JoshRosen/spark,andrewor14/spark,szhem/spark,saturday-shi/spark,stanzhai/spark,sahilTakiar/spark,ron8hu/spark,bdrillard/spark,WeichenXu123/spark,byakuinss/spark,adrian-ionescu/apache-spark,andrewor14/iolap,zzcclp/spark,maropu/spark,gengliangwang/spark,goldmedal/spark,WeichenXu123/spark,jiangxb1987/spark,sahilTakiar/spark,bdrillard/spark,highfei2011/spark,aokolnychyi/spark,wangmiao1981/spark,SHASHANKB/spark,1haodian/spark,WindCanDie/spark,alunarbeach/spark,techaddict/spark,tejasapatil/spark,setjet/spark,wzhfy/spark,1haodian/spark,esi-mineset/spark,huang1900/spark,sureshthalamati/spark,vinodkc/spark,jlopezmalla/spark,bdrillard/spark,andrewor14/spark,Panos-Bletsos/spark-cost-model-optimizer,metamx/spark,apache-spark-on-k8s/spark,mzl9039/spark,markhamstra/spark,xflin/spark,rednaxelafx/apache-spark,ajaysaini725/spark,kissmetrics/spark,aosagie/spark,minixalpha/spark,maropu/spark,shaneknapp/spark,loneknightpy/spark,jiangxb1987/spark,sureshthalamati/spark,xflin/spark,spark-test/spark,ioana-delaney/spark,ddna1021/spark,rezasafi/spark,wangmiao1981/spark,yanboliang/spark,witgo/spark,publicRoman/spark,HyukjinKwon/spark,actuaryzhang/spark,tengpeng/spark,akopich/spark,ueshin/apache-spark,ron8hu/spark,panfengfeng/spark,zzcclp/spark,aokolnychyi/spark,SHASHANKB/spark,stanzhai/spark,JoshRosen/spark,rikima/spark,JoshRosen/spark,aosagie/spark,lvdongr/spark,sahilTakiar/spark,kissmetrics/spark,huang1900/spark,narahari92/spark,aray/spark,tengpeng/spark,minixalpha/spark,bdrillard/spark,zuotingbing/spark,patrick-nicholson/spark,SHASHANKB/spark,caneGuy/spark,hvanhovell/spark,sahilTakiar/spark,JerryLead/spark,spark-test/spark,ddna1021/spark,ioana-delaney/spark,stanzhai/spark,szhem/spark,shaneknapp/spark,SnappyDataInc/spark,lvdongr/spark,brad-kaiser/spark,brad-kaiser/spark,alunarbeach/spark,jianran/spark,jlopezmalla/spark,big-pegasus/spark,spark-test/spark,jkbradley/spark,brad-kaiser/spark,zzcclp/spark,apache/spark,liyichao/spark,witgo/spark,WeichenXu123/spark,brad-kaiser/spark,JoshRosen/spark,bOOm-X/spark,kimoonkim/spark,debugger87/spark,alunarbeach/spark,jrshust/spark,esi-mineset/spark,shaneknapp/spark,shubhamchopra/spark,byakuinss/spark,nchammas/spark,holdenk/spark,LantaoJin/spark,eyalfa/spark,WeichenXu123/spark,map222/spark,LantaoJin/spark,patrick-nicholson/spark,xuanyuanking/spark,markhamstra/spark,kimoonkim/spark,darionyaphet/spark,saturday-shi/spark,ConeyLiu/spark,HyukjinKwon/spark,goldmedal/spark,michalsenkyr/spark,xuanyuanking/spark,brad-kaiser/spark,pgandhi999/spark,setjet/spark,hhbyyh/spark,eyalfa/spark,kiszk/spark,ioana-delaney/spark,jkbradley/spark,szhem/spark,bOOm-X/spark,guoxiaolongzte/spark,tejasapatil/spark,JoshRosen/spark,srowen/spark,lvdongr/spark,markhamstra/spark,gioenn/xSpark,ibm-research-ireland/sparkoscope,BryanCutler/spark,ddna1021/spark,bdrillard/spark,SnappyDataInc/spark,nchammas/spark,ioana-delaney/spark,saturday-shi/spark,loneknightpy/spark,bOOm-X/spark,gengliangwang/spark,adrian-ionescu/apache-spark,mike0sv/spark,spark-test/spark,map222/spark,WindCanDie/spark,mahak/spark,setjet/spark,bOOm-X/spark,jkbradley/spark,sachintyagi22/spark,jlopezmalla/spark,pgandhi999/spark,ibm-research-ireland/sparkoscope,michalsenkyr/spark,kiszk/spark,ajaysaini725/spark,rekhajoshm/spark,zero323/spark,vinodkc/spark,SHASHANKB/spark,rezasafi/spark,dhruve/spark,cin/spark,panfengfeng/spark,shubhamchopra/spark,actuaryzhang/spark,big-pegasus/spark,map222/spark,zero323/spark,apache/spark,goldmedal/spark,wangyum/spark,dbtsai/spark,techaddict/spark,skonto/spark,taroplus/spark,saltstar/spark,liyichao/spark,dbtsai/spark,nilsgrabbert/spark,zhouyejoe/spark,highfei2011/spark,jrshust/spark,nilsgrabbert/spark,liutang123/spark,dbtsai/spark,ioana-delaney/spark,lxsmnv/spark,icexelloss/spark,taroplus/spark,mike0sv/spark,highfei2011/spark,mdespriee/spark,nlalevee/spark,shubhamchopra/spark,kevinyu98/spark,narahari92/spark,stanzhai/spark,highfei2011/spark,kiszk/spark,shuangshuangwang/spark,big-pegasus/spark,Panos-Bletsos/spark-cost-model-optimizer,pgandhi999/spark,facaiy/spark,1haodian/spark,Aegeaner/spark,janewangfb/spark,saturday-shi/spark,ConeyLiu/spark,Aegeaner/spark,ericvandenbergfb/spark,BryanCutler/spark,skonto/spark,darionyaphet/spark,zhouyejoe/spark,holdenk/spark,debugger87/spark,WeichenXu123/spark,darionyaphet/spark,skonto/spark,gengliangwang/spark,mike0sv/spark,zuotingbing/spark,facaiy/spark,shaneknapp/spark,andrewor14/iolap,cin/spark,map222/spark,zuotingbing/spark,LantaoJin/spark,byakuinss/spark,darionyaphet/spark,aray/spark,spark-test/spark,xuanyuanking/spark,minixalpha/spark,goldmedal/spark,zuotingbing/spark,nlalevee/spark,apache/spark,ptkool/spark,aokolnychyi/spark,spark-test/spark,ddna1021/spark,poffuomo/spark,debugger87/spark,guoxiaolongzte/spark,srowen/spark,ericvandenbergfb/spark,stanzhai/spark,brad-kaiser/spark,ueshin/apache-spark,andrewor14/iolap,jkbradley/spark,Panos-Bletsos/spark-cost-model-optimizer,LantaoJin/spark,mzl9039/spark,bravo-zhang/spark,ahnqirage/spark,debugger87/spark,highfei2011/spark,aokolnychyi/spark,apache/spark,alunarbeach/spark,actuaryzhang/spark,setjet/spark,nchammas/spark,hvanhovell/spark,andrewor14/spark,mahak/spark,apache-spark-on-k8s/spark,Panos-Bletsos/spark-cost-model-optimizer,xuanyuanking/spark,zzcclp/spark,liyichao/spark,lxsmnv/spark,mahak/spark,zero323/spark,Panos-Bletsos/spark-cost-model-optimizer,joseph-torres/spark,nlalevee/spark,adrian-ionescu/apache-spark,minixalpha/spark,shubhamchopra/spark,michalsenkyr/spark,debugger87/spark,panfengfeng/spark,srowen/spark,mzl9039/spark,sachintyagi22/spark,janewangfb/spark,rekhajoshm/spark,wangmiao1981/spark,guoxiaolongzte/spark,patrick-nicholson/spark,gioenn/xSpark,lxsmnv/spark,ptkool/spark,HyukjinKwon/spark,pronix/spark,SnappyDataInc/spark,jlopezmalla/spark,sureshthalamati/spark,minixalpha/spark,rezasafi/spark,someorz/spark,sureshthalamati/spark,shaneknapp/spark,tejasapatil/spark,zuotingbing/spark,poffuomo/spark,dongjoon-hyun/spark,huang1900/spark,goldmedal/spark,andrewor14/spark,tengpeng/spark,loneknightpy/spark,alunarbeach/spark,maropu/spark,ron8hu/spark,big-pegasus/spark,cloud-fan/spark,ron8hu/spark,szhem/spark,kiszk/spark,joseph-torres/spark,jianran/spark,setjet/spark,maropu/spark,byakuinss/spark,Aegeaner/spark,skonto/spark,kiszk/spark,facaiy/spark,ioana-delaney/spark,facaiy/spark,JerryLead/spark,mdespriee/spark,milliman/spark,wangmiao1981/spark,lvdongr/spark,xflin/spark,zhouyejoe/spark,nlalevee/spark,rikima/spark,wangmiao1981/spark,sachintyagi22/spark,rezasafi/spark,HyukjinKwon/spark,liyichao/spark,publicRoman/spark,JerryLead/spark,wzhfy/spark,UndeadBaneGitHub/spark,haowu80s/spark,nilsgrabbert/spark,michalsenkyr/spark,dongjoon-hyun/spark,map222/spark,yanboliang/spark,yanboliang/spark,mdespriee/spark,ahnqirage/spark,wangmiao1981/spark,jrshust/spark,jlopezmalla/spark,mahak/spark,publicRoman/spark,taroplus/spark,caneGuy/spark,poffuomo/spark,HyukjinKwon/spark,sachintyagi22/spark,adrian-ionescu/apache-spark,darionyaphet/spark,shubhamchopra/spark,milliman/spark,zuotingbing/spark,szhem/spark,actuaryzhang/spark,mzl9039/spark,liutang123/spark,poffuomo/spark,esi-mineset/spark,saltstar/spark,wzhfy/spark,caneGuy/spark,ptkool/spark,MLnick/spark,aokolnychyi/spark,poffuomo/spark,poffuomo/spark,szhem/spark,liutang123/spark,xflin/spark,MLnick/spark,ptkool/spark,huang1900/spark,hhbyyh/spark,dongjoon-hyun/spark,jrshust/spark,ioana-delaney/spark,dhruve/spark,lxsmnv/spark,pronix/spark,LantaoJin/spark,mdespriee/spark,yanboliang/spark,michalsenkyr/spark,ajaysaini725/spark,facaiy/spark,bravo-zhang/spark,cloud-fan/spark,taroplus/spark,saltstar/spark,witgo/spark,mike0sv/spark,ahnqirage/spark,kissmetrics/spark,1haodian/spark,dhruve/spark,taroplus/spark,zero323/spark,ueshin/apache-spark,wzhfy/spark,witgo/spark,mdespriee/spark,bravo-zhang/spark,aosagie/spark,ericvandenbergfb/spark,ConeyLiu/spark,hhbyyh/spark,dongjoon-hyun/spark,chuckchen/spark,dongjoon-hyun/spark,guoxiaolongzte/spark,shuangshuangwang/spark,rezasafi/spark,liutang123/spark,icexelloss/spark,dhruve/spark,techaddict/spark,1haodian/spark,esi-mineset/spark,techaddict/spark,Panos-Bletsos/spark-cost-model-optimizer,panfengfeng/spark,aray/spark,jiangxb1987/spark,nchammas/spark,shuangshuangwang/spark,SnappyDataInc/spark,brad-kaiser/spark,nilsgrabbert/spark,aosagie/spark,tejasapatil/spark,vinodkc/spark,joseph-torres/spark,cin/spark,ahnqirage/spark,sahilTakiar/spark,adrian-ionescu/apache-spark,holdenk/spark,shaneknapp/spark,akopich/spark,patrick-nicholson/spark,rikima/spark,caneGuy/spark,jrshust/spark,Aegeaner/spark,gioenn/xSpark,pronix/spark,Aegeaner/spark,janewangfb/spark,mzl9039/spark,debugger87/spark,tejasapatil/spark,BryanCutler/spark,vinodkc/spark,patrick-nicholson/spark,JerryLead/spark,techaddict/spark,ahnqirage/spark,apache/spark,rekhajoshm/spark,WindCanDie/spark,MLnick/spark,darionyaphet/spark,ericvandenbergfb/spark,ConeyLiu/spark,MLnick/spark,saltstar/spark,map222/spark,andrewor14/spark,lvdongr/spark,saltstar/spark,xuanyuanking/spark,jrshust/spark,hhbyyh/spark,xuanyuanking/spark,kevinyu98/spark,UndeadBaneGitHub/spark,goldmedal/spark,map222/spark,mike0sv/spark,kimoonkim/spark,joseph-torres/spark,metamx/spark,Panos-Bletsos/spark-cost-model-optimizer,joseph-torres/spark,lxsmnv/spark,setjet/spark,esi-mineset/spark,facaiy/spark,holdenk/spark,dhruve/spark,hvanhovell/spark,janewangfb/spark,witgo/spark,hvanhovell/spark,highfei2011/spark,tengpeng/spark,witgo/spark,sahilTakiar/spark,nlalevee/spark,Aegeaner/spark,UndeadBaneGitHub/spark,kiszk/spark,skonto/spark,rednaxelafx/apache-spark,MLnick/spark,ConeyLiu/spark,stanzhai/spark,apache-spark-on-k8s/spark,janewangfb/spark,cin/spark,saltstar/spark,JoshRosen/spark,jianran/spark,stanzhai/spark,Aegeaner/spark,cin/spark,markhamstra/spark,WeichenXu123/spark,akopich/spark,xuanyuanking/spark,saturday-shi/spark,rezasafi/spark,zhouyejoe/spark,janewangfb/spark,kimoonkim/spark,liyichao/spark,SnappyDataInc/spark,aray/spark,jiangxb1987/spark,apache-spark-on-k8s/spark,rednaxelafx/apache-spark,rednaxelafx/apache-spark,panfengfeng/spark,metamx/spark,rekhajoshm/spark,aosagie/spark,bdrillard/spark,andrewor14/spark,chuckchen/spark,aokolnychyi/spark,zero323/spark,lxsmnv/spark,1haodian/spark,liyichao/spark,milliman/spark,aokolnychyi/spark,ron8hu/spark,gengliangwang/spark,szhem/spark,zhouyejoe/spark,HyukjinKwon/spark,ibm-research-ireland/sparkoscope,someorz/spark,wangyum/spark,skonto/spark,witgo/spark,srowen/spark,liutang123/spark,patrick-nicholson/spark,ibm-research-ireland/sparkoscope,eyalfa/spark,pronix/spark,hvanhovell/spark,cloud-fan/spark,kissmetrics/spark,patrick-nicholson/spark,jiangxb1987/spark,janewangfb/spark,andrewor14/iolap,HyukjinKwon/spark,chuckchen/spark,JerryLead/spark,yanboliang/spark,vinodkc/spark,ericvandenbergfb/spark,xflin/spark,holdenk/spark