[SPARK-2871] [PySpark] add RDD.lookup(key)
[SPARK-2871] [PySpark] add RDD.lookup(key)

RDD.lookup(key)

        Return the list of values in the RDD for key `key`. This operation
        is done efficiently if the RDD has a known partitioner by only
        searching the partition that the key maps to.

        >>> l = range(1000)
        >>> rdd = sc.parallelize(zip(l, l), 10)
        >>> rdd.lookup(42)  # slow
        [42]
        >>> sorted = rdd.sortByKey()
        >>> sorted.lookup(42)  # fast
        [42]

It also clean up the code in RDD.py, and fix several bugs (related to preservesPartitioning).

Author: Davies Liu <davies.liu@gmail.com>

Closes #2093 from davies/lookup and squashes the following commits:

1789cd4 [Davies Liu] `f` in foreach could be generator or not.
2871b80 [Davies Liu] Merge branch 'master' into lookup
c6390ea [Davies Liu] address all comments
0f1bce8 [Davies Liu] add test case for lookup()
be0e8ba [Davies Liu] fix preservesPartitioning
eb1305d [Davies Liu] add RDD.lookup(key)

sachintyagi22/spark,alunarbeach/spark,chuckchen/spark,debugger87/spark,shuangshuangwang/spark,liutang123/spark,LantaoJin/spark,dhruve/spark,zuotingbing/spark,jiangxb1987/spark,lvdongr/spark,stanzhai/spark,witgo/spark,ioana-delaney/spark,byakuinss/spark,WeichenXu123/spark,patrick-nicholson/spark,eyalfa/spark,bdrillard/spark,wangmiao1981/spark,loneknightpy/spark,WindCanDie/spark,WeichenXu123/spark,rezasafi/spark,pronix/spark,aokolnychyi/spark,tengpeng/spark,yanboliang/spark,shubhamchopra/spark,bravo-zhang/spark,liutang123/spark,saltstar/spark,janewangfb/spark,sachintyagi22/spark,ueshin/apache-spark,narahari92/spark,Panos-Bletsos/spark-cost-model-optimizer,lvdongr/spark,rednaxelafx/apache-spark,zuotingbing/spark,1haodian/spark,sachintyagi22/spark,dongjoon-hyun/spark,minixalpha/spark,ajaysaini725/spark,hhbyyh/spark,WindCanDie/spark,spark-test/spark,patrick-nicholson/spark,janewangfb/spark,lxsmnv/spark,cin/spark,kevinyu98/spark,icexelloss/spark,JerryLead/spark,jkbradley/spark,sahilTakiar/spark,andrewor14/spark,ConeyLiu/spark,actuaryzhang/spark,minixalpha/spark,wzhfy/spark,SnappyDataInc/spark,loneknightpy/spark,haowu80s/spark,witgo/spark,lxsmnv/spark,icexelloss/spark,chuckchen/spark,debugger87/spark,sureshthalamati/spark,ConeyLiu/spark,jlopezmalla/spark,ericvandenbergfb/spark,tengpeng/spark,goldmedal/spark,liyichao/spark,SHASHANKB/spark,loneknightpy/spark,michalsenkyr/spark,publicRoman/spark,ptkool/spark,WeichenXu123/spark,wzhfy/spark,rednaxelafx/apache-spark,BryanCutler/spark,publicRoman/spark,saturday-shi/spark,cin/spark,ibm-research-ireland/sparkoscope,kimoonkim/spark,jiangxb1987/spark,nlalevee/spark,andrewor14/spark,big-pegasus/spark,mdespriee/spark,eyalfa/spark,shubhamchopra/spark,jianran/spark,lxsmnv/spark,rezasafi/spark,yanboliang/spark,jiangxb1987/spark,caneGuy/spark,aray/spark,narahari92/spark,michalsenkyr/spark,mike0sv/spark,janewangfb/spark,nchammas/spark,ahnqirage/spark,nilsgrabbert/spark,poffuomo/spark,actuaryzhang/spark,BryanCutler/spark,ddna1021/spark,mdespriee/spark,goldmedal/spark,byakuinss/spark,debugger87/spark,cin/spark,jlopezmalla/spark,highfei2011/spark,wangyum/spark,kiszk/spark,aosagie/spark,andrewor14/iolap,guoxiaolongzte/spark,highfei2011/spark,hvanhovell/spark,alunarbeach/spark,techaddict/spark,esi-mineset/spark,chuckchen/spark,zuotingbing/spark,lvdongr/spark,nchammas/spark,ioana-delaney/spark,wangmiao1981/spark,ueshin/apache-spark,saltstar/spark,szhem/spark,someorz/spark,vinodkc/spark,apache/spark,skonto/spark,ron8hu/spark,icexelloss/spark,apache-spark-on-k8s/spark,wangmiao1981/spark,Panos-Bletsos/spark-cost-model-optimizer,1haodian/spark,jianran/spark,jlopezmalla/spark,WeichenXu123/spark,BryanCutler/spark,apache/spark,jkbradley/spark,adrian-ionescu/apache-spark,apache-spark-on-k8s/spark,maropu/spark,chuckchen/spark,ConeyLiu/spark,jianran/spark,jiangxb1987/spark,1haodian/spark,mzl9039/spark,chuckchen/spark,michalsenkyr/spark,kissmetrics/spark,aray/spark,UndeadBaneGitHub/spark,andrewor14/iolap,zzcclp/spark,ahnqirage/spark,UndeadBaneGitHub/spark,wzhfy/spark,akopich/spark,hvanhovell/spark,ibm-research-ireland/sparkoscope,loneknightpy/spark,eyalfa/spark,shubhamchopra/spark,patrick-nicholson/spark,mdespriee/spark,sureshthalamati/spark,pgandhi999/spark,byakuinss/spark,SnappyDataInc/spark,wangyum/spark,sureshthalamati/spark,someorz/spark,ptkool/spark,sachintyagi22/spark,nchammas/spark,zzcclp/spark,bOOm-X/spark,someorz/spark,michalsenkyr/spark,pronix/spark,big-pegasus/spark,ericvandenbergfb/spark,Panos-Bletsos/spark-cost-model-optimizer,bravo-zhang/spark,joseph-torres/spark,esi-mineset/spark,xuanyuanking/spark,spark-test/spark,mike0sv/spark,tejasapatil/spark,dbtsai/spark,eyalfa/spark,JerryLead/spark,ddna1021/spark,ajaysaini725/spark,poffuomo/spark,JerryLead/spark,stanzhai/spark,actuaryzhang/spark,holdenk/spark,mzl9039/spark,lxsmnv/spark,ueshin/apache-spark,vinodkc/spark,shuangshuangwang/spark,BryanCutler/spark,matthewfranglen/spark,map222/spark,techaddict/spark,akopich/spark,akopich/spark,apache-spark-on-k8s/spark,apache/spark,bdrillard/spark,bOOm-X/spark,joseph-torres/spark,taroplus/spark,poffuomo/spark,Aegeaner/spark,liyichao/spark,witgo/spark,ddna1021/spark,shuangshuangwang/spark,LantaoJin/spark,dongjoon-hyun/spark,dhruve/spark,ConeyLiu/spark,joseph-torres/spark,publicRoman/spark,witgo/spark,zero323/spark,jiangxb1987/spark,eyalfa/spark,gengliangwang/spark,UndeadBaneGitHub/spark,dongjoon-hyun/spark,wangmiao1981/spark,mdespriee/spark,LantaoJin/spark,skonto/spark,ioana-delaney/spark,guoxiaolongzte/spark,zero323/spark,darionyaphet/spark,stanzhai/spark,bravo-zhang/spark,witgo/spark,Panos-Bletsos/spark-cost-model-optimizer,rekhajoshm/spark,lvdongr/spark,haowu80s/spark,darionyaphet/spark,aosagie/spark,facaiy/spark,facaiy/spark,huang1900/spark,yanboliang/spark,aray/spark,narahari92/spark,srowen/spark,kevinyu98/spark,actuaryzhang/spark,JerryLead/spark,shaneknapp/spark,shubhamchopra/spark,huang1900/spark,xflin/spark,gengliangwang/spark,goldmedal/spark,jiangxb1987/spark,milliman/spark,sureshthalamati/spark,ajaysaini725/spark,andrewor14/spark,kevinyu98/spark,rikima/spark,xuanyuanking/spark,jrshust/spark,dhruve/spark,ericvandenbergfb/spark,HyukjinKwon/spark,publicRoman/spark,holdenk/spark,zero323/spark,saltstar/spark,mahak/spark,sureshthalamati/spark,byakuinss/spark,rikima/spark,publicRoman/spark,nchammas/spark,ahnqirage/spark,kevinyu98/spark,HyukjinKwon/spark,taroplus/spark,ueshin/apache-spark,szhem/spark,bOOm-X/spark,WindCanDie/spark,alunarbeach/spark,highfei2011/spark,kimoonkim/spark,someorz/spark,brad-kaiser/spark,tejasapatil/spark,1haodian/spark,liyichao/spark,dbtsai/spark,panfengfeng/spark,ueshin/apache-spark,cloud-fan/spark,kissmetrics/spark,JoshRosen/spark,bdrillard/spark,hhbyyh/spark,xuanyuanking/spark,cloud-fan/spark,markhamstra/spark,zuotingbing/spark,setjet/spark,holdenk/spark,pronix/spark,szhem/spark,tengpeng/spark,big-pegasus/spark,ajaysaini725/spark,rezasafi/spark,skonto/spark,liutang123/spark,aokolnychyi/spark,kevinyu98/spark,vinodkc/spark,mzl9039/spark,map222/spark,maropu/spark,JoshRosen/spark,mdespriee/spark,facaiy/spark,byakuinss/spark,cin/spark,pronix/spark,pgandhi999/spark,darionyaphet/spark,markhamstra/spark,michalsenkyr/spark,bOOm-X/spark,wzhfy/spark,darionyaphet/spark,mahak/spark,sahilTakiar/spark,1haodian/spark,wzhfy/spark,liyichao/spark,loneknightpy/spark,xflin/spark,rikima/spark,ddna1021/spark,nchammas/spark,mike0sv/spark,ron8hu/spark,hvanhovell/spark,UndeadBaneGitHub/spark,bOOm-X/spark,sahilTakiar/spark,MLnick/spark,kimoonkim/spark,1haodian/spark,maropu/spark,guoxiaolongzte/spark,janewangfb/spark,apache/spark,mzl9039/spark,caneGuy/spark,hvanhovell/spark,ibm-research-ireland/sparkoscope,wzhfy/spark,szhem/spark,mzl9039/spark,zhouyejoe/spark,ioana-delaney/spark,jlopezmalla/spark,zhouyejoe/spark,dbtsai/spark,SHASHANKB/spark,sahilTakiar/spark,metamx/spark,xflin/spark,SnappyDataInc/spark,debugger87/spark,map222/spark,kiszk/spark,actuaryzhang/spark,WeichenXu123/spark,liyichao/spark,bravo-zhang/spark,srowen/spark,nilsgrabbert/spark,hhbyyh/spark,skonto/spark,joseph-torres/spark,stanzhai/spark,brad-kaiser/spark,taroplus/spark,aosagie/spark,Aegeaner/spark,kimoonkim/spark,loneknightpy/spark,xflin/spark,zzcclp/spark,minixalpha/spark,kimoonkim/spark,hvanhovell/spark,witgo/spark,jianran/spark,aokolnychyi/spark,esi-mineset/spark,big-pegasus/spark,SHASHANKB/spark,actuaryzhang/spark,rikima/spark,mike0sv/spark,SnappyDataInc/spark,zhouyejoe/spark,haowu80s/spark,bravo-zhang/spark,dhruve/spark,map222/spark,andrewor14/spark,ibm-research-ireland/sparkoscope,rikima/spark,mdespriee/spark,ptkool/spark,Aegeaner/spark,gioenn/xSpark,icexelloss/spark,bdrillard/spark,rednaxelafx/apache-spark,guoxiaolongzte/spark,byakuinss/spark,UndeadBaneGitHub/spark,apache-spark-on-k8s/spark,metamx/spark,shaneknapp/spark,ajaysaini725/spark,facaiy/spark,zzcclp/spark,esi-mineset/spark,srowen/spark,tengpeng/spark,patrick-nicholson/spark,MLnick/spark,janewangfb/spark,huang1900/spark,ddna1021/spark,andrewor14/iolap,wangyum/spark,JoshRosen/spark,loneknightpy/spark,adrian-ionescu/apache-spark,ajaysaini725/spark,aosagie/spark,aosagie/spark,gioenn/xSpark,lvdongr/spark,milliman/spark,alunarbeach/spark,maropu/spark,ahnqirage/spark,maropu/spark,bdrillard/spark,skonto/spark,cin/spark,kissmetrics/spark,MLnick/spark,zero323/spark,vinodkc/spark,metamx/spark,adrian-ionescu/apache-spark,ConeyLiu/spark,ron8hu/spark,andrewor14/iolap,techaddict/spark,rezasafi/spark,jlopezmalla/spark,narahari92/spark,janewangfb/spark,jrshust/spark,markhamstra/spark,aokolnychyi/spark,ajaysaini725/spark,Aegeaner/spark,cin/spark,alunarbeach/spark,someorz/spark,mahak/spark,LantaoJin/spark,nlalevee/spark,mike0sv/spark,rekhajoshm/spark,shubhamchopra/spark,jlopezmalla/spark,someorz/spark,LantaoJin/spark,jianran/spark,milliman/spark,shaneknapp/spark,wangyum/spark,tengpeng/spark,wangmiao1981/spark,kissmetrics/spark,holdenk/spark,techaddict/spark,facaiy/spark,darionyaphet/spark,joseph-torres/spark,szhem/spark,wangyum/spark,ron8hu/spark,tejasapatil/spark,icexelloss/spark,bravo-zhang/spark,shubhamchopra/spark,ibm-research-ireland/sparkoscope,liyichao/spark,brad-kaiser/spark,BryanCutler/spark,huang1900/spark,1haodian/spark,hhbyyh/spark,panfengfeng/spark,kiszk/spark,metamx/spark,cloud-fan/spark,HyukjinKwon/spark,saturday-shi/spark,ahnqirage/spark,publicRoman/spark,adrian-ionescu/apache-spark,dongjoon-hyun/spark,rezasafi/spark,chuckchen/spark,SHASHANKB/spark,aosagie/spark,panfengfeng/spark,xuanyuanking/spark,metamx/spark,liutang123/spark,taroplus/spark,aokolnychyi/spark,kiszk/spark,kissmetrics/spark,tengpeng/spark,xflin/spark,nlalevee/spark,pgandhi999/spark,mike0sv/spark,esi-mineset/spark,gioenn/xSpark,patrick-nicholson/spark,vinodkc/spark,maropu/spark,adrian-ionescu/apache-spark,caneGuy/spark,spark-test/spark,dbtsai/spark,Panos-Bletsos/spark-cost-model-optimizer,jlopezmalla/spark,shaneknapp/spark,xuanyuanking/spark,techaddict/spark,jrshust/spark,ddna1021/spark,michalsenkyr/spark,joseph-torres/spark,gioenn/xSpark,aray/spark,xflin/spark,jianran/spark,poffuomo/spark,setjet/spark,ericvandenbergfb/spark,Panos-Bletsos/spark-cost-model-optimizer,kiszk/spark,vinodkc/spark,minixalpha/spark,saltstar/spark,highfei2011/spark,nilsgrabbert/spark,JoshRosen/spark,ericvandenbergfb/spark,shaneknapp/spark,adrian-ionescu/apache-spark,spark-test/spark,someorz/spark,BryanCutler/spark,sureshthalamati/spark,cloud-fan/spark,akopich/spark,kissmetrics/spark,ericvandenbergfb/spark,rekhajoshm/spark,akopich/spark,HyukjinKwon/spark,panfengfeng/spark,bOOm-X/spark,SHASHANKB/spark,nilsgrabbert/spark,kimoonkim/spark,xuanyuanking/spark,JerryLead/spark,milliman/spark,goldmedal/spark,dhruve/spark,gioenn/xSpark,ptkool/spark,ron8hu/spark,skonto/spark,panfengfeng/spark,apache-spark-on-k8s/spark,debugger87/spark,WindCanDie/spark,kevinyu98/spark,sahilTakiar/spark,highfei2011/spark,rezasafi/spark,jkbradley/spark,ueshin/apache-spark,milliman/spark,andrewor14/spark,caneGuy/spark,aokolnychyi/spark,highfei2011/spark,JoshRosen/spark,holdenk/spark,rednaxelafx/apache-spark,nlalevee/spark,spark-test/spark,shuangshuangwang/spark,holdenk/spark,HyukjinKwon/spark,jrshust/spark,ueshin/apache-spark,lvdongr/spark,UndeadBaneGitHub/spark,big-pegasus/spark,gioenn/xSpark,markhamstra/spark,pgandhi999/spark,minixalpha/spark,poffuomo/spark,setjet/spark,lxsmnv/spark,lxsmnv/spark,saltstar/spark,ConeyLiu/spark,bdrillard/spark,dongjoon-hyun/spark,dongjoon-hyun/spark,apache/spark,metamx/spark,saturday-shi/spark,darionyaphet/spark,Aegeaner/spark,markhamstra/spark,rikima/spark,sureshthalamati/spark,wangmiao1981/spark,darionyaphet/spark,goldmedal/spark,Panos-Bletsos/spark-cost-model-optimizer,MLnick/spark,alunarbeach/spark,andrewor14/spark,esi-mineset/spark,WeichenXu123/spark,szhem/spark,ioana-delaney/spark,markhamstra/spark,jrshust/spark,rednaxelafx/apache-spark,caneGuy/spark,ron8hu/spark,markhamstra/spark,bravo-zhang/spark,ibm-research-ireland/sparkoscope,ptkool/spark,rednaxelafx/apache-spark,SHASHANKB/spark,guoxiaolongzte/spark,saturday-shi/spark,stanzhai/spark,MLnick/spark,nchammas/spark,dbtsai/spark,zero323/spark,haowu80s/spark,gengliangwang/spark,guoxiaolongzte/spark,sachintyagi22/spark,jkbradley/spark,aosagie/spark,lvdongr/spark,ericvandenbergfb/spark,zuotingbing/spark,rednaxelafx/apache-spark,kevinyu98/spark,hhbyyh/spark,jrshust/spark,minixalpha/spark,milliman/spark,guoxiaolongzte/spark,shuangshuangwang/spark,publicRoman/spark,SHASHANKB/spark,huang1900/spark,narahari92/spark,cloud-fan/spark,alunarbeach/spark,andrewor14/spark,xflin/spark,tejasapatil/spark,szhem/spark,ahnqirage/spark,ioana-delaney/spark,esi-mineset/spark,WindCanDie/spark,nlalevee/spark,byakuinss/spark,brad-kaiser/spark,gengliangwang/spark,pronix/spark,mahak/spark,map222/spark,Aegeaner/spark,saturday-shi/spark,pgandhi999/spark,debugger87/spark,aray/spark,mahak/spark,mzl9039/spark,kissmetrics/spark,gengliangwang/spark,cloud-fan/spark,mzl9039/spark,saltstar/spark,tejasapatil/spark,srowen/spark,nilsgrabbert/spark,MLnick/spark,srowen/spark,zzcclp/spark,wzhfy/spark,saltstar/spark,HyukjinKwon/spark,highfei2011/spark,nlalevee/spark,wangyum/spark,taroplus/spark,kiszk/spark,aray/spark,andrewor14/iolap,caneGuy/spark,JerryLead/spark,zuotingbing/spark,SnappyDataInc/spark,sahilTakiar/spark,MLnick/spark,panfengfeng/spark,sahilTakiar/spark,patrick-nicholson/spark,poffuomo/spark,icexelloss/spark,bdrillard/spark,ddna1021/spark,hvanhovell/spark,narahari92/spark,LantaoJin/spark,srowen/spark,yanboliang/spark,stanzhai/spark,hvanhovell/spark,yanboliang/spark,big-pegasus/spark,wangyum/spark,zhouyejoe/spark,eyalfa/spark,shuangshuangwang/spark,brad-kaiser/spark,patrick-nicholson/spark,map222/spark,JoshRosen/spark,ConeyLiu/spark,goldmedal/spark,bOOm-X/spark,nilsgrabbert/spark,setjet/spark,shubhamchopra/spark,shuangshuangwang/spark,tejasapatil/spark,lxsmnv/spark,rikima/spark,mahak/spark,ibm-research-ireland/sparkoscope,zero323/spark,michalsenkyr/spark,metamx/spark,mike0sv/spark,mahak/spark,gengliangwang/spark,maropu/spark,WeichenXu123/spark,andrewor14/iolap,brad-kaiser/spark,rekhajoshm/spark,rezasafi/spark,HyukjinKwon/spark,big-pegasus/spark,mdespriee/spark,holdenk/spark,kiszk/spark,joseph-torres/spark,shaneknapp/spark,chuckchen/spark,liutang123/spark,Aegeaner/spark,facaiy/spark,dhruve/spark,dbtsai/spark,minixalpha/spark,dbtsai/spark,saturday-shi/spark,milliman/spark,sachintyagi22/spark,actuaryzhang/spark,WindCanDie/spark,caneGuy/spark,rekhajoshm/spark,panfengfeng/spark,liyichao/spark,apache/spark,haowu80s/spark,cin/spark,LantaoJin/spark,zero323/spark,srowen/spark,vinodkc/spark,akopich/spark,janewangfb/spark,akopich/spark,apache-spark-on-k8s/spark,UndeadBaneGitHub/spark,spark-test/spark,shaneknapp/spark,huang1900/spark,jkbradley/spark,spark-test/spark,jiangxb1987/spark,rekhajoshm/spark,stanzhai/spark,tejasapatil/spark,SnappyDataInc/spark,haowu80s/spark,setjet/spark,nilsgrabbert/spark,pgandhi999/spark,andrewor14/iolap,zzcclp/spark,JoshRosen/spark,yanboliang/spark,brad-kaiser/spark,tengpeng/spark,taroplus/spark,liutang123/spark,apache-spark-on-k8s/spark,poffuomo/spark,eyalfa/spark,yanboliang/spark,zzcclp/spark,sachintyagi22/spark,gioenn/xSpark,zuotingbing/spark,jianran/spark,BryanCutler/spark,liutang123/spark,icexelloss/spark,ahnqirage/spark,adrian-ionescu/apache-spark,JerryLead/spark,setjet/spark,setjet/spark,hhbyyh/spark,ioana-delaney/spark,aray/spark,facaiy/spark,gengliangwang/spark,jkbradley/spark,dongjoon-hyun/spark,techaddict/spark,witgo/spark,zhouyejoe/spark,hhbyyh/spark,debugger87/spark,nchammas/spark,nlalevee/spark,ptkool/spark,ron8hu/spark,SnappyDataInc/spark,skonto/spark,ptkool/spark,xuanyuanking/spark,huang1900/spark,saturday-shi/spark,techaddict/spark,rekhajoshm/spark,dhruve/spark,pgandhi999/spark,WindCanDie/spark,pronix/spark,zhouyejoe/spark,map222/spark,goldmedal/spark,apache/spark,narahari92/spark,zhouyejoe/spark,cloud-fan/spark,wangmiao1981/spark,jrshust/spark,jkbradley/spark,kimoonkim/spark,aokolnychyi/spark,taroplus/spark