[SPARK-11437] [PYSPARK] Don't .take when converting RDD to DataFrame with provided schema
[SPARK-11437] [PYSPARK] Don't .take when converting RDD to DataFrame with provided schema

When creating a DataFrame from an RDD in PySpark, `createDataFrame` calls `.take(10)` to verify the first 10 rows of the RDD match the provided schema. Similar to https://issues.apache.org/jira/browse/SPARK-8070, but that issue affected cases where a schema was not provided.

Verifying the first 10 rows is of limited utility and causes the DAG to be executed non-lazily. If necessary, I believe this verification should be done lazily on all rows. However, since the caller is providing a schema to follow, I think it's acceptable to simply fail if the schema is incorrect.

marmbrus We chatted about this at SparkSummitEU. davies you made a similar change for the infer-schema path in https://github.com/apache/spark/pull/6606

Author: Jason White <jason.white@shopify.com>

Closes #9392 from JasonMWhite/createDataFrame_without_take.

mike0sv/spark,dbtsai/spark,yanboliang/spark,apache-spark-on-k8s/spark,map222/spark,SnappyDataInc/spark,zzcclp/spark,haowu80s/spark,gengliangwang/spark,saturday-shi/spark,kimoonkim/spark,bOOm-X/spark,nilsgrabbert/spark,zuotingbing/spark,gioenn/xSpark,andrewor14/spark,zero323/spark,wangmiao1981/spark,saturday-shi/spark,loneknightpy/spark,MLnick/spark,esi-mineset/spark,apache-spark-on-k8s/spark,chuckchen/spark,zuotingbing/spark,holdenk/spark,hvanhovell/spark,jianran/spark,aosagie/spark,highfei2011/spark,wzhfy/spark,mike0sv/spark,goldmedal/spark,milliman/spark,maropu/spark,ahnqirage/spark,dbtsai/spark,debugger87/spark,rekhajoshm/spark,sahilTakiar/spark,rezasafi/spark,adrian-ionescu/apache-spark,ueshin/apache-spark,map222/spark,panfengfeng/spark,esi-mineset/spark,holdenk/spark,ron8hu/spark,alunarbeach/spark,yanboliang/spark,caneGuy/spark,BryanCutler/spark,darionyaphet/spark,aokolnychyi/spark,UndeadBaneGitHub/spark,haowu80s/spark,dongjoon-hyun/spark,sahilTakiar/spark,someorz/spark,BryanCutler/spark,rednaxelafx/apache-spark,sureshthalamati/spark,big-pegasus/spark,holdenk/spark,shubhamchopra/spark,xuanyuanking/spark,techaddict/spark,dhruve/spark,hvanhovell/spark,jrshust/spark,ConeyLiu/spark,taroplus/spark,JoshRosen/spark,zzcclp/spark,taroplus/spark,kimoonkim/spark,jianran/spark,highfei2011/spark,ddna1021/spark,mdespriee/spark,gioenn/xSpark,nlalevee/spark,bravo-zhang/spark,rezasafi/spark,esi-mineset/spark,hhbyyh/spark,liyichao/spark,ahnqirage/spark,jianran/spark,WeichenXu123/spark,tengpeng/spark,joseph-torres/spark,chuckchen/spark,metamx/spark,rezasafi/spark,shuangshuangwang/spark,minixalpha/spark,rednaxelafx/apache-spark,aosagie/spark,gioenn/xSpark,icexelloss/spark,ahnqirage/spark,jianran/spark,panfengfeng/spark,ioana-delaney/spark,shaneknapp/spark,joseph-torres/spark,xuanyuanking/spark,tengpeng/spark,jlopezmalla/spark,wzhfy/spark,Panos-Bletsos/spark-cost-model-optimizer,UndeadBaneGitHub/spark,pgandhi999/spark,stanzhai/spark,ueshin/apache-spark,sureshthalamati/spark,sachintyagi22/spark,SHASHANKB/spark,sachintyagi22/spark,ericvandenbergfb/spark,mdespriee/spark,zzcclp/spark,ioana-delaney/spark,zuotingbing/spark,ConeyLiu/spark,maropu/spark,rezasafi/spark,actuaryzhang/spark,byakuinss/spark,ueshin/apache-spark,andrewor14/spark,wangyum/spark,minixalpha/spark,mzl9039/spark,spark-test/spark,narahari92/spark,yanboliang/spark,wangyum/spark,markhamstra/spark,byakuinss/spark,liutang123/spark,map222/spark,WeichenXu123/spark,bravo-zhang/spark,rikima/spark,bravo-zhang/spark,gengliangwang/spark,poffuomo/spark,cin/spark,stanzhai/spark,hvanhovell/spark,shaneknapp/spark,facaiy/spark,pgandhi999/spark,kevinyu98/spark,apache-spark-on-k8s/spark,zhouyejoe/spark,markhamstra/spark,holdenk/spark,setjet/spark,setjet/spark,bravo-zhang/spark,zero323/spark,big-pegasus/spark,setjet/spark,xuanyuanking/spark,mahak/spark,publicRoman/spark,nilsgrabbert/spark,xuanyuanking/spark,ajaysaini725/spark,brad-kaiser/spark,xuanyuanking/spark,shaneknapp/spark,nlalevee/spark,skonto/spark,goldmedal/spark,pgandhi999/spark,MLnick/spark,icexelloss/spark,michalsenkyr/spark,guoxiaolongzte/spark,aokolnychyi/spark,ibm-research-ireland/sparkoscope,someorz/spark,jiangxb1987/spark,zhouyejoe/spark,gengliangwang/spark,bdrillard/spark,patrick-nicholson/spark,sachintyagi22/spark,ioana-delaney/spark,byakuinss/spark,maropu/spark,saturday-shi/spark,ron8hu/spark,michalsenkyr/spark,sahilTakiar/spark,dhruve/spark,Panos-Bletsos/spark-cost-model-optimizer,mzl9039/spark,big-pegasus/spark,wangmiao1981/spark,rikima/spark,ahnqirage/spark,zuotingbing/spark,rekhajoshm/spark,narahari92/spark,publicRoman/spark,icexelloss/spark,ptkool/spark,wangmiao1981/spark,techaddict/spark,ibm-research-ireland/sparkoscope,debugger87/spark,sureshthalamati/spark,cloud-fan/spark,WindCanDie/spark,gengliangwang/spark,srowen/spark,wangmiao1981/spark,skonto/spark,caneGuy/spark,vinodkc/spark,aray/spark,jlopezmalla/spark,ptkool/spark,eyalfa/spark,someorz/spark,kiszk/spark,srowen/spark,publicRoman/spark,andrewor14/spark,ibm-research-ireland/sparkoscope,liyichao/spark,saltstar/spark,SHASHANKB/spark,brad-kaiser/spark,metamx/spark,ConeyLiu/spark,tejasapatil/spark,sahilTakiar/spark,witgo/spark,andrewor14/spark,taroplus/spark,shaneknapp/spark,joseph-torres/spark,apache/spark,nlalevee/spark,jrshust/spark,goldmedal/spark,haowu80s/spark,setjet/spark,markhamstra/spark,metamx/spark,actuaryzhang/spark,witgo/spark,Aegeaner/spark,someorz/spark,aokolnychyi/spark,aosagie/spark,zuotingbing/spark,rednaxelafx/apache-spark,LantaoJin/spark,HyukjinKwon/spark,sureshthalamati/spark,ericvandenbergfb/spark,ddna1021/spark,bOOm-X/spark,jkbradley/spark,shuangshuangwang/spark,setjet/spark,LantaoJin/spark,ddna1021/spark,akopich/spark,shubhamchopra/spark,maropu/spark,apache/spark,zhouyejoe/spark,ron8hu/spark,HyukjinKwon/spark,akopich/spark,sureshthalamati/spark,facaiy/spark,kissmetrics/spark,chuckchen/spark,huang1900/spark,narahari92/spark,chuckchen/spark,pgandhi999/spark,nchammas/spark,SHASHANKB/spark,kimoonkim/spark,eyalfa/spark,zzcclp/spark,cloud-fan/spark,jlopezmalla/spark,jlopezmalla/spark,loneknightpy/spark,yanboliang/spark,shubhamchopra/spark,icexelloss/spark,mahak/spark,ddna1021/spark,kissmetrics/spark,sahilTakiar/spark,hvanhovell/spark,ddna1021/spark,apache/spark,panfengfeng/spark,srowen/spark,LantaoJin/spark,xflin/spark,chuckchen/spark,xuanyuanking/spark,dbtsai/spark,Panos-Bletsos/spark-cost-model-optimizer,janewangfb/spark,big-pegasus/spark,ConeyLiu/spark,holdenk/spark,WeichenXu123/spark,shuangshuangwang/spark,jiangxb1987/spark,apache-spark-on-k8s/spark,spark-test/spark,someorz/spark,techaddict/spark,bOOm-X/spark,Aegeaner/spark,xflin/spark,darionyaphet/spark,metamx/spark,milliman/spark,kissmetrics/spark,lvdongr/spark,liutang123/spark,hhbyyh/spark,jianran/spark,highfei2011/spark,publicRoman/spark,rikima/spark,bdrillard/spark,cloud-fan/spark,saturday-shi/spark,szhem/spark,stanzhai/spark,1haodian/spark,wzhfy/spark,wangmiao1981/spark,big-pegasus/spark,ajaysaini725/spark,joseph-torres/spark,ueshin/apache-spark,liutang123/spark,dhruve/spark,shuangshuangwang/spark,dbtsai/spark,janewangfb/spark,sahilTakiar/spark,zzcclp/spark,cin/spark,michalsenkyr/spark,ibm-research-ireland/sparkoscope,MLnick/spark,michalsenkyr/spark,chuckchen/spark,guoxiaolongzte/spark,joseph-torres/spark,WeichenXu123/spark,guoxiaolongzte/spark,tejasapatil/spark,ron8hu/spark,publicRoman/spark,aokolnychyi/spark,dongjoon-hyun/spark,vinodkc/spark,dongjoon-hyun/spark,shuangshuangwang/spark,dongjoon-hyun/spark,dbtsai/spark,taroplus/spark,cin/spark,darionyaphet/spark,srowen/spark,lxsmnv/spark,1haodian/spark,UndeadBaneGitHub/spark,aray/spark,poffuomo/spark,lvdongr/spark,kevinyu98/spark,zhouyejoe/spark,brad-kaiser/spark,bOOm-X/spark,mzl9039/spark,jkbradley/spark,bdrillard/spark,WeichenXu123/spark,haowu80s/spark,andrewor14/spark,michalsenkyr/spark,adrian-ionescu/apache-spark,brad-kaiser/spark,szhem/spark,patrick-nicholson/spark,caneGuy/spark,JerryLead/spark,ericvandenbergfb/spark,debugger87/spark,SHASHANKB/spark,eyalfa/spark,mdespriee/spark,markhamstra/spark,nchammas/spark,jiangxb1987/spark,janewangfb/spark,matthewfranglen/spark,spark-test/spark,kevinyu98/spark,lvdongr/spark,guoxiaolongzte/spark,debugger87/spark,SnappyDataInc/spark,SnappyDataInc/spark,brad-kaiser/spark,rednaxelafx/apache-spark,mahak/spark,narahari92/spark,liyichao/spark,cin/spark,wangmiao1981/spark,vinodkc/spark,lvdongr/spark,mike0sv/spark,Aegeaner/spark,pgandhi999/spark,markhamstra/spark,BryanCutler/spark,liyichao/spark,dhruve/spark,holdenk/spark,akopich/spark,mike0sv/spark,hvanhovell/spark,lxsmnv/spark,JoshRosen/spark,poffuomo/spark,panfengfeng/spark,bdrillard/spark,nilsgrabbert/spark,kissmetrics/spark,apache-spark-on-k8s/spark,andrewor14/spark,zero323/spark,JerryLead/spark,shubhamchopra/spark,SnappyDataInc/spark,tengpeng/spark,ueshin/apache-spark,stanzhai/spark,saturday-shi/spark,dongjoon-hyun/spark,alunarbeach/spark,rednaxelafx/apache-spark,Aegeaner/spark,akopich/spark,wzhfy/spark,Panos-Bletsos/spark-cost-model-optimizer,srowen/spark,pgandhi999/spark,UndeadBaneGitHub/spark,ioana-delaney/spark,aray/spark,szhem/spark,jiangxb1987/spark,mdespriee/spark,mzl9039/spark,wzhfy/spark,SHASHANKB/spark,big-pegasus/spark,BryanCutler/spark,mike0sv/spark,hhbyyh/spark,ioana-delaney/spark,map222/spark,byakuinss/spark,vinodkc/spark,tejasapatil/spark,taroplus/spark,ahnqirage/spark,wangyum/spark,nilsgrabbert/spark,HyukjinKwon/spark,metamx/spark,facaiy/spark,wangyum/spark,jiangxb1987/spark,tengpeng/spark,ericvandenbergfb/spark,1haodian/spark,goldmedal/spark,HyukjinKwon/spark,yanboliang/spark,debugger87/spark,byakuinss/spark,jkbradley/spark,saturday-shi/spark,stanzhai/spark,highfei2011/spark,JoshRosen/spark,janewangfb/spark,wangyum/spark,JerryLead/spark,highfei2011/spark,liutang123/spark,zzcclp/spark,szhem/spark,milliman/spark,mahak/spark,nlalevee/spark,dhruve/spark,liutang123/spark,kiszk/spark,alunarbeach/spark,tejasapatil/spark,cin/spark,goldmedal/spark,Panos-Bletsos/spark-cost-model-optimizer,someorz/spark,facaiy/spark,1haodian/spark,zhouyejoe/spark,narahari92/spark,Panos-Bletsos/spark-cost-model-optimizer,saltstar/spark,aokolnychyi/spark,aokolnychyi/spark,Aegeaner/spark,JoshRosen/spark,narahari92/spark,bdrillard/spark,shubhamchopra/spark,MLnick/spark,wangyum/spark,kevinyu98/spark,shubhamchopra/spark,Aegeaner/spark,jiangxb1987/spark,ptkool/spark,WindCanDie/spark,setjet/spark,poffuomo/spark,stanzhai/spark,SHASHANKB/spark,kiszk/spark,hhbyyh/spark,WindCanDie/spark,metamx/spark,sahilTakiar/spark,kimoonkim/spark,MLnick/spark,actuaryzhang/spark,icexelloss/spark,bdrillard/spark,bravo-zhang/spark,esi-mineset/spark,saltstar/spark,jlopezmalla/spark,spark-test/spark,adrian-ionescu/apache-spark,apache/spark,rednaxelafx/apache-spark,JerryLead/spark,witgo/spark,mzl9039/spark,LantaoJin/spark,vinodkc/spark,patrick-nicholson/spark,mahak/spark,WindCanDie/spark,bravo-zhang/spark,1haodian/spark,aray/spark,xflin/spark,LantaoJin/spark,aokolnychyi/spark,mahak/spark,kevinyu98/spark,shuangshuangwang/spark,tejasapatil/spark,goldmedal/spark,jrshust/spark,byakuinss/spark,narahari92/spark,skonto/spark,lvdongr/spark,ajaysaini725/spark,wzhfy/spark,panfengfeng/spark,ericvandenbergfb/spark,maropu/spark,xflin/spark,ahnqirage/spark,darionyaphet/spark,tejasapatil/spark,map222/spark,patrick-nicholson/spark,lxsmnv/spark,ptkool/spark,kiszk/spark,cloud-fan/spark,mike0sv/spark,byakuinss/spark,zero323/spark,kissmetrics/spark,andrewor14/spark,jkbradley/spark,ptkool/spark,facaiy/spark,bOOm-X/spark,milliman/spark,caneGuy/spark,ueshin/apache-spark,minixalpha/spark,techaddict/spark,ueshin/apache-spark,skonto/spark,zero323/spark,mdespriee/spark,michalsenkyr/spark,panfengfeng/spark,publicRoman/spark,alunarbeach/spark,huang1900/spark,joseph-torres/spark,akopich/spark,eyalfa/spark,sureshthalamati/spark,vinodkc/spark,stanzhai/spark,rikima/spark,mzl9039/spark,liyichao/spark,kiszk/spark,alunarbeach/spark,maropu/spark,jianran/spark,markhamstra/spark,map222/spark,nchammas/spark,sachintyagi22/spark,spark-test/spark,xflin/spark,darionyaphet/spark,actuaryzhang/spark,huang1900/spark,lvdongr/spark,poffuomo/spark,skonto/spark,saltstar/spark,caneGuy/spark,taroplus/spark,yanboliang/spark,haowu80s/spark,aray/spark,WindCanDie/spark,ConeyLiu/spark,HyukjinKwon/spark,zuotingbing/spark,holdenk/spark,nchammas/spark,ConeyLiu/spark,patrick-nicholson/spark,maropu/spark,HyukjinKwon/spark,janewangfb/spark,LantaoJin/spark,ahnqirage/spark,JoshRosen/spark,dbtsai/spark,shaneknapp/spark,setjet/spark,lxsmnv/spark,esi-mineset/spark,lxsmnv/spark,joseph-torres/spark,LantaoJin/spark,huang1900/spark,Panos-Bletsos/spark-cost-model-optimizer,dongjoon-hyun/spark,ibm-research-ireland/sparkoscope,taroplus/spark,saturday-shi/spark,pgandhi999/spark,cin/spark,saltstar/spark,jkbradley/spark,metamx/spark,HyukjinKwon/spark,sachintyagi22/spark,minixalpha/spark,gengliangwang/spark,minixalpha/spark,BryanCutler/spark,UndeadBaneGitHub/spark,aosagie/spark,zhouyejoe/spark,chuckchen/spark,jiangxb1987/spark,guoxiaolongzte/spark,sureshthalamati/spark,hhbyyh/spark,zuotingbing/spark,bOOm-X/spark,apache/spark,sachintyagi22/spark,adrian-ionescu/apache-spark,kiszk/spark,nchammas/spark,xuanyuanking/spark,minixalpha/spark,hvanhovell/spark,wzhfy/spark,actuaryzhang/spark,cin/spark,tengpeng/spark,kevinyu98/spark,nilsgrabbert/spark,ajaysaini725/spark,WindCanDie/spark,michalsenkyr/spark,techaddict/spark,JerryLead/spark,apache/spark,rikima/spark,bdrillard/spark,aosagie/spark,highfei2011/spark,ericvandenbergfb/spark,liutang123/spark,rednaxelafx/apache-spark,lvdongr/spark,goldmedal/spark,milliman/spark,JerryLead/spark,map222/spark,nlalevee/spark,WindCanDie/spark,ron8hu/spark,loneknightpy/spark,Aegeaner/spark,lxsmnv/spark,xflin/spark,SnappyDataInc/spark,actuaryzhang/spark,eyalfa/spark,aray/spark,haowu80s/spark,huang1900/spark,someorz/spark,milliman/spark,JoshRosen/spark,panfengfeng/spark,minixalpha/spark,patrick-nicholson/spark,vinodkc/spark,adrian-ionescu/apache-spark,caneGuy/spark,esi-mineset/spark,loneknightpy/spark,UndeadBaneGitHub/spark,dhruve/spark,szhem/spark,jrshust/spark,kimoonkim/spark,witgo/spark,xflin/spark,jrshust/spark,hvanhovell/spark,SnappyDataInc/spark,aosagie/spark,icexelloss/spark,loneknightpy/spark,mike0sv/spark,apache/spark,tengpeng/spark,gioenn/xSpark,UndeadBaneGitHub/spark,akopich/spark,SnappyDataInc/spark,debugger87/spark,esi-mineset/spark,ConeyLiu/spark,ajaysaini725/spark,mzl9039/spark,lxsmnv/spark,kimoonkim/spark,1haodian/spark,huang1900/spark,techaddict/spark,witgo/spark,loneknightpy/spark,spark-test/spark,janewangfb/spark,huang1900/spark,nchammas/spark,nlalevee/spark,loneknightpy/spark,kevinyu98/spark,eyalfa/spark,JoshRosen/spark,apache-spark-on-k8s/spark,spark-test/spark,caneGuy/spark,shubhamchopra/spark,ajaysaini725/spark,gengliangwang/spark,guoxiaolongzte/spark,szhem/spark,WeichenXu123/spark,ibm-research-ireland/sparkoscope,rezasafi/spark,jlopezmalla/spark,szhem/spark,gioenn/xSpark,adrian-ionescu/apache-spark,zero323/spark,highfei2011/spark,BryanCutler/spark,liyichao/spark,hhbyyh/spark,wangyum/spark,saltstar/spark,kiszk/spark,rezasafi/spark,dhruve/spark,milliman/spark,BryanCutler/spark,jrshust/spark,ibm-research-ireland/sparkoscope,cloud-fan/spark,actuaryzhang/spark,wangmiao1981/spark,jkbradley/spark,witgo/spark,brad-kaiser/spark,guoxiaolongzte/spark,jrshust/spark,aray/spark,rekhajoshm/spark,adrian-ionescu/apache-spark,kissmetrics/spark,rekhajoshm/spark,witgo/spark,alunarbeach/spark,cloud-fan/spark,gioenn/xSpark,dongjoon-hyun/spark,mdespriee/spark,1haodian/spark,saltstar/spark,poffuomo/spark,akopich/spark,liutang123/spark,liyichao/spark,kissmetrics/spark,mdespriee/spark,nlalevee/spark,tengpeng/spark,dbtsai/spark,poffuomo/spark,ptkool/spark,patrick-nicholson/spark,zero323/spark,facaiy/spark,big-pegasus/spark,hhbyyh/spark,ptkool/spark,facaiy/spark,janewangfb/spark,bravo-zhang/spark,mahak/spark,jianran/spark,cloud-fan/spark,rekhajoshm/spark,nilsgrabbert/spark,ddna1021/spark,shuangshuangwang/spark,MLnick/spark,techaddict/spark,nilsgrabbert/spark,tejasapatil/spark,zhouyejoe/spark,ddna1021/spark,zzcclp/spark,alunarbeach/spark,gioenn/xSpark,skonto/spark,apache-spark-on-k8s/spark,ron8hu/spark,publicRoman/spark,MLnick/spark,srowen/spark,eyalfa/spark,WeichenXu123/spark,ajaysaini725/spark,jlopezmalla/spark,gengliangwang/spark,kimoonkim/spark,darionyaphet/spark,bOOm-X/spark,rikima/spark,ioana-delaney/spark,JerryLead/spark,yanboliang/spark,shaneknapp/spark,rekhajoshm/spark,jkbradley/spark,rezasafi/spark,sachintyagi22/spark,icexelloss/spark,nchammas/spark,aosagie/spark,markhamstra/spark,srowen/spark,brad-kaiser/spark,debugger87/spark,skonto/spark,rekhajoshm/spark,ron8hu/spark,rikima/spark,shaneknapp/spark,SHASHANKB/spark,ericvandenbergfb/spark,darionyaphet/spark,ioana-delaney/spark