[SPARK-7339] [PYSPARK] PySpark shuffle spill memory sometimes are not correct
[SPARK-7339] [PYSPARK] PySpark shuffle spill memory sometimes are not correct

In PySpark we get memory used before and after spill, then use the difference of these two value as memorySpilled, but if the before value is small than after value, then we will get a negative value, but this scenario 0 value may be more reasonable.

Below is the result in HistoryServer we have tested:
Index	ID	Attempt	Status	Locality Level	Executor ID / Host	Launch Time	Duration	GC Time	Input Size / Records	Write Time	Shuffle Write Size / Records	Shuffle Spill (Memory)	Shuffle Spill (Disk)	Errors
0	0	0	SUCCESS	NODE_LOCAL	3 / vm119	2015/05/04 17:31:06	21 s	0.1 s	128.1 MB (hadoop) / 3237	70 ms	10.1 MB / 2529	0.0 B	5.7 MB
2	2	0	SUCCESS	NODE_LOCAL	1 / vm118	2015/05/04 17:31:06	22 s	89 ms	128.1 MB (hadoop) / 3205	0.1 s	10.1 MB / 2529	-1048576.0 B	5.9 MB
1	1	0	SUCCESS	NODE_LOCAL	2 / vm117	2015/05/04 17:31:06	22 s	0.1 s	128.1 MB (hadoop) / 3271	68 ms	10.1 MB / 2529	-1048576.0 B	5.6 MB
4	4	0	SUCCESS	NODE_LOCAL	2 / vm117	2015/05/04 17:31:06	22 s	0.1 s	128.1 MB (hadoop) / 3192	51 ms	10.1 MB / 2529	-1048576.0 B	5.9 MB
3	3	0	SUCCESS	NODE_LOCAL	3 / vm119	2015/05/04 17:31:06	22 s	0.1 s	128.1 MB (hadoop) / 3262	51 ms	10.1 MB / 2529	1024.0 KB	5.8 MB
5	5	0	SUCCESS	NODE_LOCAL	1 / vm118	2015/05/04 17:31:06	22 s	89 ms	128.1 MB (hadoop) / 3256	93 ms	10.1 MB / 2529	-1048576.0 B	5.7 MB

/cc davies

Author: linweizhong <linweizhong@huawei.com>

Closes #5887 from Sephiroth-Lin/spark-7339 and squashes the following commits:

9186c81 [linweizhong] Use max function to get a nonnegative value
d41672b [linweizhong] Update MemoryBytesSpilled when memorySpilled > 0

bOOm-X/spark,michalsenkyr/spark,sahilTakiar/spark,Panos-Bletsos/spark-cost-model-optimizer,kimoonkim/spark,aosagie/spark,kimoonkim/spark,bOOm-X/spark,tengpeng/spark,zuotingbing/spark,ConeyLiu/spark,ahnqirage/spark,szhem/spark,nilsgrabbert/spark,aray/spark,spark-test/spark,nlalevee/spark,BryanCutler/spark,zuotingbing/spark,publicRoman/spark,JoshRosen/spark,pgandhi999/spark,rekhajoshm/spark,ibm-research-ireland/sparkoscope,byakuinss/spark,szhem/spark,tengpeng/spark,ConeyLiu/spark,tejasapatil/spark,icexelloss/spark,andrewor14/spark,hvanhovell/spark,holdenk/spark,huang1900/spark,Panos-Bletsos/spark-cost-model-optimizer,maropu/spark,liutang123/spark,wangmiao1981/spark,metamx/spark,narahari92/spark,actuaryzhang/spark,Aegeaner/spark,esi-mineset/spark,stanzhai/spark,kiszk/spark,nlalevee/spark,skonto/spark,techaddict/spark,gioenn/xSpark,map222/spark,metamx/spark,ibm-research-ireland/sparkoscope,aray/spark,big-pegasus/spark,apache/spark,big-pegasus/spark,jkbradley/spark,tejasapatil/spark,brad-kaiser/spark,ConeyLiu/spark,kimoonkim/spark,yanboliang/spark,wangmiao1981/spark,janewangfb/spark,skonto/spark,icexelloss/spark,sureshthalamati/spark,vinodkc/spark,panfengfeng/spark,goldmedal/spark,dbtsai/spark,goldmedal/spark,pgandhi999/spark,zhouyejoe/spark,ddna1021/spark,SHASHANKB/spark,nlalevee/spark,gengliangwang/spark,pgandhi999/spark,hvanhovell/spark,taroplus/spark,aokolnychyi/spark,brad-kaiser/spark,jlopezmalla/spark,witgo/spark,wangmiao1981/spark,rednaxelafx/apache-spark,goldmedal/spark,facaiy/spark,rekhajoshm/spark,ptkool/spark,eyalfa/spark,jlopezmalla/spark,1haodian/spark,jiangxb1987/spark,jrshust/spark,dhruve/spark,michalsenkyr/spark,byakuinss/spark,kiszk/spark,jiangxb1987/spark,darionyaphet/spark,ibm-research-ireland/sparkoscope,wangyum/spark,hhbyyh/spark,ahnqirage/spark,zero323/spark,wangmiao1981/spark,kissmetrics/spark,highfei2011/spark,tejasapatil/spark,guoxiaolongzte/spark,setjet/spark,liutang123/spark,ddna1021/spark,mike0sv/spark,guoxiaolongzte/spark,Panos-Bletsos/spark-cost-model-optimizer,maropu/spark,lvdongr/spark,minixalpha/spark,techaddict/spark,big-pegasus/spark,ericvandenbergfb/spark,rekhajoshm/spark,ron8hu/spark,HyukjinKwon/spark,akopich/spark,ConeyLiu/spark,andrewor14/spark,alunarbeach/spark,zzcclp/spark,dhruve/spark,markhamstra/spark,Aegeaner/spark,esi-mineset/spark,pgandhi999/spark,nchammas/spark,esi-mineset/spark,rednaxelafx/apache-spark,mahak/spark,nilsgrabbert/spark,ueshin/apache-spark,wzhfy/spark,ericvandenbergfb/spark,tengpeng/spark,lxsmnv/spark,setjet/spark,aosagie/spark,bravo-zhang/spark,michalsenkyr/spark,WindCanDie/spark,taroplus/spark,maropu/spark,Aegeaner/spark,apache/spark,yanboliang/spark,ajaysaini725/spark,ptkool/spark,highfei2011/spark,ptkool/spark,MLnick/spark,eyalfa/spark,goldmedal/spark,apache-spark-on-k8s/spark,rikima/spark,rednaxelafx/apache-spark,srowen/spark,shuangshuangwang/spark,ahnqirage/spark,jiangxb1987/spark,shubhamchopra/spark,panfengfeng/spark,BryanCutler/spark,holdenk/spark,ueshin/apache-spark,BryanCutler/spark,apache-spark-on-k8s/spark,joseph-torres/spark,janewangfb/spark,taroplus/spark,mike0sv/spark,liyichao/spark,gengliangwang/spark,bdrillard/spark,akopich/spark,xflin/spark,poffuomo/spark,debugger87/spark,dbtsai/spark,nchammas/spark,hhbyyh/spark,nilsgrabbert/spark,ron8hu/spark,guoxiaolongzte/spark,tengpeng/spark,xflin/spark,shaneknapp/spark,ioana-delaney/spark,sahilTakiar/spark,narahari92/spark,ajaysaini725/spark,ajaysaini725/spark,cin/spark,Aegeaner/spark,hvanhovell/spark,joseph-torres/spark,icexelloss/spark,witgo/spark,patrick-nicholson/spark,maropu/spark,tejasapatil/spark,dbtsai/spark,skonto/spark,JoshRosen/spark,sahilTakiar/spark,shubhamchopra/spark,brad-kaiser/spark,bOOm-X/spark,SnappyDataInc/spark,witgo/spark,shuangshuangwang/spark,aokolnychyi/spark,panfengfeng/spark,alunarbeach/spark,WindCanDie/spark,mike0sv/spark,dhruve/spark,MLnick/spark,tejasapatil/spark,hvanhovell/spark,hvanhovell/spark,haowu80s/spark,minixalpha/spark,chuckchen/spark,zero323/spark,kiszk/spark,jlopezmalla/spark,lxsmnv/spark,gioenn/xSpark,holdenk/spark,mzl9039/spark,SHASHANKB/spark,shaneknapp/spark,patrick-nicholson/spark,apache/spark,techaddict/spark,publicRoman/spark,sahilTakiar/spark,cloud-fan/spark,janewangfb/spark,jiangxb1987/spark,metamx/spark,pgandhi999/spark,apache-spark-on-k8s/spark,kissmetrics/spark,bravo-zhang/spark,apache-spark-on-k8s/spark,MLnick/spark,jianran/spark,xuanyuanking/spark,jianran/spark,pgandhi999/spark,ahnqirage/spark,goldmedal/spark,alunarbeach/spark,zuotingbing/spark,lxsmnv/spark,shaneknapp/spark,milliman/spark,huang1900/spark,1haodian/spark,saturday-shi/spark,ConeyLiu/spark,HyukjinKwon/spark,szhem/spark,JerryLead/spark,LantaoJin/spark,WeichenXu123/spark,bOOm-X/spark,wzhfy/spark,ueshin/apache-spark,milliman/spark,shubhamchopra/spark,joseph-torres/spark,aosagie/spark,saturday-shi/spark,rikima/spark,SnappyDataInc/spark,gengliangwang/spark,ibm-research-ireland/sparkoscope,pgandhi999/spark,narahari92/spark,markhamstra/spark,aokolnychyi/spark,someorz/spark,sureshthalamati/spark,vinodkc/spark,someorz/spark,ddna1021/spark,ericvandenbergfb/spark,janewangfb/spark,ConeyLiu/spark,jkbradley/spark,ddna1021/spark,guoxiaolongzte/spark,zzcclp/spark,panfengfeng/spark,SnappyDataInc/spark,wangmiao1981/spark,rezasafi/spark,chuckchen/spark,dbtsai/spark,lxsmnv/spark,cloud-fan/spark,wangmiao1981/spark,nchammas/spark,wangyum/spark,dongjoon-hyun/spark,chuckchen/spark,stanzhai/spark,andrewor14/spark,wangmiao1981/spark,sachintyagi22/spark,nlalevee/spark,ahnqirage/spark,JerryLead/spark,1haodian/spark,dongjoon-hyun/spark,nchammas/spark,Aegeaner/spark,eyalfa/spark,szhem/spark,shuangshuangwang/spark,cin/spark,janewangfb/spark,milliman/spark,actuaryzhang/spark,ioana-delaney/spark,nchammas/spark,lvdongr/spark,SnappyDataInc/spark,HyukjinKwon/spark,dongjoon-hyun/spark,vinodkc/spark,shuangshuangwang/spark,1haodian/spark,jrshust/spark,jrshust/spark,UndeadBaneGitHub/spark,big-pegasus/spark,rekhajoshm/spark,xflin/spark,akopich/spark,ddna1021/spark,loneknightpy/spark,aray/spark,loneknightpy/spark,cin/spark,caneGuy/spark,bravo-zhang/spark,tejasapatil/spark,rednaxelafx/apache-spark,someorz/spark,huang1900/spark,WeichenXu123/spark,saltstar/spark,dbtsai/spark,saltstar/spark,aray/spark,ibm-research-ireland/sparkoscope,lvdongr/spark,publicRoman/spark,1haodian/spark,zzcclp/spark,saltstar/spark,maropu/spark,kissmetrics/spark,guoxiaolongzte/spark,saltstar/spark,liutang123/spark,zhouyejoe/spark,ueshin/apache-spark,darionyaphet/spark,loneknightpy/spark,JerryLead/spark,mzl9039/spark,big-pegasus/spark,jrshust/spark,wzhfy/spark,1haodian/spark,wzhfy/spark,WeichenXu123/spark,nchammas/spark,haowu80s/spark,michalsenkyr/spark,1haodian/spark,byakuinss/spark,szhem/spark,xflin/spark,mdespriee/spark,minixalpha/spark,pronix/spark,debugger87/spark,caneGuy/spark,gioenn/xSpark,ron8hu/spark,poffuomo/spark,rezasafi/spark,gioenn/xSpark,ron8hu/spark,vinodkc/spark,saturday-shi/spark,ddna1021/spark,narahari92/spark,kissmetrics/spark,yanboliang/spark,nchammas/spark,metamx/spark,ptkool/spark,byakuinss/spark,mahak/spark,liyichao/spark,lvdongr/spark,HyukjinKwon/spark,LantaoJin/spark,apache/spark,metamx/spark,ibm-research-ireland/sparkoscope,jkbradley/spark,apache/spark,vinodkc/spark,skonto/spark,ioana-delaney/spark,akopich/spark,lvdongr/spark,ajaysaini725/spark,kevinyu98/spark,setjet/spark,esi-mineset/spark,someorz/spark,adrian-ionescu/apache-spark,joseph-torres/spark,maropu/spark,hvanhovell/spark,nlalevee/spark,zero323/spark,huang1900/spark,chuckchen/spark,sachintyagi22/spark,ueshin/apache-spark,shubhamchopra/spark,aokolnychyi/spark,mdespriee/spark,byakuinss/spark,gioenn/xSpark,mzl9039/spark,adrian-ionescu/apache-spark,ioana-delaney/spark,zzcclp/spark,wangyum/spark,xuanyuanking/spark,brad-kaiser/spark,ericvandenbergfb/spark,caneGuy/spark,ericvandenbergfb/spark,aray/spark,jiangxb1987/spark,saltstar/spark,sachintyagi22/spark,loneknightpy/spark,HyukjinKwon/spark,pronix/spark,srowen/spark,huang1900/spark,caneGuy/spark,mzl9039/spark,JerryLead/spark,jkbradley/spark,shubhamchopra/spark,jiangxb1987/spark,maropu/spark,WindCanDie/spark,caneGuy/spark,dongjoon-hyun/spark,narahari92/spark,zero323/spark,liyichao/spark,ajaysaini725/spark,liyichao/spark,spark-test/spark,UndeadBaneGitHub/spark,zero323/spark,BryanCutler/spark,rikima/spark,mzl9039/spark,facaiy/spark,kevinyu98/spark,apache-spark-on-k8s/spark,mdespriee/spark,huang1900/spark,jlopezmalla/spark,SnappyDataInc/spark,WindCanDie/spark,dhruve/spark,wangyum/spark,actuaryzhang/spark,srowen/spark,rezasafi/spark,skonto/spark,bravo-zhang/spark,ericvandenbergfb/spark,loneknightpy/spark,map222/spark,aosagie/spark,aosagie/spark,kevinyu98/spark,akopich/spark,akopich/spark,sahilTakiar/spark,srowen/spark,aray/spark,JoshRosen/spark,szhem/spark,saturday-shi/spark,xflin/spark,rezasafi/spark,poffuomo/spark,caneGuy/spark,LantaoJin/spark,tengpeng/spark,holdenk/spark,alunarbeach/spark,spark-test/spark,nilsgrabbert/spark,JoshRosen/spark,andrewor14/spark,panfengfeng/spark,ericvandenbergfb/spark,taroplus/spark,taroplus/spark,sahilTakiar/spark,UndeadBaneGitHub/spark,actuaryzhang/spark,gioenn/xSpark,liutang123/spark,minixalpha/spark,ron8hu/spark,guoxiaolongzte/spark,bdrillard/spark,saturday-shi/spark,patrick-nicholson/spark,liutang123/spark,andrewor14/spark,JerryLead/spark,shubhamchopra/spark,MLnick/spark,highfei2011/spark,poffuomo/spark,darionyaphet/spark,liutang123/spark,shaneknapp/spark,brad-kaiser/spark,saltstar/spark,saturday-shi/spark,someorz/spark,setjet/spark,patrick-nicholson/spark,nlalevee/spark,mike0sv/spark,markhamstra/spark,stanzhai/spark,kiszk/spark,mahak/spark,zuotingbing/spark,metamx/spark,kimoonkim/spark,sureshthalamati/spark,ddna1021/spark,gengliangwang/spark,dhruve/spark,shuangshuangwang/spark,markhamstra/spark,Panos-Bletsos/spark-cost-model-optimizer,ConeyLiu/spark,JoshRosen/spark,rednaxelafx/apache-spark,sachintyagi22/spark,goldmedal/spark,yanboliang/spark,yanboliang/spark,bOOm-X/spark,hhbyyh/spark,ajaysaini725/spark,setjet/spark,kevinyu98/spark,milliman/spark,pronix/spark,zhouyejoe/spark,panfengfeng/spark,icexelloss/spark,someorz/spark,chuckchen/spark,dongjoon-hyun/spark,adrian-ionescu/apache-spark,darionyaphet/spark,mdespriee/spark,SnappyDataInc/spark,eyalfa/spark,spark-test/spark,gengliangwang/spark,taroplus/spark,wangyum/spark,SHASHANKB/spark,WeichenXu123/spark,wangyum/spark,zhouyejoe/spark,kiszk/spark,hvanhovell/spark,gengliangwang/spark,darionyaphet/spark,markhamstra/spark,milliman/spark,holdenk/spark,xflin/spark,mike0sv/spark,actuaryzhang/spark,narahari92/spark,MLnick/spark,big-pegasus/spark,adrian-ionescu/apache-spark,hhbyyh/spark,mahak/spark,brad-kaiser/spark,jlopezmalla/spark,WeichenXu123/spark,setjet/spark,ioana-delaney/spark,rikima/spark,dongjoon-hyun/spark,cin/spark,highfei2011/spark,dhruve/spark,darionyaphet/spark,cloud-fan/spark,holdenk/spark,joseph-torres/spark,ahnqirage/spark,nilsgrabbert/spark,sureshthalamati/spark,facaiy/spark,apache/spark,yanboliang/spark,big-pegasus/spark,sureshthalamati/spark,rednaxelafx/apache-spark,holdenk/spark,mdespriee/spark,kimoonkim/spark,sachintyagi22/spark,alunarbeach/spark,byakuinss/spark,pronix/spark,sureshthalamati/spark,SnappyDataInc/spark,alunarbeach/spark,srowen/spark,lxsmnv/spark,kevinyu98/spark,xuanyuanking/spark,tengpeng/spark,janewangfb/spark,WeichenXu123/spark,map222/spark,WindCanDie/spark,debugger87/spark,zero323/spark,ahnqirage/spark,ptkool/spark,gioenn/xSpark,loneknightpy/spark,jianran/spark,jkbradley/spark,patrick-nicholson/spark,stanzhai/spark,poffuomo/spark,kiszk/spark,cloud-fan/spark,highfei2011/spark,brad-kaiser/spark,bdrillard/spark,zuotingbing/spark,techaddict/spark,spark-test/spark,patrick-nicholson/spark,skonto/spark,dhruve/spark,MLnick/spark,bOOm-X/spark,taroplus/spark,icexelloss/spark,wzhfy/spark,skonto/spark,markhamstra/spark,Panos-Bletsos/spark-cost-model-optimizer,mzl9039/spark,publicRoman/spark,zzcclp/spark,jianran/spark,zuotingbing/spark,JoshRosen/spark,dbtsai/spark,facaiy/spark,esi-mineset/spark,Aegeaner/spark,facaiy/spark,shuangshuangwang/spark,mzl9039/spark,SHASHANKB/spark,haowu80s/spark,Panos-Bletsos/spark-cost-model-optimizer,xuanyuanking/spark,patrick-nicholson/spark,gengliangwang/spark,jrshust/spark,debugger87/spark,witgo/spark,xuanyuanking/spark,andrewor14/spark,jlopezmalla/spark,jianran/spark,panfengfeng/spark,tengpeng/spark,icexelloss/spark,rekhajoshm/spark,pronix/spark,dbtsai/spark,ptkool/spark,jianran/spark,kevinyu98/spark,minixalpha/spark,chuckchen/spark,sureshthalamati/spark,actuaryzhang/spark,liyichao/spark,stanzhai/spark,facaiy/spark,kissmetrics/spark,aosagie/spark,ueshin/apache-spark,rezasafi/spark,rikima/spark,michalsenkyr/spark,bravo-zhang/spark,BryanCutler/spark,kevinyu98/spark,rekhajoshm/spark,stanzhai/spark,map222/spark,nlalevee/spark,SHASHANKB/spark,UndeadBaneGitHub/spark,janewangfb/spark,liyichao/spark,ioana-delaney/spark,lvdongr/spark,UndeadBaneGitHub/spark,haowu80s/spark,haowu80s/spark,adrian-ionescu/apache-spark,metamx/spark,WindCanDie/spark,haowu80s/spark,mahak/spark,cin/spark,shaneknapp/spark,lxsmnv/spark,spark-test/spark,bdrillard/spark,stanzhai/spark,shaneknapp/spark,Aegeaner/spark,michalsenkyr/spark,publicRoman/spark,lxsmnv/spark,poffuomo/spark,pronix/spark,sachintyagi22/spark,JerryLead/spark,rekhajoshm/spark,apache/spark,bdrillard/spark,cloud-fan/spark,hhbyyh/spark,saltstar/spark,srowen/spark,mahak/spark,alunarbeach/spark,map222/spark,michalsenkyr/spark,actuaryzhang/spark,vinodkc/spark,jlopezmalla/spark,UndeadBaneGitHub/spark,witgo/spark,map222/spark,zhouyejoe/spark,wzhfy/spark,vinodkc/spark,setjet/spark,icexelloss/spark,huang1900/spark,zero323/spark,shuangshuangwang/spark,techaddict/spark,rikima/spark,hhbyyh/spark,aokolnychyi/spark,mike0sv/spark,minixalpha/spark,witgo/spark,UndeadBaneGitHub/spark,caneGuy/spark,tejasapatil/spark,publicRoman/spark,aokolnychyi/spark,rezasafi/spark,wangyum/spark,bravo-zhang/spark,ajaysaini725/spark,rednaxelafx/apache-spark,facaiy/spark,kissmetrics/spark,xflin/spark,ibm-research-ireland/sparkoscope,HyukjinKwon/spark,zhouyejoe/spark,jrshust/spark,darionyaphet/spark,witgo/spark,goldmedal/spark,eyalfa/spark,debugger87/spark,highfei2011/spark,xuanyuanking/spark,mike0sv/spark,nilsgrabbert/spark,zzcclp/spark,publicRoman/spark,LantaoJin/spark,kissmetrics/spark,jkbradley/spark,adrian-ionescu/apache-spark,kiszk/spark,sachintyagi22/spark,someorz/spark,kimoonkim/spark,aray/spark,mdespriee/spark,byakuinss/spark,Panos-Bletsos/spark-cost-model-optimizer,wzhfy/spark,esi-mineset/spark,dongjoon-hyun/spark,zhouyejoe/spark,jianran/spark,shubhamchopra/spark,cloud-fan/spark,MLnick/spark,debugger87/spark,adrian-ionescu/apache-spark,ron8hu/spark,spark-test/spark,jiangxb1987/spark,JoshRosen/spark,ron8hu/spark,shaneknapp/spark,srowen/spark,chuckchen/spark,aosagie/spark,map222/spark,bdrillard/spark,HyukjinKwon/spark,JerryLead/spark,techaddict/spark,akopich/spark,SHASHANKB/spark,loneknightpy/spark,jrshust/spark,milliman/spark,eyalfa/spark,WeichenXu123/spark,hhbyyh/spark,zzcclp/spark,yanboliang/spark,ueshin/apache-spark,apache-spark-on-k8s/spark,markhamstra/spark,apache-spark-on-k8s/spark,bdrillard/spark,rezasafi/spark,debugger87/spark,techaddict/spark,LantaoJin/spark,saturday-shi/spark,cin/spark,lvdongr/spark,cin/spark,highfei2011/spark,joseph-torres/spark,LantaoJin/spark,SHASHANKB/spark,milliman/spark,joseph-torres/spark,eyalfa/spark,liutang123/spark,esi-mineset/spark,nilsgrabbert/spark,jkbradley/spark,ioana-delaney/spark,sahilTakiar/spark,ptkool/spark,poffuomo/spark,WindCanDie/spark,xuanyuanking/spark,minixalpha/spark,guoxiaolongzte/spark,cloud-fan/spark,matthewfranglen/spark,BryanCutler/spark,bOOm-X/spark,liyichao/spark,BryanCutler/spark,szhem/spark,mahak/spark,mdespriee/spark,kimoonkim/spark,narahari92/spark,rikima/spark,bravo-zhang/spark,aokolnychyi/spark,zuotingbing/spark,andrewor14/spark,LantaoJin/spark