[SPARK-9805] [MLLIB] [PYTHON] [STREAMING] Added _eventually for ml streaming pyspark tests
[SPARK-9805] [MLLIB] [PYTHON] [STREAMING] Added _eventually for ml streaming pyspark tests

Recently, PySpark ML streaming tests have been flaky, most likely because of the batches not being processed in time.  Proposal: Replace the use of _ssc_wait (which waits for a fixed amount of time) with a method which waits for a fixed amount of time but can terminate early based on a termination condition method.  With this, we can extend the waiting period (to make tests less flaky) but also stop early when possible (making tests faster on average, which I verified locally).

CC: mengxr tdas freeman-lab

Author: Joseph K. Bradley <joseph@databricks.com>

Closes #8087 from jkbradley/streaming-ml-tests.

kiszk/spark,bravo-zhang/spark,spark-test/spark,bravo-zhang/spark,shuangshuangwang/spark,szhem/spark,mdespriee/spark,nchammas/spark,stanzhai/spark,saltstar/spark,wangmiao1981/spark,Aegeaner/spark,srowen/spark,adrian-ionescu/apache-spark,nlalevee/spark,dhruve/spark,dongjoon-hyun/spark,nchammas/spark,nlalevee/spark,skonto/spark,taroplus/spark,rekhajoshm/spark,rikima/spark,zzcclp/spark,brad-kaiser/spark,nchammas/spark,holdenk/spark,icexelloss/spark,aokolnychyi/spark,saturday-shi/spark,chuckchen/spark,liutang123/spark,aosagie/spark,debugger87/spark,witgo/spark,JoshRosen/spark,tengpeng/spark,andrewor14/spark,someorz/spark,aokolnychyi/spark,cin/spark,HyukjinKwon/spark,caneGuy/spark,gengliangwang/spark,shaneknapp/spark,xflin/spark,ueshin/apache-spark,aosagie/spark,jlopezmalla/spark,BryanCutler/spark,nchammas/spark,bOOm-X/spark,publicRoman/spark,wzhfy/spark,ConeyLiu/spark,Aegeaner/spark,skonto/spark,markhamstra/spark,panfengfeng/spark,ConeyLiu/spark,highfei2011/spark,patrick-nicholson/spark,shaneknapp/spark,rekhajoshm/spark,hhbyyh/spark,lvdongr/spark,JerryLead/spark,someorz/spark,hhbyyh/spark,jkbradley/spark,maropu/spark,ptkool/spark,WeichenXu123/spark,jiangxb1987/spark,taroplus/spark,dongjoon-hyun/spark,metamx/spark,xuanyuanking/spark,liutang123/spark,shaneknapp/spark,someorz/spark,minixalpha/spark,icexelloss/spark,aokolnychyi/spark,apache/spark,WeichenXu123/spark,stanzhai/spark,sachintyagi22/spark,SHASHANKB/spark,zzcclp/spark,dongjoon-hyun/spark,pgandhi999/spark,lxsmnv/spark,liyichao/spark,highfei2011/spark,bravo-zhang/spark,ericvandenbergfb/spark,markhamstra/spark,saltstar/spark,dbtsai/spark,panfengfeng/spark,bdrillard/spark,aray/spark,guoxiaolongzte/spark,alunarbeach/spark,xflin/spark,milliman/spark,andrewor14/spark,LantaoJin/spark,jiangxb1987/spark,zuotingbing/spark,patrick-nicholson/spark,pgandhi999/spark,caneGuy/spark,bOOm-X/spark,ahnqirage/spark,ioana-delaney/spark,joseph-torres/spark,UndeadBaneGitHub/spark,huang1900/spark,SHASHANKB/spark,dongjoon-hyun/spark,debugger87/spark,big-pegasus/spark,tengpeng/spark,vinodkc/spark,milliman/spark,sahilTakiar/spark,ahnqirage/spark,hhbyyh/spark,BryanCutler/spark,ConeyLiu/spark,BryanCutler/spark,michalsenkyr/spark,lvdongr/spark,mzl9039/spark,sachintyagi22/spark,patrick-nicholson/spark,shubhamchopra/spark,actuaryzhang/spark,kevinyu98/spark,lxsmnv/spark,shubhamchopra/spark,zhouyejoe/spark,hvanhovell/spark,sachintyagi22/spark,liutang123/spark,sureshthalamati/spark,WeichenXu123/spark,ibm-research-ireland/sparkoscope,darionyaphet/spark,kevinyu98/spark,JerryLead/spark,ajaysaini725/spark,bOOm-X/spark,akopich/spark,1haodian/spark,HyukjinKwon/spark,gengliangwang/spark,ptkool/spark,spark-test/spark,zuotingbing/spark,gengliangwang/spark,actuaryzhang/spark,zero323/spark,tengpeng/spark,jianran/spark,ConeyLiu/spark,szhem/spark,aosagie/spark,sureshthalamati/spark,skonto/spark,WeichenXu123/spark,publicRoman/spark,lxsmnv/spark,eyalfa/spark,gengliangwang/spark,ioana-delaney/spark,highfei2011/spark,gioenn/xSpark,vinodkc/spark,BryanCutler/spark,Panos-Bletsos/spark-cost-model-optimizer,maropu/spark,someorz/spark,dongjoon-hyun/spark,janewangfb/spark,mdespriee/spark,loneknightpy/spark,ueshin/apache-spark,SnappyDataInc/spark,map222/spark,witgo/spark,zero323/spark,xflin/spark,sahilTakiar/spark,JoshRosen/spark,dhruve/spark,zzcclp/spark,cloud-fan/spark,chuckchen/spark,huang1900/spark,UndeadBaneGitHub/spark,tejasapatil/spark,esi-mineset/spark,saturday-shi/spark,narahari92/spark,gioenn/xSpark,zzcclp/spark,bOOm-X/spark,rezasafi/spark,haowu80s/spark,SHASHANKB/spark,someorz/spark,shaneknapp/spark,apache-spark-on-k8s/spark,BryanCutler/spark,ibm-research-ireland/sparkoscope,saturday-shi/spark,ron8hu/spark,jkbradley/spark,ioana-delaney/spark,esi-mineset/spark,rekhajoshm/spark,witgo/spark,eyalfa/spark,bOOm-X/spark,metamx/spark,ddna1021/spark,pronix/spark,guoxiaolongzte/spark,wangmiao1981/spark,haowu80s/spark,guoxiaolongzte/spark,xflin/spark,mahak/spark,darionyaphet/spark,nlalevee/spark,pronix/spark,sureshthalamati/spark,metamx/spark,setjet/spark,publicRoman/spark,andrewor14/spark,caneGuy/spark,janewangfb/spark,sahilTakiar/spark,narahari92/spark,zhouyejoe/spark,wangyum/spark,alunarbeach/spark,SnappyDataInc/spark,xuanyuanking/spark,rezasafi/spark,yanboliang/spark,jianran/spark,kimoonkim/spark,xuanyuanking/spark,JerryLead/spark,eyalfa/spark,gengliangwang/spark,shaneknapp/spark,aosagie/spark,minixalpha/spark,kimoonkim/spark,mike0sv/spark,goldmedal/spark,jiangxb1987/spark,shuangshuangwang/spark,lxsmnv/spark,loneknightpy/spark,brad-kaiser/spark,wangmiao1981/spark,sachintyagi22/spark,mdespriee/spark,wzhfy/spark,zuotingbing/spark,aosagie/spark,WindCanDie/spark,vinodkc/spark,LantaoJin/spark,JoshRosen/spark,ajaysaini725/spark,spark-test/spark,ajaysaini725/spark,jlopezmalla/spark,joseph-torres/spark,liyichao/spark,pgandhi999/spark,pgandhi999/spark,chuckchen/spark,markhamstra/spark,map222/spark,sureshthalamati/spark,yanboliang/spark,mdespriee/spark,zero323/spark,pgandhi999/spark,icexelloss/spark,darionyaphet/spark,apache/spark,alunarbeach/spark,szhem/spark,taroplus/spark,yanboliang/spark,esi-mineset/spark,Aegeaner/spark,BryanCutler/spark,highfei2011/spark,bravo-zhang/spark,icexelloss/spark,kevinyu98/spark,joseph-torres/spark,panfengfeng/spark,stanzhai/spark,wangmiao1981/spark,ConeyLiu/spark,actuaryzhang/spark,mahak/spark,witgo/spark,SHASHANKB/spark,goldmedal/spark,zero323/spark,apache/spark,1haodian/spark,rednaxelafx/apache-spark,andrewor14/spark,milliman/spark,narahari92/spark,Panos-Bletsos/spark-cost-model-optimizer,liyichao/spark,ahnqirage/spark,ron8hu/spark,nilsgrabbert/spark,apache-spark-on-k8s/spark,kissmetrics/spark,rezasafi/spark,sahilTakiar/spark,maropu/spark,witgo/spark,chuckchen/spark,caneGuy/spark,holdenk/spark,taroplus/spark,haowu80s/spark,srowen/spark,gioenn/xSpark,guoxiaolongzte/spark,SnappyDataInc/spark,icexelloss/spark,szhem/spark,jlopezmalla/spark,bdrillard/spark,apache/spark,vinodkc/spark,ioana-delaney/spark,zhouyejoe/spark,ptkool/spark,markhamstra/spark,wangyum/spark,shuangshuangwang/spark,zero323/spark,tejasapatil/spark,ddna1021/spark,actuaryzhang/spark,pronix/spark,1haodian/spark,poffuomo/spark,kimoonkim/spark,eyalfa/spark,joseph-torres/spark,jianran/spark,poffuomo/spark,kiszk/spark,matthewfranglen/spark,narahari92/spark,loneknightpy/spark,apache/spark,cloud-fan/spark,brad-kaiser/spark,mahak/spark,apache-spark-on-k8s/spark,caneGuy/spark,ibm-research-ireland/sparkoscope,nlalevee/spark,szhem/spark,kissmetrics/spark,aokolnychyi/spark,haowu80s/spark,Aegeaner/spark,maropu/spark,ddna1021/spark,akopich/spark,kissmetrics/spark,ron8hu/spark,ajaysaini725/spark,shaneknapp/spark,wzhfy/spark,lvdongr/spark,publicRoman/spark,taroplus/spark,poffuomo/spark,cloud-fan/spark,1haodian/spark,akopich/spark,big-pegasus/spark,jkbradley/spark,ericvandenbergfb/spark,pronix/spark,brad-kaiser/spark,saltstar/spark,bOOm-X/spark,ptkool/spark,shuangshuangwang/spark,kiszk/spark,WeichenXu123/spark,brad-kaiser/spark,liutang123/spark,ibm-research-ireland/sparkoscope,jkbradley/spark,wangyum/spark,janewangfb/spark,ptkool/spark,facaiy/spark,liutang123/spark,huang1900/spark,ericvandenbergfb/spark,cin/spark,zero323/spark,mdespriee/spark,aray/spark,srowen/spark,gengliangwang/spark,kimoonkim/spark,ibm-research-ireland/sparkoscope,wangmiao1981/spark,facaiy/spark,liyichao/spark,ioana-delaney/spark,michalsenkyr/spark,saturday-shi/spark,zero323/spark,apache-spark-on-k8s/spark,eyalfa/spark,jrshust/spark,SnappyDataInc/spark,SHASHANKB/spark,LantaoJin/spark,witgo/spark,taroplus/spark,rekhajoshm/spark,hvanhovell/spark,WindCanDie/spark,JerryLead/spark,big-pegasus/spark,nilsgrabbert/spark,dongjoon-hyun/spark,kevinyu98/spark,apache/spark,kevinyu98/spark,Panos-Bletsos/spark-cost-model-optimizer,brad-kaiser/spark,darionyaphet/spark,wangyum/spark,holdenk/spark,jrshust/spark,mzl9039/spark,mzl9039/spark,nchammas/spark,byakuinss/spark,sahilTakiar/spark,michalsenkyr/spark,ahnqirage/spark,minixalpha/spark,bravo-zhang/spark,poffuomo/spark,MLnick/spark,markhamstra/spark,wzhfy/spark,narahari92/spark,darionyaphet/spark,ddna1021/spark,mike0sv/spark,mike0sv/spark,narahari92/spark,dbtsai/spark,ahnqirage/spark,nlalevee/spark,alunarbeach/spark,holdenk/spark,Aegeaner/spark,big-pegasus/spark,aray/spark,zhouyejoe/spark,gioenn/xSpark,shubhamchopra/spark,kissmetrics/spark,michalsenkyr/spark,hvanhovell/spark,setjet/spark,wangyum/spark,chuckchen/spark,LantaoJin/spark,pgandhi999/spark,ibm-research-ireland/sparkoscope,dbtsai/spark,ericvandenbergfb/spark,guoxiaolongzte/spark,MLnick/spark,ajaysaini725/spark,ddna1021/spark,lvdongr/spark,techaddict/spark,sureshthalamati/spark,xuanyuanking/spark,wangmiao1981/spark,techaddict/spark,zhouyejoe/spark,WindCanDie/spark,actuaryzhang/spark,ConeyLiu/spark,skonto/spark,byakuinss/spark,rednaxelafx/apache-spark,loneknightpy/spark,debugger87/spark,byakuinss/spark,wzhfy/spark,eyalfa/spark,kiszk/spark,hvanhovell/spark,markhamstra/spark,wzhfy/spark,someorz/spark,huang1900/spark,chuckchen/spark,xflin/spark,akopich/spark,gioenn/xSpark,cin/spark,kissmetrics/spark,guoxiaolongzte/spark,cloud-fan/spark,zzcclp/spark,stanzhai/spark,SHASHANKB/spark,ahnqirage/spark,mdespriee/spark,byakuinss/spark,yanboliang/spark,SnappyDataInc/spark,bravo-zhang/spark,WeichenXu123/spark,aosagie/spark,aray/spark,panfengfeng/spark,sureshthalamati/spark,LantaoJin/spark,rednaxelafx/apache-spark,holdenk/spark,rezasafi/spark,spark-test/spark,dhruve/spark,HyukjinKwon/spark,shaneknapp/spark,rednaxelafx/apache-spark,jlopezmalla/spark,lxsmnv/spark,panfengfeng/spark,mahak/spark,milliman/spark,tejasapatil/spark,mike0sv/spark,dhruve/spark,janewangfb/spark,cin/spark,xuanyuanking/spark,bdrillard/spark,setjet/spark,jrshust/spark,rikima/spark,JerryLead/spark,kissmetrics/spark,patrick-nicholson/spark,aosagie/spark,adrian-ionescu/apache-spark,bdrillard/spark,mike0sv/spark,rikima/spark,LantaoJin/spark,nchammas/spark,ddna1021/spark,ericvandenbergfb/spark,rikima/spark,michalsenkyr/spark,saturday-shi/spark,ahnqirage/spark,publicRoman/spark,hhbyyh/spark,cloud-fan/spark,loneknightpy/spark,facaiy/spark,dbtsai/spark,saturday-shi/spark,WeichenXu123/spark,stanzhai/spark,ioana-delaney/spark,lxsmnv/spark,kimoonkim/spark,WindCanDie/spark,skonto/spark,big-pegasus/spark,darionyaphet/spark,mahak/spark,jiangxb1987/spark,liyichao/spark,debugger87/spark,yanboliang/spark,goldmedal/spark,holdenk/spark,kevinyu98/spark,gioenn/xSpark,pronix/spark,zuotingbing/spark,shubhamchopra/spark,gioenn/xSpark,ConeyLiu/spark,mike0sv/spark,cin/spark,JerryLead/spark,liyichao/spark,rikima/spark,minixalpha/spark,ueshin/apache-spark,tejasapatil/spark,zzcclp/spark,mzl9039/spark,hvanhovell/spark,aray/spark,nchammas/spark,joseph-torres/spark,chuckchen/spark,byakuinss/spark,mzl9039/spark,lvdongr/spark,dhruve/spark,WindCanDie/spark,mike0sv/spark,lxsmnv/spark,tejasapatil/spark,ioana-delaney/spark,ptkool/spark,kimoonkim/spark,kiszk/spark,shubhamchopra/spark,UndeadBaneGitHub/spark,jrshust/spark,ueshin/apache-spark,mzl9039/spark,aokolnychyi/spark,brad-kaiser/spark,spark-test/spark,apache-spark-on-k8s/spark,aray/spark,bravo-zhang/spark,JerryLead/spark,JoshRosen/spark,srowen/spark,map222/spark,Panos-Bletsos/spark-cost-model-optimizer,jlopezmalla/spark,MLnick/spark,saltstar/spark,setjet/spark,aray/spark,patrick-nicholson/spark,facaiy/spark,zhouyejoe/spark,tengpeng/spark,rezasafi/spark,nilsgrabbert/spark,tejasapatil/spark,zuotingbing/spark,kimoonkim/spark,adrian-ionescu/apache-spark,byakuinss/spark,huang1900/spark,markhamstra/spark,icexelloss/spark,hhbyyh/spark,xflin/spark,poffuomo/spark,panfengfeng/spark,map222/spark,tengpeng/spark,shuangshuangwang/spark,panfengfeng/spark,lvdongr/spark,goldmedal/spark,debugger87/spark,rekhajoshm/spark,maropu/spark,HyukjinKwon/spark,kiszk/spark,saturday-shi/spark,patrick-nicholson/spark,xuanyuanking/spark,rekhajoshm/spark,1haodian/spark,milliman/spark,dhruve/spark,ron8hu/spark,jianran/spark,goldmedal/spark,MLnick/spark,nlalevee/spark,icexelloss/spark,map222/spark,ron8hu/spark,xflin/spark,jkbradley/spark,HyukjinKwon/spark,maropu/spark,kevinyu98/spark,ueshin/apache-spark,kiszk/spark,metamx/spark,srowen/spark,shuangshuangwang/spark,wzhfy/spark,stanzhai/spark,ibm-research-ireland/sparkoscope,big-pegasus/spark,map222/spark,aokolnychyi/spark,byakuinss/spark,kissmetrics/spark,wangyum/spark,goldmedal/spark,techaddict/spark,sahilTakiar/spark,hvanhovell/spark,highfei2011/spark,andrewor14/spark,shubhamchopra/spark,actuaryzhang/spark,saltstar/spark,nilsgrabbert/spark,JoshRosen/spark,michalsenkyr/spark,esi-mineset/spark,rednaxelafx/apache-spark,lvdongr/spark,bdrillard/spark,debugger87/spark,jiangxb1987/spark,Panos-Bletsos/spark-cost-model-optimizer,sachintyagi22/spark,nilsgrabbert/spark,ueshin/apache-spark,haowu80s/spark,UndeadBaneGitHub/spark,jianran/spark,sahilTakiar/spark,spark-test/spark,LantaoJin/spark,big-pegasus/spark,debugger87/spark,milliman/spark,skonto/spark,1haodian/spark,SnappyDataInc/spark,patrick-nicholson/spark,Aegeaner/spark,jianran/spark,zzcclp/spark,BryanCutler/spark,saltstar/spark,rezasafi/spark,minixalpha/spark,Panos-Bletsos/spark-cost-model-optimizer,szhem/spark,janewangfb/spark,stanzhai/spark,caneGuy/spark,UndeadBaneGitHub/spark,hhbyyh/spark,liyichao/spark,vinodkc/spark,cin/spark,techaddict/spark,MLnick/spark,jkbradley/spark,caneGuy/spark,jrshust/spark,hvanhovell/spark,ron8hu/spark,akopich/spark,HyukjinKwon/spark,maropu/spark,ajaysaini725/spark,guoxiaolongzte/spark,skonto/spark,xuanyuanking/spark,apache-spark-on-k8s/spark,dbtsai/spark,gengliangwang/spark,MLnick/spark,zhouyejoe/spark,michalsenkyr/spark,akopich/spark,ptkool/spark,SnappyDataInc/spark,liutang123/spark,wangmiao1981/spark,bOOm-X/spark,zuotingbing/spark,holdenk/spark,huang1900/spark,vinodkc/spark,sachintyagi22/spark,apache-spark-on-k8s/spark,loneknightpy/spark,highfei2011/spark,spark-test/spark,facaiy/spark,jrshust/spark,dongjoon-hyun/spark,facaiy/spark,goldmedal/spark,dbtsai/spark,MLnick/spark,someorz/spark,szhem/spark,setjet/spark,esi-mineset/spark,publicRoman/spark,metamx/spark,sachintyagi22/spark,highfei2011/spark,joseph-torres/spark,loneknightpy/spark,rednaxelafx/apache-spark,vinodkc/spark,nilsgrabbert/spark,zuotingbing/spark,1haodian/spark,mahak/spark,mahak/spark,rikima/spark,srowen/spark,shuangshuangwang/spark,UndeadBaneGitHub/spark,andrewor14/spark,rezasafi/spark,Aegeaner/spark,alunarbeach/spark,yanboliang/spark,rekhajoshm/spark,cin/spark,dbtsai/spark,ericvandenbergfb/spark,ueshin/apache-spark,shubhamchopra/spark,haowu80s/spark,ericvandenbergfb/spark,jiangxb1987/spark,publicRoman/spark,joseph-torres/spark,HyukjinKwon/spark,techaddict/spark,jkbradley/spark,map222/spark,nlalevee/spark,esi-mineset/spark,poffuomo/spark,jianran/spark,jlopezmalla/spark,pronix/spark,alunarbeach/spark,andrewor14/spark,WindCanDie/spark,tejasapatil/spark,milliman/spark,mdespriee/spark,JoshRosen/spark,pgandhi999/spark,ajaysaini725/spark,alunarbeach/spark,mzl9039/spark,tengpeng/spark,adrian-ionescu/apache-spark,adrian-ionescu/apache-spark,aokolnychyi/spark,cloud-fan/spark,techaddict/spark,facaiy/spark,srowen/spark,huang1900/spark,tengpeng/spark,rednaxelafx/apache-spark,JoshRosen/spark,minixalpha/spark,darionyaphet/spark,adrian-ionescu/apache-spark,jlopezmalla/spark,metamx/spark,yanboliang/spark,nilsgrabbert/spark,dhruve/spark,wangyum/spark,actuaryzhang/spark,setjet/spark,WindCanDie/spark,janewangfb/spark,ron8hu/spark,apache/spark,bdrillard/spark,techaddict/spark,Panos-Bletsos/spark-cost-model-optimizer,jiangxb1987/spark,janewangfb/spark,setjet/spark,esi-mineset/spark,SHASHANKB/spark,adrian-ionescu/apache-spark,hhbyyh/spark,akopich/spark,witgo/spark,ddna1021/spark,cloud-fan/spark,poffuomo/spark,bdrillard/spark,UndeadBaneGitHub/spark,rikima/spark,metamx/spark,taroplus/spark,minixalpha/spark,liutang123/spark,saltstar/spark,sureshthalamati/spark,jrshust/spark,eyalfa/spark,narahari92/spark