raft rapid group recreation test
raft rapid group recreation test

this simulates a rare error surfaced by #906 (and apparently others).
the test has a handful of concurrent actors add, commit to
(with retries) and remove a group.

currently this sometimes panics Raft:

E0506 13:09:45.552958   78550 log.go:188] applied(304) is out of range [prevApplied(10), committed(303)]
panic: applied(304) is out of range [prevApplied(10), committed(303)]

and other times (especially with a high number of writers) there are
deadlock issues.

i will continue to investigate these phenomena and hopefully attach
a fix to this PR once it merges.

jamesgraves/cockroach,jgautheron/cockroach,bg451/cockroach,yosiat/cockroach,petermattis/cockroach,ajayaa/cockroach,ajayaa/cockroach,nieyy/cockroach,arjun4084346/cockroach,bg451/cockroach,abhishekgahlot/cockroach,fengshao0907/cockroach,cuongdo/cockroach,domluna/cockroach,vaibhavarora/cockroach,jamesgraves/cockroach,arjun4084346/cockroach,veteranlu/cockroach,nkhuyu/cockroach,mbonaci/cockroach,slavau/cockroach,c4pt0r/cockroach,domluna/cockroach,yosiat/cockroach,bg451/cockroach,maxlang/cockroach,paperstreet/cockroach,nieyy/cockroach,mbertschler/cockroach,nkhuyu/cockroach,briceburg/cockroach,yosiat/cockroach,chrisseto/cockroach,freakynit/cockroach,nieyy/cockroach,veteranlu/cockroach,arypurnomoz/cockroach,sunya123/cockroach,nkhuyu/cockroach,jmank88/cockroach,petermattis/cockroach,vivekmenezes/cockroach,vaibhavarora/cockroach,mbonaci/cockroach,freakynit/cockroach,alunarbeach/cockroach,sunya123/cockroach,paperstreet/cockroach,GokulSrinivas/cockroach,alunarbeach/cockroach,mbertschler/cockroach,c4pt0r/cockroach,alunarbeach/cockroach,arypurnomoz/cockroach,vivekmenezes/cockroach,SustainFund/cockroach,cockroachdb/cockroach,Abioy/cockroach,ajayaa/cockroach,freakynit/cockroach,il9ue/cockroach,yosiat/cockroach,hassyma/cockroach,vivekmenezes/cockroach,freakynit/cockroach,sunya123/cockroach,yiixiansheng/cockroach,iqstack/cockroach,briceburg/cockroach,domluna/cockroach,vaibhavarora/cockroach,fengshao0907/cockroach,ngaut/cockroach,petermattis/cockroach,hassyma/cockroach,GokulSrinivas/cockroach,XisiHuang/cockroach,GokulSrinivas/cockroach,chrisseto/cockroach,arypurnomoz/cockroach,slavau/cockroach,loucash/cockroach,c4pt0r/cockroach,vaibhavarora/cockroach,abhishekgahlot/cockroach,loucash/cockroach,jmank88/cockroach,jess-edwards/cockroach,slavau/cockroach,cuongdo/cockroach,domluna/cockroach,ngaut/cockroach,ngaut/cockroach,domluna/cockroach,cockroachdb/cockroach,vaibhavarora/cockroach,alunarbeach/cockroach,abhishekgahlot/cockroach,paperstreet/cockroach,linearb/cockroach,cuongdo/cockroach,fengshao0907/cockroach,sunya123/cockroach,cuongdo/cockroach,vivekmenezes/cockroach,SustainFund/cockroach,petermattis/cockroach,jamesgraves/cockroach,iqstack/cockroach,il9ue/cockroach,il9ue/cockroach,XisiHuang/cockroach,mbertschler/cockroach,nkhuyu/cockroach,bg451/cockroach,nieyy/cockroach,cuongdo/cockroach,nieyy/cockroach,arypurnomoz/cockroach,slavau/cockroach,ajayaa/cockroach,abhishekgahlot/cockroach,mbertschler/cockroach,slavau/cockroach,hassyma/cockroach,maxlang/cockroach,jgautheron/cockroach,yiixiansheng/cockroach,jgautheron/cockroach,cockroachdb/cockroach,glycerine/cockroach,cockroachdb/cockroach,sunya123/cockroach,freakynit/cockroach,vivekmenezes/cockroach,Abioy/cockroach,alex/cockroach,jess-edwards/cockroach,marcuswestin/cockroach,domluna/cockroach,loucash/cockroach,vaibhavarora/cockroach,iqstack/cockroach,fengshao0907/cockroach,paperstreet/cockroach,briceburg/cockroach,Abioy/cockroach,hassyma/cockroach,jess-edwards/cockroach,il9ue/cockroach,cuongdo/cockroach,briceburg/cockroach,alunarbeach/cockroach,jmank88/cockroach,marcuswestin/cockroach,GokulSrinivas/cockroach,yiixiansheng/cockroach,bg451/cockroach,cockroachdb/cockroach,alunarbeach/cockroach,chrisseto/cockroach,c4pt0r/cockroach,alex/cockroach,maxlang/cockroach,maxlang/cockroach,petermattis/cockroach,ajayaa/cockroach,domluna/cockroach,veteranlu/cockroach,jamesgraves/cockroach,mbonaci/cockroach,vaibhavarora/cockroach,jgautheron/cockroach,chrisseto/cockroach,hassyma/cockroach,glycerine/cockroach,vivekmenezes/cockroach,glycerine/cockroach,abhishekgahlot/cockroach,paperstreet/cockroach,petermattis/cockroach,nieyy/cockroach,yiixiansheng/cockroach,mbertschler/cockroach,c4pt0r/cockroach,veteranlu/cockroach,SustainFund/cockroach,linearb/cockroach,abhishekgahlot/cockroach,il9ue/cockroach,il9ue/cockroach,jmank88/cockroach,arjun4084346/cockroach,briceburg/cockroach,loucash/cockroach,alex/cockroach,jess-edwards/cockroach,iqstack/cockroach,jamesgraves/cockroach,jess-edwards/cockroach,iqstack/cockroach,jgautheron/cockroach,petermattis/cockroach,marcuswestin/cockroach,veteranlu/cockroach,nieyy/cockroach,paperstreet/cockroach,maxlang/cockroach,ngaut/cockroach,paperstreet/cockroach,mbonaci/cockroach,chrisseto/cockroach,loucash/cockroach,arjun4084346/cockroach,XisiHuang/cockroach,veteranlu/cockroach,Abioy/cockroach,cuongdo/cockroach,jmank88/cockroach,sunya123/cockroach,XisiHuang/cockroach,SustainFund/cockroach,SustainFund/cockroach,glycerine/cockroach,marcuswestin/cockroach,alex/cockroach,bg451/cockroach,arypurnomoz/cockroach,mbonaci/cockroach,glycerine/cockroach,nkhuyu/cockroach,iqstack/cockroach,linearb/cockroach,sunya123/cockroach,jess-edwards/cockroach,sunya123/cockroach,chrisseto/cockroach,fengshao0907/cockroach,glycerine/cockroach,linearb/cockroach,vivekmenezes/cockroach,c4pt0r/cockroach,cuongdo/cockroach,nieyy/cockroach,Abioy/cockroach,bg451/cockroach,XisiHuang/cockroach,cockroachdb/cockroach,arjun4084346/cockroach,jamesgraves/cockroach,XisiHuang/cockroach,arypurnomoz/cockroach,vaibhavarora/cockroach,alex/cockroach,marcuswestin/cockroach,fengshao0907/cockroach,arjun4084346/cockroach,jgautheron/cockroach,marcuswestin/cockroach,jamesgraves/cockroach,yosiat/cockroach,paperstreet/cockroach,ngaut/cockroach,linearb/cockroach,marcuswestin/cockroach,mbertschler/cockroach,arjun4084346/cockroach,briceburg/cockroach,maxlang/cockroach,jess-edwards/cockroach,loucash/cockroach,linearb/cockroach,Abioy/cockroach,cockroachdb/cockroach,ngaut/cockroach,yosiat/cockroach,mbonaci/cockroach,jmank88/cockroach,yiixiansheng/cockroach,ngaut/cockroach,GokulSrinivas/cockroach,glycerine/cockroach,arjun4084346/cockroach,nkhuyu/cockroach,maxlang/cockroach,veteranlu/cockroach,veteranlu/cockroach,jmank88/cockroach,petermattis/cockroach,maxlang/cockroach,freakynit/cockroach,alex/cockroach,ajayaa/cockroach,SustainFund/cockroach,slavau/cockroach,mbertschler/cockroach,iqstack/cockroach,cockroachdb/cockroach,yiixiansheng/cockroach,il9ue/cockroach,il9ue/cockroach,vivekmenezes/cockroach,marcuswestin/cockroach,linearb/cockroach,glycerine/cockroach,hassyma/cockroach,GokulSrinivas/cockroach