test: Allow prometheus state clear to pass etcd leader change test
test: Allow prometheus state clear to pass etcd leader change test

If the prometheus process on a conformant cluster has no PVs, the
data in prometheus is lost and the current query will flag that as
a leader election (new prometheus scrapes the old etcd and sees the
increase). The tightened tolerance turned out to be too tight for
serial tests, where we allow the machine set test to replace one
of the workers. If we were using PVs for prometheus in all machine
set clusters this would be ok, but it's valid today to have a
machine set cluster with no PV provisioning.

The test now ignores restarts that happen before the beginning of
prometheus history, and only looks at leader changes over the time
prometheus is up and running with the latest data.

hroyrh/origin,jsafrane/origin,jeremyeder/origin,mahak/origin,coreydaley/origin,mahak/origin,deads2k/origin,jeremyeder/origin,JacobTanenbaum/origin,bparees/origin,JacobTanenbaum/origin,spadgett/origin,jeremyeder/origin,jsafrane/origin,sallyom/origin,jsafrane/origin,bparees/origin,gabemontero/origin,JacobTanenbaum/origin,sallyom/origin,coreydaley/origin,jeremyeder/origin,xiuwang/origin,hroyrh/origin,deads2k/origin,mfojtik/origin,coreydaley/origin,akram/origin,soltysh/origin,csrwng/origin,xiuwang/origin,mfojtik/origin,xiuwang/origin,akram/origin,jeremyeder/origin,soltysh/origin,mfojtik/origin,sallyom/origin,openshift/origin,bparees/origin,mahak/origin,openshift/origin,gabemontero/origin,csrwng/origin,gabemontero/origin,spadgett/origin,openshift/origin,spadgett/origin,spadgett/origin,soltysh/origin,hroyrh/origin,csrwng/origin,akram/origin,spadgett/origin,deads2k/origin