Fix failing ep-engine checkpoint unit test
Fix failing ep-engine checkpoint unit test

The following issues were fixed:
1. Simplified the logic to join the worker threads created by the main
   thread. Essentially, since we do not care about the actual return
   value from the threads, it is alright to return a NULL from the
   threads and throw it away in the main thread. If at all we must
   assert some condition, we should do so in the worker thread. The
   previous method of calling pthread_join() was causing a segfault
   due to improper casting of the return value.
2. Fixed the persistence worker thread to read the queue items front
   to back instead of back to front. If the persistence thread is slow
   for whatever reason, reading the queue back to front may cause the
   flush operation to be discovered before one or more of the regular
   data items. Consequently, the persistence thread will quit before
   it has finished reading the entire queue. This was causing a timing
   dependent assertion failure.
3. Fixed the persistence worker thread to also account for checkpoint
   start and end operations. The previous code assumed that the
   presence of a flush operation indicated end of the queue. However,
   this may not be true because new checkpoint start and end ops may
   get introduced while processing the regular queue items.
4. Fixed the cleanup worker thread to *not* assert the numItemsRemoved
   be zero. The reason this may not be zero is because we may create
   new checkpoints during the test and thus end up with one or more
   closed and unreferenced checkpoints. Consequently, a call to the
   removeClosedUnrefCheckpoints() function may return a non-zero value.

Change-Id: Ie9725ac830ae3ce30a4b3e8c0cb4e9da62b958bd
Reviewed-on: http://review.couchbase.org/7696
Tested-by: Trond Norbye <60edd2ef23891a753f231b0c6f161dc634079a93@gmail.com>
Reviewed-by: Trond Norbye <60edd2ef23891a753f231b0c6f161dc634079a93@gmail.com>

abhinavdangeti/ep-engine,abhinavdangeti/ep-engine,couchbaselabs/ep-engine,teligent-ru/ep-engine,daverigby/kv_engine,daverigby/ep-engine,daverigby/ep-engine,teligent-ru/ep-engine,abhinavdangeti/ep-engine,abhinavdangeti/ep-engine,zbase/ep-engine,zbase/ep-engine,membase/ep-engine,zbase/ep-engine,membase/ep-engine,daverigby/kv_engine,sriganes/ep-engine,membase/ep-engine,jimwwalker/ep-engine,daverigby/ep-engine,hisundar/ep-engine,daverigby/ep-engine,teligent-ru/ep-engine,teligent-ru/ep-engine,owendCB/ep-engine,jimwwalker/ep-engine,hisundar/ep-engine,zbase/ep-engine,couchbaselabs/ep-engine,sriganes/ep-engine,couchbase/ep-engine,daverigby/kv_engine,zbase/ep-engine,daverigby/kv_engine,couchbase/ep-engine,abhinavdangeti/ep-engine,couchbase/ep-engine,hisundar/ep-engine,jimwwalker/ep-engine,membase/ep-engine,sriganes/ep-engine,owendCB/ep-engine,couchbaselabs/ep-engine,couchbaselabs/ep-engine,owendCB/ep-engine,owendCB/ep-engine,couchbaselabs/ep-engine,sriganes/ep-engine,hisundar/ep-engine,jimwwalker/ep-engine,couchbase/ep-engine