[Tests] Fix flaky test_failures by using semaphore instead of sleep (#10942)
[Tests] Fix flaky test_failures by using semaphore instead of sleep (#10942)

* Use semaphore to avoid timing issues.

* Addressed code review.
richardliaw/ray,richardliaw/ray,richardliaw/ray,ray-project/ray,pcmoritz/ray-1,ray-project/ray,pcmoritz/ray-1,pcmoritz/ray-1,pcmoritz/ray-1,richardliaw/ray,richardliaw/ray,pcmoritz/ray-1,pcmoritz/ray-1,ray-project/ray,richardliaw/ray,ray-project/ray,ray-project/ray,richardliaw/ray,ray-project/ray,richardliaw/ray,ray-project/ray,ray-project/ray,pcmoritz/ray-1,pcmoritz/ray-1