Make object_manager_test::test_object_transfer_retry less flaky. (#4057)
Make object_manager_test::test_object_transfer_retry less flaky. (#4057)

* Make object_manager_test::test_object_transfer_retry less flaky.

* Make the test pass.

atumanov/ray,atumanov/ray,ujvl/ray-ng,stephanie-wang/ray,ray-project/ray,pcmoritz/ray-1,robertnishihara/ray,ray-project/ray,richardliaw/ray,stephanie-wang/ray,stephanie-wang/ray,atumanov/ray,ray-project/ray,pcmoritz/ray-1,atumanov/ray,richardliaw/ray,stephanie-wang/ray,pcmoritz/ray-1,pcmoritz/ray-1,atumanov/ray,ray-project/ray,ray-project/ray,richardliaw/ray,pcmoritz/ray-1,pcmoritz/ray-1,richardliaw/ray,richardliaw/ray,stephanie-wang/ray,ray-project/ray,richardliaw/ray,stephanie-wang/ray,robertnishihara/ray,ray-project/ray,atumanov/ray,pcmoritz/ray-1,stephanie-wang/ray,stephanie-wang/ray,pcmoritz/ray-1,ujvl/ray-ng,ujvl/ray-ng,atumanov/ray,robertnishihara/ray,atumanov/ray,ujvl/ray-ng,robertnishihara/ray,ray-project/ray,robertnishihara/ray,robertnishihara/ray,richardliaw/ray,robertnishihara/ray,robertnishihara/ray,richardliaw/ray,stephanie-wang/ray,robertnishihara/ray