Ensure checkpoints are properly saved when hitting stop time limit
Ensure checkpoints are properly saved when hitting stop time limit

Summary:
Context: https://fburl.com/tasks/3nhdm1rv and https://fburl.com/test/4i6icfbd

This should fix test_e2e_base_training_wo_prepare_data breakage due to D22598579: when stop time limit https://fburl.com/diffusion/rrp4jst5 gets triggered, a checkpoint isn't saved as it should in https://fburl.com/diffusion/hbn0c6o5. The error isn't related to Manifold usage in export_ensemble - that's just the first place where we notice that no checkpoint was written.

So for toy tests that finish quickly, it's possible for training to end before any checkpoint has been saved.

It was affecting dev-nosan but not opt probably because opt training finishes quickly enough to not hit the stop time limit?

Reviewed By: akinh

Differential Revision: D22639199

fbshipit-source-id: ec4da15bb14e14c2066af6946d7a34db333178eb

pytorch/fairseq,pytorch/fairseq,pytorch/fairseq