fix pyspeech jobs hanging at last epoch
fix pyspeech jobs hanging at last epoch

Summary:
Fixes problem from https://fb.workplace.com/groups/332923290658088/permalink/571010136849401/.

In general, we seem to sometimes hang at the last epoch because one of the ranks exits early (if the rank is rank 0 then the job will likely fail). This  happens mostly for aligned training because number of batches vary the most there between different ranks.  I've debugged this problem to be:

1. Before D20160715, this hanging would occur if the job fails at the last epoch, restarts, and there is inconsistency between batches for each rank. This is because when we restart at epoch Y update X, if there exists a rank equal to or less than X batches,  then fairseq will detect that rank as end of epoch and move on to epoch Y + 1. If epoch Y + 1 > max_epochs, then this rank exits. Ranks with batches > X will detect not end of epoch and try to sync, causing hanging. This situation is quite rare as we need to fail on final epoch, if you restart before final epoch epoch syncing will guarantee that all ranks are in sync.

2. After D20160715 and D20393469, this hanging would occur very often as epoch syncing no longer works because `next_epoch_itr ` no longer increments the epoch if you just loaded from a checkpoint. This means if you fail and restart at any point you might face the hanging problem above.

The fix for 1 is to explicitly sync the epochs **before** we check if training is done.
The fix for 2 is to increment epoch explicitly when syncing epochs.

Differential Revision: D20746950

fbshipit-source-id: bfce8e78619949667526a3b877a288f5f5cf2dda

pytorch/fairseq,pytorch/fairseq,pytorch/fairseq