fix crash when num procs > 1 (#2623)
fix crash when num procs > 1 (#2623)

Summary:
this fixes a crash when using num_workers > 0 and num processes > 1 (e.g. by launching on slurm with tasks-per-node > 1 or using torch.distributed/torchrun with num gpus > 1). the root cause is some kind of bug in torch.split() that holds on to memory. when data loaders recycle, the program crashes when trying to release already released memory

Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/2623

Reviewed By: arbabu123

Differential Revision: D32342431

Pulled By: alexeib

fbshipit-source-id: 278c7e0b09cfd770f375865d472bc52fb5835bd2

pytorch/fairseq,pytorch/fairseq,pytorch/fairseq