enable pin_memory for DataLoaders (#3560)
enable pin_memory for DataLoaders (#3560)

Summary:
To avoid the creation of a cuda:0 context, I needed to make sure that the `BackgroundConsumer` thread had its cuda device context set to the correct GPU.

Pull Request resolved: https://github.com/pytorch/fairseq/pull/3560

Reviewed By: myleott

Differential Revision: D28573071

Pulled By: msbaines

fbshipit-source-id: c2bedf67d8f356a29fa82eb4d8f15983efce3ffc

pytorch/fairseq,pytorch/fairseq,pytorch/fairseq