add sanity check that ops do not cross gpus
add sanity check that ops do not cross gpus

Summary: Debugging nets can be tiresome, so it is good if we can do some sanity checks. This adds a sanity check that all non-NCCL and non-Copy operators do not reference blobs that have different device scope than the operator. This check is only added to the data_parallel_model, so it should be safe. This check would had caught a subtle bugin prigoyal's training pipeline.

Reviewed By: dzhulgakov

Differential Revision: D4230444

fbshipit-source-id: 3d4a843162134a7a504053d95ff97a552e6b8a6d

davinwang/caffe2,pietern/caffe2,davinwang/caffe2,davinwang/caffe2,bwasti/caffe2,Yangqing/caffe2,pietern/caffe2,davinwang/caffe2,caffe2/caffe2,Yangqing/caffe2,xzturn/caffe2,xzturn/caffe2,bwasti/caffe2,Yangqing/caffe2,xzturn/caffe2,xzturn/caffe2,pietern/caffe2,pietern/caffe2,sf-wind/caffe2,sf-wind/caffe2,bwasti/caffe2,sf-wind/caffe2,davinwang/caffe2,Yangqing/caffe2,pietern/caffe2,bwasti/caffe2,bwasti/caffe2,sf-wind/caffe2,Yangqing/caffe2,sf-wind/caffe2,xzturn/caffe2