Use 8-bit quantization only in cases when it makes sense.
Use 8-bit quantization only in cases when it makes sense.

Summary:
In some cases we were doing quantization even we we should not. This diff is
preventing this from happening.

Reviewed By: rayleichen

Differential Revision: D6953547

fbshipit-source-id: 7c65baaf969e5e1bddb68ca8182f4f3b43f2431d

caffe2/caffe2,Yangqing/caffe2,xzturn/caffe2,Yangqing/caffe2,xzturn/caffe2,Yangqing/caffe2,xzturn/caffe2,xzturn/caffe2,Yangqing/caffe2,Yangqing/caffe2,xzturn/caffe2