Work around differences in internal and external Gym.
Work around differences in internal and external Gym.

PiperOrigin-RevId: 236956022

tensorflow/tensor2tensor,tensorflow/tensor2tensor,tensorflow/tensor2tensor,tensorflow/tensor2tensor,tensorflow/tensor2tensor